A multi-modal pose assessment method and apparatus

Abstract

The application discloses a kind of multi-modal pose evaluation method and device, belong to health monitoring field, through the dynamic video of human body is collected by camera, obtain video data;Through the pressure distribution data of pose related that pressure sensor is set in cushion and vest is collected;Through the fiber-optic sensor that is set in vest is pasted with spine and measures the bending data of spine;Uniform time resolution and spatial registration are carried out to data, feature extraction is carried out to three kinds of data, and the features after extraction are weighted and fused;Design and train deep learning model: the multiple features after weighting and fusing are input into the deep learning model, and output evaluation result, through the above steps, the pose of person in different scenes can be accurately evaluated.

Description

Technical Field

[0001] This invention relates to the field of health monitoring, and in particular to a multimodal pose assessment method and apparatus. Background Technology

[0002] With increasing societal focus on children's health, issues such as scoliosis and postural disorders caused by poor standing, sitting, and exercise habits have become a key area of ​​public health concern. Currently, some posture monitoring systems exist on the market, but these systems often rely on a single sensor, such as a camera or pressure sensor, making it difficult to comprehensively assess the multidimensional characteristics of posture and thus unable to accurately evaluate a person's posture in different scenarios. Summary of the Invention

[0003] In order to overcome the shortcomings of the prior art, one of the objectives of this invention is to provide a multimodal pose assessment method that can accurately assess human pose in different scenarios.

[0004] To overcome the shortcomings of the prior art, the second objective of this invention is to provide a multimodal pose assessment device that can accurately assess a person's pose in different scenarios.

[0005] One of the objectives of this invention is achieved through the following technical solution:

[0006] A multimodal pose evaluation method includes the following steps:

[0007] Data acquisition: The system acquires dynamic video of the human body through a camera to obtain video data; pressure sensors installed on the seat cushion and vest collect posture-related pressure distribution data; and fiber optic sensors installed on the vest are used to measure spinal curvature data by fitting the spine.

[0008] Data fusion: Video data and pressure distribution data are time-aligned using multiple spline interpolation; bending data is downsampled to match the pressure distribution data, thus unifying the temporal resolution of the data from the camera, pressure sensor, and fiber optic sensor; a global attitude coordinate system is established, and an affine transformation method is used to map the pressure sensor mesh to the topology of the skeleton key points, calculate the force in the corresponding area, optimize the alignment error between the coordinate systems of the camera, pressure sensor, and fiber optic sensor, and improve spatial matching accuracy;

[0009] Feature extraction: Deep neural networks are used to estimate the pose of video data, extract the coordinates of human key points, and calculate the trunk tilt angle, head offset, and motion frequency; the coordinates of the pressure center, asymmetry index, longitudinal pressure gradient, and lateral pressure fluctuation are calculated based on pressure distribution data; the curvature of the spine at various locations and the spinal curvature angle are calculated based on the spinal curvature data; attention weights for different extracted features are calculated, and the features are weighted and fused.

[0010] Design and train a deep learning model: Construct a deep learning model using a composite loss function, optimize classification and regression tasks, and prevent modal collapse; Collect multiple pose data including incorrect and normal poses, and label the data using a multimodal fusion method by combining video key point detection, pressure distribution map analysis, and fiber curvature signal calculation; Train the model using a staged training method.

[0011] Result evaluation: Input the weighted and fused features into the deep learning model and output the evaluation results.

[0012] Furthermore, the data fusion step also includes employing a time-series window sliding mechanism to perform linear interpolation on the video data, the pressure distribution data, and the spinal curvature data within a short time scale, in order to reduce abrupt errors.

[0013] Furthermore, in the data fusion step, the establishment of the global pose coordinate system is specifically as follows: with the 7th cervical vertebra as the origin, the spine direction as the Z-axis, the shoulder direction as the X-axis, and the front-back direction as the Y-axis.

[0014] Furthermore, in the feature extraction step, the torso tilt angle Here are the coordinates of key points on the left hip. Coordinates of key points on the right hip; head offset. P nose Let's say it's the coordinate point of the tip of the nose. The coordinates of the shoulder center point are given; the motion frequency is obtained by analyzing the displacement spectrum of key shoulder points to extract the frequencies of the main motion modes.

[0015] Furthermore, in the feature extraction step, the pressure center coordinates p i x represents the pressure value measured by the i-th pressure sensor; i y i Indicates the coordinates of the sensor; asymmetry index In the formula ∑p left The total pressure from the left-side pressure sensor is ∑p. right The total pressure from the pressure sensor on the right, ∑p total Total pressure; longitudinal pressure gradient ΔP is the pressure difference between the upper and lower pressure sensors, and Δh is the vertical distance between the two pressure sensors; lateral pressure fluctuation p i Let be the pressure value of the i-th sensor, μ be the average pressure of all sensors, and N be the number of sensors.

[0016] Furthermore, in the feature extraction step, the curvature of the spine at various locations... Δλ B λ is the amount of shift in the center Bragg wavelength of the FBG. B S is the initial Bragg wavelength of the FBG, and S is the strain sensitivity factor; the spinal curvature angle α = ∫κ(s)ds, where s is the arc length coordinate along the length of the spine, and κ(s) is the local curvature of the spine at different positions.

[0017] Furthermore, in the step of designing and training the deep learning model, three sub-models are designed, corresponding to dynamic video, pressure sensor and fiber optic sensor respectively. The three sub-models output the confidence distribution of the pose category respectively, and decision fusion is performed through DS synthesis rules.

[0018] Furthermore, in the step of designing and training the deep learning model, the composite loss function is L = λ1L cls +λ2L reg +λ3L ortho In the formula, λ1 is the weight parameter of the classification loss, and L cls For classification loss, λ² is the weighting parameter for regression loss, and L... reg For regression loss, λ3 is the weighting parameter of orthogonal loss, and L... ortho This is an orthogonal loss.

[0019] Furthermore, in the step of designing and training the deep learning model, the staged training method specifically involves: First, single-modal pre-training is performed, where the video branch is initialized using HRNet to improve the accuracy of keypoint detection; second, the backbone network for feature extraction of each modality is fixed, and only the fusion layer is optimized to ensure cross-modal information alignment; third, end-to-end fine-tuning is performed, adjusting the learning rate to 1e. -4 To balance convergence speed and optimization accuracy.

[0020] Furthermore, in the result evaluation step, the user's real-time posture is dynamically displayed through a 3D human body model or skeletal point projection, the poor posture is marked with color coding, the pressure distribution is displayed through a heat map, highlighting areas with excessive pressure, and the trend of long-term posture changes is tracked by combining time series curves, and the change curve of spinal curvature is displayed in real time.

[0021] The second objective of this invention is achieved by the following technical solution:

[0022] A multimodal pose evaluation device is provided for implementing any of the above-described multimodal pose evaluation methods. The multimodal pose evaluation device includes...

[0023] A camera captures dynamic video of the human body to obtain video data;

[0024] A pressure sensor is installed on the seat cushion and vest to collect posture-related pressure distribution data.

[0025] An optical fiber sensor is mounted on a vest and is fitted to the spine to measure spinal curvature data.

[0026] The processor analyzes the acquired video data, pressure distribution data, and bending data to assess the user's pose.

[0027] Compared to existing technologies, the multimodal pose assessment method of this invention acquires dynamic video of the human body through a camera to obtain video data; collects pose-related pressure distribution data through pressure sensors installed on a cushion and vest; measures spinal curvature data by using a fiber optic sensor installed on the vest to fit the spine; performs unified temporal resolution and spatial registration on the data; extracts features from the three types of data; and performs weighted fusion of the extracted features. A deep learning model is designed and trained: the weighted fused features are input into the deep learning model, and the evaluation result is output. Through the above steps, the pose of a person in different scenarios can be accurately assessed. Attached Figure Description

[0028] Figure 1 This is a flowchart of the multimodal pose evaluation method of the present invention;

[0029] Figure 2 This is a schematic diagram of the vest used in the multimodal pose evaluation method of the present invention;

[0030] Figure 3 This is a schematic diagram of the cushion used in the multimodal pose evaluation method of the present invention;

[0031] Figure 4 This is a schematic diagram of skeleton key point extraction in the multimodal pose evaluation method of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0034] Please see Figure 1 The present invention provides a multimodal pose evaluation method, comprising the following steps:

[0035] Data acquisition: The system acquires dynamic video of the human body through a camera to obtain video data; pressure sensors installed on the seat cushion and vest collect posture-related pressure distribution data; and fiber optic sensors installed on the vest are used to measure spinal curvature data by fitting the spine.

[0036] Data fusion: Video data and pressure distribution data are time-aligned using multiple spline interpolation; bending data is downsampled to match the pressure distribution data, thus unifying the temporal resolution of the data from the camera, pressure sensor, and fiber optic sensor; a global attitude coordinate system is established, and an affine transformation method is used to map the pressure sensor mesh to the topology of the skeleton key points, calculate the force in the corresponding area, optimize the alignment error between the coordinate systems of the camera, pressure sensor, and fiber optic sensor, and improve spatial matching accuracy;

[0037] Feature extraction: Deep neural networks are used to estimate the pose of video data, extract the coordinates of human key points, and calculate the trunk tilt angle, head offset, and motion frequency; the coordinates of the pressure center, asymmetry index, longitudinal pressure gradient, and lateral pressure fluctuation are calculated based on pressure distribution data; the curvature of the spine at various locations and the spinal curvature angle are calculated based on the spinal curvature data; attention weights for different extracted features are calculated, and the features are weighted and fused.

[0038] Design and train a deep learning model: Construct a deep learning model using a composite loss function, optimize classification and regression tasks, and prevent modal collapse; Collect multiple pose data including incorrect and normal poses, and label the data using a multimodal fusion method by combining video key point detection, pressure distribution map analysis, and fiber curvature signal calculation; Train the model using a staged training method.

[0039] Result evaluation: Input the weighted and fused features into the deep learning model and output the evaluation results.

[0040] Specifically, the data collection steps are as follows:

[0041] A binocular camera is used to capture dynamic video of the human body to facilitate the extraction of key skeletal points. In this embodiment, the user is a child, and assessing the child's posture can help detect and evaluate health conditions such as scoliosis and postural disorders. The current video frame is acquired from the binocular camera and the video frame sequence is updated; the video frame sequence is sent to the human posture estimation and behavior recognition module, and the 3D coordinates of key points are obtained through 3D reconstruction of the left and right camera images to obtain video data.

[0042] Please continue reading. Figure 2 as well as Figure 3 Pressure sensors are installed on the back of the vest and the seat cushion. Since different sitting postures result in different pressures on different parts of the back and seat cushion, collecting pressure data on different parts of the back and seat cushion can more accurately assess the body's posture.

[0043] The fiber optic sensor is positioned in the center of the corrective vest, arranged in a double S-shape to fit the spine, and is used to measure the curvature of the spine.

[0044] The multimodal pose assessment method of this invention further includes a data preprocessing step, which is located after the data acquisition step and before the data fusion step. Specifically, the data preprocessing step involves: acquiring pose-related pressure distribution data using pressure sensors installed on the cushion and vest; and processing video data through background removal and denoising, and skeletal point extraction (e.g.,...). Figure 4 The system extracts dynamic trajectory information of key areas through steps such as time-series data generation and image standardization. Pressure data is analyzed through signal filtering, zonal pressure analysis, and time-series trend analysis to detect changes in pressure distribution in different areas and their potential adverse postural effects. For fiber optic sensor-based spinal curvature monitoring, fiber Bragg grating (FBG) sensing technology is additionally introduced to acquire real-time changes in spinal curvature. By fusing fiber optic data with video skeletal point information, adaptive filtering and nonlinear regression algorithms are employed to further improve the detection accuracy of spinal deformation, providing more comprehensive support for posture correction.

[0045] The data fusion process includes unifying the temporal resolution and spatial registration.

[0046] Because different sensors have different sampling rates, a unified time resolution is required:

[0047] Video data (30fps) and pressure data (100Hz) are time-aligned using cubic spline interpolation to improve data continuity and accuracy; fiber optic data (200Hz) is downsampled to 100Hz to match pressure sensor data, and anti-aliasing filtering is used to reduce information loss; a time window sliding mechanism is used to perform linear interpolation within a short time scale (250ms) to reduce abrupt errors.

[0048] Spatial Registration: To ensure that data from different modalities are fused and analyzed in the same coordinate system, a global attitude coordinate system is established, with the 7th cervical vertebra (C7) as the origin. The longitudinal direction of the vest (spine direction) is defined as the Z-axis, the shoulder direction as the X-axis, and the anterior-posterior direction as the Y-axis. An affine transformation method is used to map the pressure sensor mesh onto the skeleton keypoint topology, and the force conditions in the corresponding regions are calculated. The ICP algorithm is used to optimize the alignment error between different sensor coordinate systems, improving spatial matching accuracy.

[0049] The feature extraction steps include feature extraction from video data, feature extraction from pressure distribution data, and feature extraction from spinal curvature data.

[0050] An improved HRNet-W48 deep neural network was used for pose estimation during video data feature extraction, extracting 26 key points of the human body. Key point coordinate data underwent low-pass filtering to reduce the impact of jitter noise on feature calculation. Trunk tilt angle, head offset, and motion frequency features were calculated. The trunk tilt angle measures spinal alignment, the head offset reflects head posture asymmetry, and the motion frequency features are used to assess long-term posture stability. Trunk tilt angle... Here are the coordinates of key points on the left hip. Coordinates of key points on the right hip; head offset. P nose Let's say it's the coordinate point of the tip of the nose. The coordinates of the shoulder center point are used; the displacement spectrum of the key points of the shoulder is analyzed using Fast Fourier Transform (FFT) to extract the frequencies of the main motion modes.

[0051] The purpose of pressure distribution data feature extraction is to evaluate the force distribution characteristics using pressure sensor data from the seat cushion and vest. Feature extraction includes calculating the pressure center coordinates, asymmetry index, longitudinal pressure gradient, and lateral pressure fluctuation. The pressure center coordinates are used to analyze the stability of the seated center of gravity; the asymmetry index measures the balance of force distribution from left to right; the longitudinal pressure gradient assesses the variation of back pressure along the spine; and the lateral pressure fluctuation measures the fluctuation of force in different areas of the back. Pressure center coordinates... p i x represents the pressure value measured by the i-th pressure sensor; i y i Indicates the coordinates of the sensor; asymmetry index In the formula ∑p left The total pressure from the left-side pressure sensor is ∑p. right The total pressure from the pressure sensor on the right, ∑p total Total pressure; longitudinal pressure gradient ΔP is the pressure difference between the upper and lower pressure sensors, and Δh is the vertical distance between the two pressure sensors; lateral pressure fluctuation p i Let be the pressure value of the i-th sensor, μ be the average pressure of all sensors, and N be the number of sensors. Wavelet transform is used to analyze the pressure data, decomposing the signal into different frequency bands to identify long-term poor posture patterns.

[0052] When extracting features from spinal curvature data, a fiber optic sensor, specifically a fiber Bragg grating (FBG) sensor, is used. An FBG fiber demodulation algorithm is employed to calculate the curvature and spinal curvature angle at various locations along the spine. (The curvature at different locations of the spine is then described.) Δλ B λ represents the drift of the central Bragg wavelength of the FBG, reflecting the strain effect caused by fiber deformation; B Let λ be the initial Bragg wavelength of the FBG, and S be the strain sensitivity factor, which depends on the fiber material and marking method. The spinal curvature angle α = ∫κ(s)ds, where s is the arc length coordinate along the length of the spine, and κ(s) is the local curvature of the spine at different locations. A three-dimensional curve of the spine is constructed based on the curvature-torsion model. The local coordinate system of the spine is solved using the Frenet-Serret formula to analyze the spinal morphology. Since video skeleton features and pressure distribution features are complementary, a self-attention mechanism is used for feature fusion to enhance the correlation between multimodal information. Attention weights for different modal features are calculated, and weighted fusion is performed. The MLP (Multilayer Perceptron) uses a two-layer fully connected network with ReLU activation function to extract cross-modal information.

[0053] The specific steps for designing and training a deep learning model are as follows:

[0054] We employ MTL-GCN (Multi-Task Graph Convolutional Network) to model the relationships of the human skeleton through graph neural networks and combine it with cross-modal feature learning to achieve high-precision pose classification and spinal morphology prediction.

[0055] A composite loss function is used to optimize classification and regression tasks and prevent mode collapse.

[0056] L=λ1L cls +λ2L reg +λ3L ortho

[0057] λ1 is the weight parameter of the classification loss, which controls the impact of the classification loss on the overall optimization. cls For classification loss, Focal Loss is used to improve classification accuracy. λ² is a weighting parameter for the regression loss, controlling its contribution to the regression loss. regFor the regression loss, Huber Loss (δ = 1.0) is used to improve the robustness of spinal curvature regression. λ3 is the weighting parameter of the orthogonal loss, controlling the influence of the orthogonal loss. ortho Orthogonal loss is used to prevent modality collapse, ensuring that different modalities (such as sEMG signals and visual features) remain independent in the feature space and avoiding information loss due to redundancy. Through orthogonality constraints, features from different modalities are forced to maintain mathematical orthogonality, thereby improving the effectiveness of multimodal fusion.

[0058] Three sub-models are designed, corresponding to dynamic video, pressure sensor, and fiber optic sensor, respectively. The three sub-models output the confidence distribution of the pose category, and decision fusion is performed through DS synthesis rules.

[0059] The model training scheme is as follows

[0060] In terms of data, the dataset includes posture data from 200 adolescents aged 10-16, covering seven typical incorrect and normal postures. The definitions of incorrect posture categories are based on biomechanical and clinical standards, such as a forward head posture angle greater than 15°, a Cobb angle exceeding 10° in scoliosis, and a sacral angle less than 25° due to posterior pelvic tilt. Data annotation employs a multimodal fusion approach, combining video keypoint detection, pressure distribution map analysis, and fiber optic curvature signal calculation to ensure accurate posture category classification.

[0061] To improve the model's generalization ability, data augmentation was performed on different modal data. For video data, a random occlusion strategy was used to simulate viewpoint occlusion, and Gaussian noise (σ = 0.01) was added to enhance robustness. Temporal interpolation was also performed to simulate frame rate variations. For stress data, finite element analysis was used to simulate elastic deformation caused by different body shapes, and random sensor failure simulation was introduced to improve system robustness. For fiber optic data, curvature perturbation (±5%) was added to simulate measurement errors.

[0062] The training strategy employs a phased training approach to ensure the model's stability and effectiveness when fusing multimodal information. The first phase involves single-modal pre-training, where the video branch is initialized using HRNet to improve keypoint detection accuracy. The second phase fixes the backbone networks for feature extraction from each modality, optimizing only the fusion layer to ensure cross-modal information alignment. The third phase involves end-to-end fine-tuning, adjusting the learning rate to 1e. -4 To balance convergence speed and optimization accuracy.

[0063] For model compression, knowledge distillation and quantization techniques are employed to improve deployment efficiency. During knowledge distillation, the teacher model is a complete MTL-GCN (18.7M parameters), while the student model is a lightweight LightGCN (4.2M parameters). The distillation temperature is set to 3, and the Kullback-Leibler (KL) divergence loss function is used for training to ensure the student model effectively inherits the knowledge from the teacher model. Furthermore, in the quantization deployment stage, the model is converted from FP32 to INT8, the quantization calibration set contains 200 samples, and it is finally deployed to the TensorRT engine to improve inference speed and energy efficiency.

[0064] For classification tasks, classification accuracy is used as the primary metric, while for spinal posture regression, the mean squared error (MAE) of spinal curvature is used as the evaluation metric. End-to-end latency is also considered, with the goal of achieving a real-time performance of less than 200ms for the entire system. Furthermore, model parameter count and power consumption (mJ / inference) are used as auxiliary evaluation metrics to ensure the system's suitability for practical deployment. OpenPose+Random Forest, MediaPipe+LSTM, and unimodal GCN are selected as baseline models for comparison to verify the superiority of the proposed method. Simultaneously, ablation experiments are conducted to analyze the contributions of different modules.

[0065] The specific evaluation steps are as follows: input the weighted and fused features into the deep learning model and output the evaluation results. In the final decision-making stage, the classification results of different modalities may have uncertainties. Dempster-Shafer (DS) evidence theory is used to fuse multimodal decisions to improve the credibility of the final prediction.

[0066] The system provides user feedback on posture analysis results in a user-friendly and intuitive manner, offering specific suggestions and long-term trend tracking to help users effectively improve their postural health. In terms of data visualization, the module dynamically displays the user's real-time posture using a 3D human body model or skeletal point projection, visually marking areas of poor posture with color coding (e.g., green for normal, red for abnormal). It also displays pressure distribution through heatmaps, highlighting areas of excessive pressure, and tracks long-term posture changes using time-series curves. Real-time display of spinal curvature changes is also included. Furthermore, the historical data review function supports daily, weekly, or monthly posture score trend charts, providing users with a comprehensive understanding of the improvement results. Regarding user operation, the system automatically generates personalized suggestions based on the assessment results, such as "adjust sitting posture to reduce back pressure," presented through various formats including text prompts, voice broadcasts, or video demonstrations. Users can also set improvement goals (e.g., "maintain good posture for more than 80% of the time"), and the system tracks goal achievement and provides feedback. To enhance the user experience, the system provides reminders via pop-ups, sounds, or vibrations, allowing users to customize the frequency and mode of reminders according to their individual needs.

[0067] A multimodal posture assessment method also includes a correction step, which follows the result assessment step. For real-time posture guidance, the system provides real-time voice feedback to remind users, such as prompting "Please sit up straight," and supports multiple languages ​​to meet the needs of users with different language backgrounds. Simultaneously, the interface displays motion correction animations, such as detailed demonstrations of how to adjust shoulder or waist positions, and provides relevant educational information to help users understand the harm of poor posture and the importance of correction, thus enhancing their motivation to improve. Furthermore, the system incorporates vibration alerts in key areas (such as seats or wearable devices). When poor posture is detected, a slight vibration is triggered to gently remind the user to adjust their posture promptly.

[0068] Regarding device adjustments, the correction execution module works in conjunction with smart hardware. For example, through the electric adjustment function of the smart chair, the height, backrest angle, and lumbar support are automatically adjusted to help users achieve an ideal ergonomic sitting posture. Furthermore, the module supports the integration of wearable correction devices, such as smart correction straps or pressure-sensing garments, which can monitor the user's posture in real time and provide physical support or corrective force as needed. The support intensity can be dynamically adjusted according to changes in the user's movements, ensuring comfort and effectiveness during the correction process.

[0069] To enhance users' initiative and health awareness, the system also provides personalized exercise suggestions and guidance. When users maintain the same posture for an extended period, the system will send reminders suggesting appropriate stretching exercises to relieve muscle tension. Combined with a mobile phone or wearable device, users can learn correct stretching and exercise techniques through guided videos. Through the combination of these functions, the corrective execution module provides comprehensive support from real-time feedback to physical intervention, helping users gradually develop good posture habits and maintain long-term health.

[0070] This application's multimodal pose assessment method utilizes dynamic video from a camera, pressure distribution data from a pressure sensor, and spinal curvature data from a fiber optic sensor. Based on multi-dimensional data analysis, a hierarchical fusion architecture is employed, combining data-level, feature-level, and decision-level fusion to enhance the complementarity of multimodal information and improve the robustness of pose recognition. At the data level, signals from the pressure sensor and the fiber Bragg grating (FBG) sensor are fused in a unified coordinate system to enhance the spatial consistency of spinal curvature information. A vest coordinate system is adopted, with the seventh cervical vertebra (C7) as the origin, to establish a local spinal coordinate system. The spinal curvature detected by the fiber optic sensor and the longitudinal pressure gradient detected by the pressure sensor are weighted and fused. Since video skeleton features and pressure distribution features are complementary, a self-attention mechanism is used for feature fusion to enhance the correlation between multimodal information. Attention weights for different modal features are calculated and weighted fusion is performed. The MLP (Multilayer Perceptron) uses a two-layer fully connected network with ReLU activation function to extract cross-modal information. In the final decision-making stage, the classification results of different modalities may be uncertain. Dempster-Shafer (DS) evidence theory is used for multimodal decision fusion to improve the reliability of the final prediction. Three sub-models (video, pressure, and fiber optic) output confidence distributions for pose categories, and decision fusion is performed using DS synthesis rules. Bayesian optimization is used to adjust the confidence threshold of the fusion strategy to improve fusion accuracy. Through these steps, human pose can be accurately assessed in different scenarios.

[0071] This application also relates to a multimodal pose evaluation device for implementing the above-described multimodal pose evaluation method. The multimodal pose evaluation device includes...

[0072] A camera captures dynamic video of the human body to obtain video data;

[0073] Pressure sensors are installed on the seat cushion and vest to collect posture-related pressure distribution data.

[0074] Fiber optic sensor, installed on a vest, fits against the spine to measure spinal curvature data;

[0075] The processor analyzes the acquired video data, pressure distribution data, and bending data to assess the user's pose.

[0076] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These are all equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of these fall within the protection scope of the present invention.

Claims

1. A multimodal pose evaluation method, characterized in that, Includes the following steps: Data acquisition: The system acquires dynamic video of the human body through a camera to obtain video data; pressure sensors installed on the seat cushion and vest collect posture-related pressure distribution data; and fiber optic sensors installed on the vest are used to measure spinal curvature data by fitting the spine. Data fusion: Video data and pressure distribution data are time-aligned using multiple spline interpolation, and bending data is downsampled to match the pressure distribution data, so that the time resolution of the data from the camera, pressure sensor and fiber optic sensor is unified; A global attitude coordinate system is established, and the pressure sensor mesh is mapped to the skeleton key point topology using the affine transformation method. The force situation in the corresponding area is calculated, and the alignment error between the coordinate systems of the camera, pressure sensor and fiber optic sensor is optimized to improve the spatial matching accuracy. Feature extraction: Deep neural networks are used to estimate the pose of video data, extract the coordinates of key human body points, and calculate the trunk tilt angle, head offset, and motion frequency; the coordinates of the pressure center, asymmetry index, longitudinal pressure gradient, and lateral pressure fluctuation are calculated based on pressure distribution data; the curvature of the spine at various locations and the spinal curvature angle are calculated based on the spinal curvature data. Calculate the attention weights for different extracted features and perform weighted fusion of the features; Design and train a deep learning model: Construct a deep learning model using a composite loss function, optimize classification and regression tasks, and prevent modal collapse; Collect multiple pose data including incorrect and normal poses, and label the data using a multimodal fusion method by combining video key point detection, pressure distribution map analysis, and fiber curvature signal calculation; Train the model using a staged training method. Result evaluation: Input the weighted and fused features into the deep learning model and output the evaluation results.

2. The multimodal pose evaluation method according to claim 1, characterized in that: The data fusion step also includes employing a time-series window sliding mechanism to perform linear interpolation on the video data, the pressure distribution data, and the spinal curvature data within a short time scale, in order to reduce abrupt errors.

3. The multimodal pose evaluation method according to claim 1, characterized in that: In the data fusion step, the establishment of the global pose coordinate system is specifically as follows: with the 7th cervical vertebra as the origin, the spine direction as the Z-axis, the shoulder direction as the X-axis, and the front-back direction as the Y-axis.

4. The multimodal pose evaluation method according to claim 1, characterized in that: In the feature extraction step, the torso tilt angle Here are the coordinates of key points on the left hip. Coordinates of key points on the right hip; head offset. P nose Let's say it's the coordinate point of the tip of the nose. The coordinates of the shoulder center point are given; the motion frequency is obtained by analyzing the displacement spectrum of key shoulder points to extract the frequencies of the main motion modes.

5. The multimodal pose evaluation method according to claim 1, characterized in that: In the feature extraction step, the pressure center coordinates p i x represents the pressure value measured by the i-th pressure sensor; i y i Indicates the coordinates of the sensor; Asymmetric index In the formula ∑p left The total pressure from the left-side pressure sensor is ∑p. right The total pressure from the pressure sensor on the right, ∑p total Total pressure; longitudinal pressure gradient ΔP is the pressure difference between the upper and lower pressure sensors, and Δh is the vertical distance between the two pressure sensors; lateral pressure fluctuation p i Let be the pressure value of the i-th sensor, μ be the average pressure of all sensors, and N be the number of sensors.

6. The multimodal pose evaluation method according to claim 1, characterized in that: In the feature extraction step, the curvature of the spine at various locations Δλ B λ is the amount of shift in the center Bragg wavelength of the FBG. B S is the initial Bragg wavelength of the FBG, and S is the strain sensitivity factor; the spinal curvature angle α = ∫κ(s)ds, where s is the arc length coordinate along the length of the spine, and κ(s) is the local curvature of the spine at different positions.

7. The multimodal pose evaluation method according to claim 1, characterized in that: In the step of designing and training the deep learning model, three sub-models are designed, corresponding to dynamic video, pressure sensor and fiber optic sensor respectively. The three sub-models output the confidence distribution of the pose category respectively, and the decision fusion is performed by DS synthesis rules.

8. The multimodal pose evaluation method according to claim 1, characterized in that: In the steps of designing and training the deep learning model, the composite loss function is L = λ1L cls +λ2L reg +λ3L ortho In the formula, λ1 is the weight parameter of the classification loss, and L cls For classification loss, λ² is the weighting parameter for regression loss, and L... reg For regression loss, λ3 is the weighting parameter of orthogonal loss, and L... ortho This is an orthogonal loss.

9. The multimodal pose evaluation method according to claim 1, characterized in that: In the design and training of the deep learning model, the step of training the model using a phased training method specifically involves: the first phase of single-modal pre-training, wherein the video branch is initialized using HRNet to improve the accuracy of keypoint detection; the second phase of fixing the backbone network for feature extraction of each modality and optimizing only the fusion layer to ensure cross-modal information alignment. The third stage involves end-to-end fine-tuning, adjusting the learning rate to 1e. -4 To balance convergence speed and optimization accuracy.

10. The multimodal pose evaluation method according to claim 1, characterized in that: In the result evaluation step, the user's real-time posture is dynamically displayed through a 3D human body model or skeletal point projection. The poor posture is marked with color coding, the pressure distribution is displayed through a heat map, highlighting areas with excessive pressure, and the trend of long-term posture changes is tracked by combining time series curves, and the change curve of spinal curvature is displayed in real time.

11. A multimodal pose evaluation device for implementing the multimodal pose evaluation method as described in any one of claims 1-10, characterized in that: The multimodal pose evaluation device includes A camera captures dynamic video of the human body to obtain video data; A pressure sensor is installed on the seat cushion and vest to collect posture-related pressure distribution data. An optical fiber sensor is mounted on a vest and is fitted to the spine to measure spinal curvature data. The processor analyzes the acquired video data, pressure distribution data, and bending data to assess the user's pose.

Images