Multi-view cross-modal three-dimensional human pose estimation method and system for rehabilitation scenarios

Abstract

The application provides a multi-view cross-modal three-dimensional human posture estimation method and system for rehabilitation scenes, belonging to the field of artificial intelligence rehabilitation medicine. The system synchronously collects multi-modal data by three cameras distributed at 120 degrees and five inertial sensors worn on key parts of the human body, inputs the aligned data into a feature extraction and fusion unit through a time synchronization unit, extracts visual and inertial features respectively, and performs cross-view and cross-modal fusion. With the aid of a self-supervised contrast learning module, a loss function is constructed for cross-view, cross-modal and temporal consistency, realizing network training without labeled data. Further, a three-dimensional posture decoding module is used to output joint coordinates, and physiological and geometric constraints are introduced for post-processing optimization, improving the accuracy and rationality of the posture estimation. The application overcomes the problems of difficult data labeling, multiple occlusions and high real-time requirements in rehabilitation scenes, providing high-precision and robust three-dimensional posture monitoring support for rehabilitation training.

Description

Technical Field

[0001] This invention relates to a multi-view, cross-modal, three-dimensional human pose estimation method and system for rehabilitation scenarios, belonging to the fields of computer vision, artificial intelligence, and rehabilitation medicine, and is particularly suitable for multi-view, cross-modal, three-dimensional human pose estimation for rehabilitation scenarios. Background Technology

[0002] 3D human pose estimation aims to determine the accurate positions of various joints in 3D space. Traditional human pose estimation methods face numerous technical challenges in medical rehabilitation scenarios, including the adaptability of general pose estimation methods to specific lighting conditions, spatial layout, and medical equipment in rehabilitation training rooms, as well as the insufficient accuracy of general models due to the limited joint mobility, slow movement speed, and poor balance of rehabilitation patients. Furthermore, patients wearing loose hospital gowns or braces and fixation devices can affect visual feature extraction, making it difficult to obtain large-scale 3D pose annotation data in medical settings, especially for specific patient groups. Rehabilitation training requires real-time feedback, but traditional high-precision methods are often computationally complex and cannot meet real-time requirements. Common problems in rehabilitation training, such as self-occlusion and device occlusion, also make single-view methods insufficiently robust. Existing methods often rely on single visual data and do not fully utilize motion information provided by wearable sensors.

[0003] In existing technologies, deep learning-based human pose estimation methods are mainly divided into two categories: one is monocular vision-based methods, which are computationally efficient but have limited accuracy, especially in medical rehabilitation scenarios, where the accuracy is difficult to meet requirements due to occlusion and viewing angle limitations; the other is multi-view vision-based methods, which have higher accuracy but expensive equipment and strict requirements for scene setup, making them difficult to popularize in ordinary medical institutions.

[0004] In recent years, the development of self-supervised learning techniques has provided new approaches for model training on unlabeled data. Contrastive learning, as an important branch of self-supervised learning, has achieved significant results in multiple visual tasks by constructing pairs of positive and negative samples to learn feature representations. However, existing contrastive learning methods are mostly designed for general computer vision tasks and lack targeted optimization for the specific needs of medical rehabilitation scenarios.

[0005] Furthermore, multimodal fusion technology offers a new approach to improving attitude estimation accuracy. Inertial measurement units (IMUs) can provide accurate local motion information, complementing visual information. However, existing multimodal fusion methods often employ simple feature stitching or post-fusion strategies, failing to fully explore the intrinsic correlations between different modalities.

[0006] Therefore, in medical rehabilitation scenarios, there is a need for an efficient, accurate, and low-cost method for estimating human posture that can adapt to the special characteristics of the medical environment and overcome the limitations of existing technologies. Summary of the Invention

[0007] This invention aims to overcome the limitations of existing human pose estimation technologies in medical rehabilitation scenarios. It constructs a self-supervised contrastive learning framework through a cross-modal fusion architecture combining multi-view vision and inertial sensing to achieve high-precision 3D pose estimation without labeled data. This invention innovatively uses physiological and geometric constraint signals as the core supervisory signals for self-supervised learning, solving the critical problem of scarce 3D pose annotation data in medical scenarios.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A multi-view, cross-modal, three-dimensional human pose estimation system for rehabilitation scenarios is provided. The system consists of a data acquisition module, a time synchronization unit, a feature extraction and fusion unit, a self-supervised learning module, a three-dimensional pose decoding module, a pose post-processing unit, and a data output module connected in series.

[0010] The data acquisition module includes a multi-view vision acquisition module and an inertial sensing module, used to acquire multimodal data such as image data and IMU time-series motion data;

[0011] The time synchronization unit is used to synchronize and align multimodal data in time, and send the synchronized multimodal data to the feature extraction and fusion unit;

[0012] The feature extraction and fusion unit is used to extract and fuse visual features and IMU motion features to obtain high-dimensional fused features. It includes a visual feature extraction module, an IMU feature extraction module, a three-view feature fusion module, an IMU feature stitching module, and a feature fusion module.

[0013] The self-supervised learning module constructs loss functions that are consistent across perspectives, modalities, and time sequences to build a self-supervised contrastive learning framework, which in turn feeds back to optimize the parameters of the corresponding feature networks in the feature extraction and fusion unit.

[0014] The aforementioned 3D pose decoding module is a multilayer perceptual network (MLP), which decodes high-dimensional fused features to generate 3D pose coordinates of key human joints.

[0015] The posture post-processing unit is a constrained feedback adjustment unit. It constructs a loss function using the physiological and geometric constraints corresponding to the three-dimensional coordinates of key human joints, and then optimizes the parameters of the feature network in the feature extraction and fusion unit.

[0016] The data output module is used to output and display the optimized three-dimensional coordinates of key human joints.

[0017] The multi-view visual acquisition module consists of three fixed cameras arranged in a 120-degree circular array, distributed around the periphery of the monitoring area for rehabilitation patients. The inertial sensing module comprises five triaxial accelerometers, triaxial angular velocity sensors, and triaxial magnetometers, respectively worn on the left wrist, right wrist, left ankle, right ankle, and the fifth lumbar vertebra (L5). The multi-view visual acquisition module and the inertial sensing module are connected to a time synchronization unit.

[0018] The visual feature extraction module is a convolutional neural network (CNN), consisting of three modules, each corresponding to data from three multi-view visual acquisition modules; the IMU feature extraction module is a 1D-CNN, consisting of five modules, each corresponding to data from five inertial sensing modules; the three-view feature fusion module uses a multi-head attention mechanism, with its input connected to the three independent visual feature extraction modules and its output connected to the feature fusion module; the IMU feature stitching module is a Concat connection layer, with its input connected to the five independent IMU feature extraction modules and its output connected to the feature fusion module; the feature fusion module is a multilayer perceptron.

[0019] A multi-view, cross-modal 3D human pose estimation method for rehabilitation scenarios includes the following steps:

[0020] S1: Data Acquisition: Multimodal data, including three-view image data and five IMU time-series motion data, are simultaneously acquired through three fixed cameras arranged in a 120-degree circular array and five inertial sensors worn on key parts of the human body.

[0021] S2: Time synchronization: Hardware-level time synchronization is performed on the acquired multi-view video stream and inertial measurement unit data to ensure that all data are aligned on the time axis, and the synchronized data is sent to the feature extraction and fusion unit;

[0022] S3: Feature Extraction and Fusion: The feature extraction and fusion unit performs feature extraction and fusion on the three-view image data and the time-series motion data of 5 IMUs in the multimodal data to obtain high-dimensional fused features;

[0023] S4: Self-supervised contrastive learning optimization: By constructing a loss function that is consistent across perspectives, modalities, and time sequences, a self-supervised contrastive learning framework is built, which then provides feedback to optimize the network parameters in the feature extraction and fusion units;

[0024] S5: 3D pose decoding: Using a multilayer perceptron, the high-dimensional fusion features extracted by the feature extraction and fusion units are decoded into the 3D coordinates of key human joints;

[0025] S6: Post-processing optimization: Based on human physiological and geometric constraints, a loss function is constructed to further optimize the network parameters in the feature extraction and fusion units;

[0026] S7: Outputs optimized real-time 3D coordinates of key human joints to aid in the analysis of patient rehabilitation.

[0027] The key human joints mentioned include: the tip of the nose, left / right eye, left / right ear, left / right shoulder, left / right elbow, left / right wrist, left / right hip, left / right knee, and left / right ankle, totaling 17 joint points.

[0028] Furthermore, the working principle of the network architecture in step S3 is as follows:

[0029] S301: Receives three-view image data and timing motion data from 5 IMUs;

[0030] S302: The visual feature extraction module extracts visual feature vectors from the image data of each viewpoint; the IMU feature extraction module extracts IMU feature vectors from the temporal motion data of each IMU.

[0031] S303: The motion feature vector is obtained by concatenating the five IMU feature vectors using the IMU feature concatenation module;

[0032] S304: The three-view feature fusion module is used to fuse the three visual feature vectors to obtain a multi-view visual feature vector;

[0033] S305: The feature fusion module fuses multi-view visual feature vectors and motion feature vectors to obtain high-dimensional fused features.

[0034] Furthermore, step S4 specifically includes:

[0035] S401: Constructing datasets for positive and negative samples: (a) For cross-view contrastive learning: combine three visual feature vectors at the same time point in pairs as a positive sample, and combine any two visual feature vectors at different times as a negative sample; (b) For cross-modal contrastive learning: combine multi-view visual feature vectors and motion feature vectors at the same time point as a positive sample, and combine multi-view visual feature vectors and motion feature vectors at different times as a negative sample; (c) For temporal consistency learning: combine two high-dimensional fusion features at adjacent times as a positive sample, and combine two high-dimensional fusion features at non-adjacent times as a negative sample;

[0036] S402: Based on the corresponding dataset, a joint loss function is constructed using the cross-view contrast loss function, the cross-modal contrast loss function, and the temporal consistency loss function to train the network parameters in the visual feature extraction module, the IMU feature extraction module, the three-view feature fusion module, and the feature fusion module.

[0037] The cross-view contrast loss function aims to bring visual features captured by different cameras at the same moment closer to each other in the feature space, while features at different moments are further apart; the cross-modal contrast loss function aims to narrow the representation distance between visual features and inertial motion features at the same moment, achieving complementary alignment of visual and motion information; the temporal consistency contrast loss function aims to ensure that the changes in human posture between consecutive time frames are smooth and reasonable, making the fused feature representations at adjacent moments similar.

[0038] Furthermore, the cross-view contrast loss function is: ;in, For the first Visual feature vectors of each visual feature extraction module and the Visual feature vectors of each visual feature extraction module Cosine similarity; Corresponding to cross-perspective comparative learning of positive samples Corresponding cross-perspective comparative learning of negative samples;

[0039] The cross-modal contrast loss function is: ;in, It is a norm 2. To artificially set an order-of-magnitude balance factor; ; Multi-view visual feature vectors With motion feature vector Cosine similarity; To learn the positive sample set for cross-modal contrastive learning; To learn the negative sample set for cross-modal contrastive learning; The temperature coefficient is a hyperparameter.

[0040] The time-series consistency comparison loss function is: ;in, for Moment-time high-dimensional fusion features and Moment-time high-dimensional fusion features The cosine similarity is used to compare the cosine similarity of positive samples with the time sequence consistency. Cosine similarity of negative samples is used to compare temporal consistency.

[0041] The physiological constraint loss function ensures that the estimated 3D pose conforms to basic human anatomy by penalizing deviations from the average human bone length and joint range of motion. The geometric constraint loss function ensures that the 3D estimation result is geometrically consistent with the original 2D visual observation by minimizing the error of reprojecting the 3D joint coordinates onto the 2D plane of each camera.

[0042] Furthermore, the physiological constraint loss function in step S6 is:

[0043] ;

[0044] in, It is an assembly of human bones. A set of human joint angles To estimate the first [number] joint using the three-dimensional coordinates of key human joints The length of a bone, This is based on the average bone length calculated from population statistics. To estimate the first [number] joint using the three-dimensional coordinates of key human joints The degree of each joint angle, , For the first The maximum and minimum range of motion of each joint angle; To artificially set an order-of-magnitude balance factor;

[0045] The geometric constraint loss function is: ;in, It is a collection of key joints in the human body. To use the camera The projection obtained Two-dimensional coordinates of key joints in an individual's body For the first Three-dimensional coordinates of key joints in an individual's body Let be the projection function for the c-th camera.

[0046] Preferably, the self-supervised contrastive learning optimization in step S4 and the post-processing optimization in step S6 are both in offline training mode, that is, multimodal data of rehabilitation patients are collected in advance for a period of time to train the network parameters; during online real-time analysis, steps S4 and S6 are called every once in a while to fine-tune the network parameters.

[0047] Preferably, the self-supervised contrastive learning optimization in step S4 and the post-processing optimization in step S6 are trained using a joint loss function, that is, all loss functions are weighted and combined to form a composite objective function. This allows for the optimization of network parameters in the feature extraction and fusion unit to be completed in just one training session, thereby improving training efficiency.

[0048] The beneficial effects of this invention are as follows: This invention provides a multi-view, cross-modal, three-dimensional human posture estimation method and system for rehabilitation scenarios. By fusing multi-view visual data and inertial sensor data, combined with a self-supervised contrastive learning framework and physiological and geometric constraints, it achieves high-precision and robust estimation of rehabilitation patient postures without the need for large-scale labeled data. This method can adapt to the special characteristics of medical rehabilitation scenarios, overcome the limitations of existing technologies, improve the accuracy and robustness of posture estimation, and provide reliable technical support for posture monitoring and assessment during rehabilitation training. Attached Figure Description

[0049] To illustrate the objectives and technical solutions of this invention, the following figures are provided:

[0050] Figure 1 This is an architecture diagram of a multi-view cross-modal 3D human pose estimation system for rehabilitation scenarios according to the present invention; in which, solid arrows indicate data flow and dashed arrows indicate training direction.

[0051] Figure 2 This is a flowchart of a multi-view, cross-modal, three-dimensional human pose estimation method for rehabilitation scenarios according to the present invention;

[0052] Figure 3 This is a schematic diagram of the joint training mode combining offline and online training in Embodiment 3 of the present invention;

[0053] Figure 4 This is a schematic diagram of the electronic device in Embodiment 4 of the present invention. Detailed Implementation

[0054] Example 1: In rehabilitation assessment after knee replacement surgery, traditional methods rely on therapists' experience and two-dimensional video recordings, lacking precise and quantitative three-dimensional kinematic data. This leads to strong subjectivity in rehabilitation effect assessment, making it difficult to conduct refined progress tracking and program adjustments. This example proposes "a multi-view, cross-modal, three-dimensional human posture estimation system for rehabilitation scenarios," aiming to demonstrate how the system can automatically and continuously collect and analyze patient posture data in a real rehabilitation scenario, providing data support for objective quantitative assessment.

[0055] Combination Figure 1 This system, based on a cross-modal fusion architecture of multi-view vision and inertial sensing, achieves high-precision, real-time 3D human posture estimation without manual annotation through self-supervised learning and physical constraint optimization, providing reliable data support for the quantitative assessment of rehabilitation training. The system consists of a data acquisition module 1, a time synchronization unit 2, a feature extraction and fusion unit 3, a self-supervised learning module 4, a 3D posture decoding module 5, a posture post-processing unit 6, and a data output module 7, connected in series.

[0056] The data acquisition module 1 includes a multi-view vision acquisition module 11 and an inertial sensing module 12, used to acquire multimodal data such as image data and IMU time-series motion data.

[0057] The multi-view visual acquisition module 11 contains three high-definition cameras (1920×1080 resolution) arranged in a 120° circular array around the training area, approximately 2.5 meters from the center of the patient and 1.2 meters high, and has been pre-calibrated. The inertial sensing module 12 includes a three-axis accelerometer, a three-axis angular velocity sensor, and a three-axis magnetometer, totaling five, and is worn on the left wrist, right wrist, left ankle, right ankle, and the fifth lumbar vertebra (L5). The multi-view visual acquisition module 11 and the inertial sensing module 12 are respectively connected to the time synchronization unit 2.

[0058] The time synchronization unit 2 ensures microsecond-level time alignment between the video stream (e.g., 30 FPS) and IMU data (e.g., 100 Hz) through hardware triggering, and sends the synchronized multimodal data to the feature extraction and fusion unit 3.

[0059] The feature extraction and fusion unit 3 is used to extract and fuse visual features and IMU motion features to obtain high-dimensional fused features. It includes a visual feature extraction module 31, an IMU feature extraction module 32, a three-view feature fusion module 33, an IMU feature stitching module 34, and a feature fusion module 35.

[0060] The visual feature extraction module 31 uses three independent pre-trained ResNet-50 networks to process the three-view images respectively. Its input image resolution is 224×224 pixels. The original 1920×1080 image captured by the camera can be cropped and scaled through data preprocessing to adapt to the network input size. Specifically, the ResNet-50 structure is as follows: the input layer receives a 224×224 three-channel RGB image; then it passes through a convolutional layer with a 7×7 kernel and a stride of 2, outputting 64-channel features, followed by batch normalization and ReLU activation; then it passes through a max pooling layer with a 3×3 kernel and a stride of 2; next, it passes through four residual block groups in sequence: the first group contains 3 residual blocks, outputting 256 channels; the second group contains 4 residual blocks, outputting 512 channels; the third group contains 6 residual blocks, outputting 1024 channels; and the fourth group contains 3 residual blocks, outputting 2048 channels; then, a global average pooling layer compresses the feature map to 1×1×2048; finally, a fully connected layer maps the 2048-dimensional features into a 512-dimensional visual feature vector. The ResNet-50 network structure is identical for each viewpoint, and the weights are trained independently.

[0061] The IMU feature extraction module 32 uses five independent 1D-CNN networks to process the time-series data of the five inertial sensors 12. Each inertial sensor 12 acquires 9-axis data (three-axis acceleration, three-axis angular velocity, and three-axis magnetometer) at a sampling frequency of 100Hz. The IMU feature extraction module 32 adopts a 1D-CNN structure, specifically configured as follows: the input layer receives 9 channels of 9×30-dimensional temporal data with a length of T=30; then it passes through a first convolutional layer with a kernel size of 3, an output channel size of 32, and a stride of 1, followed by batch normalization and ReLU activation; then it passes through a max pooling layer with a kernel size of 2 and a stride of 2; next, it passes through a second convolutional layer with a kernel size of 3, an output channel size of 64, and a stride of 1, again followed by batch normalization and ReLU activation, and then max pooling again; then it passes through a third convolutional layer with a kernel size of 3, an output channel size of 128, and a stride of 1, followed by batch normalization and ReLU activation; finally, it passes through a global average pooling layer to compress the features into a 128-dimensional vector; finally, it outputs a 128-dimensional IMU feature vector through a fully connected layer. Each 1D-CNN network has the same structure, and the weights are trained independently.

[0062] The three-view feature fusion module 33 is a multi-head attention mechanism. Its input end is connected to three independent visual feature extraction modules 31, and its output end is connected to the feature fusion module 35. It performs weighted fusion of the three 512-dimensional visual feature vectors and outputs a 512-dimensional multi-view visual feature vector.

[0063] The IMU feature stitching module 34 is a Concat connection layer. Its input end is connected to five independent IMU feature extraction modules 32, and its output end is connected to the feature fusion module 35, which stitches the five 128-dimensional vectors into a 640-dimensional motion feature vector.

[0064] The feature fusion module 35 is a multilayer perceptron, whose input is a concatenated 512-dimensional visual feature vector and a 640-dimensional motion feature vector, totaling 1152 dimensions. This MLP contains two hidden layers, each of which is a fully connected structure with an output dimension of 1024. It uses the ReLU activation function and is regularized with Dropout with a retention rate of 0.5. The output layer is a fully connected layer that outputs a 1024-dimensional high-dimensional fused feature.

[0065] The self-supervised learning module 4 constructs loss functions for cross-view, cross-modal, and temporal consistency to build a self-supervised contrastive learning framework. This framework then feeds back to optimize the parameters of the corresponding feature network in the feature extraction and fusion unit 3, driving the aforementioned feature network to learn discriminative and consistent feature representations under unlabeled data.

[0066] The 3D pose decoding module 5 employs a three-layer multilayer perceptron, with 1024-dimensional high-dimensional fusion features as input. Hidden layer 1 is a fully connected layer with an output dimension of 512, using the ReLU activation function. Hidden layer 2 is a fully connected layer with an output dimension of 256, also using the ReLU activation function. The output layer is a fully connected layer with an output dimension of 51, corresponding to the 3D coordinates of 17 joints. Furthermore, no activation function is applied. The network in this module is pre-trained, based on the mean squared error (MSE) loss function of three-dimensional coordinates, and utilizes the high-dimensional fusion features of the labeled dataset as input.

[0067] The posture post-processing unit 6 is a constrained feedback adjustment unit. It constructs a loss function using the physiological and geometric constraints corresponding to the three-dimensional coordinates of key human joints, and then optimizes the parameters of the feature network.

[0068] The data output module 7 outputs the final optimized 3D pose sequence (including timestamps, joint coordinates, and skeletal connection relationships) to the rehabilitation assessment software in real time in a standardized format (such as JSON).

[0069] Through the close collaboration between the aforementioned hardware setup and software processes, this system achieves accurate, stable, and real-time capture of the postures of rehabilitation patients.

[0070] Example 2: This example provides a "multi-view cross-modal 3D human pose estimation method for rehabilitation scenarios" that runs in conjunction with the system in Example 1. This method takes self-supervised learning as its core and effectively avoids the bottleneck of difficult and costly annotation of ground truth 3D pose data in medical rehabilitation scenarios.

[0071] Combination Figure 2 The specific steps are as follows:

[0072] S1: Data Acquisition: Multimodal data, including three-view image data and five IMU time-series motion data, are simultaneously acquired through three fixed cameras arranged in a 120-degree circular array and five inertial sensors worn on key parts of the human body.

[0073] S2: Time synchronization: Hardware-level time synchronization is performed on the acquired multi-view video stream and inertial measurement unit data to ensure that all data are aligned on the time axis, and the synchronized data is sent to the feature extraction and fusion unit 3.

[0074] S3: Feature Extraction and Fusion: Feature extraction and fusion unit 3 performs feature extraction and fusion on the three-view image data and the time-series motion data of 5 IMUs in the multimodal data to obtain high-dimensional fused features.

[0075] Images from three different perspectives are input into a pre-trained CNN (such as ResNet) to extract visual feature vectors. ( The time-series data of the five IMUs are input into a 1D-CNN to extract IMU feature vectors. ( ).

[0076] Motion characteristics are obtained through splicing. .

[0077] Fusing multi-view visual features using multi-head attention mechanisms: Finally, an MLP fusion module is used to... and Fusion as a high-dimensional fusion feature .

[0078] S4: Self-supervised contrastive learning optimization: By constructing a loss function that is consistent across perspectives, modalities, and time sequences, a self-supervised contrastive learning framework is built, which then feeds back to optimize the network parameters in feature extraction and fusion unit 3.

[0079] To achieve unsupervised training, a dataset needs to be constructed first:

[0080] S401: Constructing datasets for positive and negative samples: (a) For cross-view contrastive learning: combine three visual feature vectors at the same time point in pairs as a positive sample, and combine any two visual feature vectors at different times as a negative sample; (b) For cross-modal contrastive learning: combine multi-view visual feature vectors and motion feature vectors at the same time point as a positive sample, and combine multi-view visual feature vectors and motion feature vectors at different times as a negative sample; (c) For temporal consistency learning: combine two high-dimensional fusion features at adjacent times as a positive sample, and combine two high-dimensional fusion features at non-adjacent times as a negative sample;

[0081] S402: Based on the corresponding dataset, a total loss function is constructed using the cross-view contrast loss function, the cross-modal contrast loss function, and the temporal consistency loss function to train the network parameters in the visual feature extraction module 31, the IMU feature extraction module 32, the three-view feature fusion module 33, and the feature fusion module 35.

[0082] Then construct the loss function:

[0083] 1) Cross-view comparison: This requires that visual features from different viewpoints at the same time be close to each other in the feature space, while features at different times are far apart. The loss function is: ;in, For the first Visual feature vectors of visual feature extraction module 31 and the Visual feature vectors of visual feature extraction module 31 Cosine similarity; Corresponding to cross-perspective comparative learning of positive samples Corresponding to cross-perspective comparative learning of negative samples.

[0084] 2) Cross-modal contrast: requires visual features from multiple perspectives at the same time. With motion characteristics When they are close to each other, the cross-modal contrastive loss function is:

[0085] ;

[0086] in, It is a 2-norm. To artificially set an order-of-magnitude balance factor; ; Multi-view visual feature vectors With motion feature vector Cosine similarity; To learn the positive sample set for cross-modal contrastive learning; To learn the negative sample set for cross-modal contrastive learning; The temperature coefficient is a hyperparameter.

[0087] 3) Temporal consistency comparison: requires high-dimensional fusion features from adjacent time points. and Approximately. The loss function is:

[0088] ;in, for Moment-time high-dimensional fusion features and Moment-time high-dimensional fusion features The cosine similarity is used to compare the cosine similarity of positive samples with the time sequence consistency. The cosine similarity of negative samples is compared to ensure temporal consistency.

[0089] These losses together constitute the total loss. This drives the learning of network parameters in feature extraction and fusion unit 3. The network is trained using the Adam optimizer with a learning rate of [missing information]. The weight decays to .

[0090] S5: 3D Pose Decoding: Utilizing a multilayer perceptron to fuse high-dimensional features extracted by feature extraction and fusion unit 3. Decoded into 3D coordinates of 17 key human joints .

[0091] S6: Post-processing optimization: Based on human physiological and geometric constraints, a loss function is constructed to further optimize the network parameters in feature extraction and fusion unit 3;

[0092] The geometric constraints are achieved by minimizing the reprojection error, i.e., by using the camera calibration parameters to estimate the first... 3D coordinates of key joints in the human body Projecting back to the three image planes, the error is calculated relative to the 2D detection points (or features): ,in Let be the projection function of the c-th camera. Let c be the 2D detection point corresponding to the c-th camera.

[0093] Physiological constraints include bone length constraints and joint angle constraints: ;in, An order of magnitude balance coefficient is set artificially.

[0094] Bone length loss ,in For skeletal assembly, , For bones 3D coordinates of the two endpoints Skeletal analysis based on anthropometry statistics Average length;

[0095] Joint angle loss ,in For the set of joint angles, This represents the physiological limit of the joint's range of motion.

[0096] For example, the average upper arm length for an adult male is approximately 30-35 cm; the knee flexion angle... , Elbow flexion angle overstretching Shoulder abduction angle .

[0097] These constraints are constructed by the total optimization loss. And then backpropagation is performed to further optimize the network parameters. Among these, An order of magnitude balance coefficient is set artificially.

[0098] S7: Outputs optimized real-time 3D coordinates of key human joints to aid in the analysis of patient rehabilitation.

[0099] This method achieves high-precision, physiologically sound 3D pose estimation through the complete self-supervised learning and constraint optimization process described above.

[0100] Example 3: In order to optimize the training strategy of the method in Example 2, improve training efficiency and adapt to the model adaptation requirements in long-term rehabilitation monitoring, this example provides "an efficient method for multi-view cross-modal three-dimensional human pose estimation for rehabilitation scenarios".

[0101] Combination Figure 3 It includes the following steps:

[0102] S1: Data Acquisition: Multimodal data, including three-view image data and five IMU time-series motion data, are simultaneously acquired through three fixed cameras arranged in a 120-degree circular array and five inertial sensors worn on key parts of the human body.

[0103] S2: Time synchronization: Hardware-level time synchronization is performed on the acquired multi-view video stream and inertial measurement unit data to ensure that all data are aligned on the time axis, and the synchronized data is sent to the feature extraction and fusion unit (3).

[0104] S3: Feature extraction and fusion: The feature extraction and fusion unit (3) performs feature extraction and fusion on the three-view image data and the time-series motion data of 5 IMUs in the multimodal data to obtain high-dimensional fused features.

[0105] S4: 3D pose decoding: The high-dimensional fusion features extracted by the feature extraction and fusion unit (3) are decoded into the 3D coordinates of key human joints using a multilayer perceptual network.

[0106] S5: Network parameter training: Construct a joint loss function from cross-viewpoint, cross-modal, temporal consistency, human physiological constraints, and geometric constraints, and use offline collected data to train the network parameters in the feature extraction and fusion unit (3).

[0107] During the offline training phase, multimodal time-series data of a batch of rehabilitation patients (covering different body types and pathological stages) were pre-collected. During training, instead of a step-by-step training strategy, a unified joint loss function was constructed. ,in , , These are cross-view, cross-modal, and temporal consistency contrast losses, respectively. and The aforementioned physiological and geometric constraint losses (calculated in the same way as in Example 2) are considered. to The weights are set according to the importance of the task (e.g., they can be set to 1.0, 0.8, 0.5, 2.0, 1.5 respectively). Through a single end-to-end training process, all network parameters, such as feature extraction, fusion, and decoding, are directly optimized, greatly improving training efficiency and enabling the model to learn feature representations that satisfy multiple constraints from the very beginning.

[0108] S6: Execute steps S1~S4 using the trained network to output the three-dimensional coordinates of key human joints in real time online, which can be used to assist in the analysis of the patient's rehabilitation.

[0109] During the online real-time analysis phase, the system, once deployed in the rehabilitation room, is not static. To adapt to individualized movement patterns or potential changes in vital signs over time (such as improved joint mobility) for specific patients, this embodiment employs an optimization mechanism: while running in real-time, the system caches patient data from a recent period (e.g., the past 24 hours) using a sliding window. Every fixed period (e.g., every 24 hours), the system automatically enters "online optimization mode," utilizing the cached new data to fine-tune the model parameters again using the aforementioned joint loss function at a smaller learning rate. This process is fully automatic, occurring only when the system is idle, and does not affect real-time service. Through this strategy combining offline pre-training and online periodic optimization, this embodiment enables the system to not only possess strong initial performance but also continuously adapt and optimize over long-term use, maintaining a high-accuracy estimation capability of the current patient's posture and better serving the assessment of dynamically changing rehabilitation progress.

[0110] Example 4: For the scenario in Example 2, Figure 4 A schematic diagram of an electronic device 90 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0111] Electronic devices can also refer to various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or claimed herein.

[0112] like Figure 4As shown, the electronic device 90 includes at least one processor 91 and a memory, such as a read-only memory (ROM) 92 or a random access memory (RAM) 93, communicatively connected to the at least one processor 91. The memory stores computer programs executable by the at least one processor. The processor 91 can perform various appropriate actions and processes based on the computer program stored in the ROM 92 or loaded from storage unit 98 into the RAM 93. The RAM 93 can also store various programs and data required for the operation of the electronic device 90. The processor 91, ROM 92, and RAM 93 are interconnected via a bus 94. An input / output (I / O) interface 95 is also connected to the bus 94.

[0113] Multiple components in electronic device 90 are connected to I / O interface 95, including: input unit 96, such as keyboard, mouse, etc.; output unit 97, such as various types of displays, speakers, etc.; storage unit 98, such as disk, optical disk, etc.; and communication unit 99, such as network card, modem, wireless transceiver, etc. Communication unit 99 allows electronic device 90 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0114] Processor 91 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 91 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 91 performs the various methods and processes described above, such as a multi-view, cross-modal, three-dimensional human pose estimation method for rehabilitation scenarios.

[0115] In some embodiments, a multi-view, cross-modal, three-dimensional human pose estimation method for rehabilitation scenarios can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 98. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 90 via ROM 92 and / or communication unit 99. When the computer program is loaded into RAM 93 and executed by processor 91, one or more steps of the multi-view, cross-modal, three-dimensional human pose estimation method for rehabilitation scenarios described above can be performed. Alternatively, in other embodiments, processor 91 can be configured to perform a multi-view, cross-modal, three-dimensional human pose estimation method for rehabilitation scenarios by any other suitable means (e.g., by means of firmware).

[0116] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0117] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0118] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0120] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0121] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0122] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made to it in form and detail without departing from the scope defined by the claims of the present invention.

Claims

1. A multi-view, cross-modal, three-dimensional human pose estimation system for rehabilitation scenarios, characterized in that, It includes a data acquisition module (1), a time synchronization unit (2), a feature extraction and fusion unit (3), a self-supervised learning module (4), a three-dimensional pose decoding module (5), a pose post-processing unit (6), and a data output module (7) connected in series. The data acquisition module (1) includes a multi-view vision acquisition module (11) and an inertial sensing module (12), used to acquire multimodal data of image data and IMU time-series motion data; The time synchronization unit (2) is used to realize the time synchronization and alignment of multimodal data, and send the synchronized multimodal data to the feature extraction and fusion unit (3). The feature extraction and fusion unit (3) uses a deep learning network to extract and fuse visual features and IMU motion features to obtain high-dimensional fused features; The self-supervised learning module (4) is used to construct a loss function that is consistent across perspectives, modalities and time sequences, realize self-supervised contrastive learning, and feed back to optimize the network parameters of the feature extraction and fusion unit (3); The three-dimensional pose decoding module (5) is a multi-layer perception network that performs three-dimensional pose decoding on high-dimensional fusion features to generate three-dimensional coordinates of key human joints. The posture post-processing unit (6) is used to construct a loss function based on human physiological constraints and geometric constraints, and to optimize the network parameters of the feature extraction and fusion unit (3); The data output module (7) is used to output and display the optimized three-dimensional coordinates of key human joints; The cross-view contrast loss function aims to bring visual features captured by different cameras at the same moment closer to each other in the feature space, while features at different moments are further apart; the cross-modal contrast loss function aims to narrow the representation distance between visual features and inertial motion features at the same moment, achieving complementary alignment of visual and motion information; the temporal consistency contrast loss function aims to ensure that the changes in human posture between consecutive time frames are smooth and reasonable, making the fused feature representations at adjacent moments similar. The physiological constraint loss function ensures that the estimated 3D pose conforms to basic human anatomy by penalizing deviations from the average human bone length and joint range of motion. The geometric constraint loss function ensures that the 3D estimation result is geometrically consistent with the original 2D visual observation by minimizing the error of reprojecting the 3D joint coordinates onto the 2D plane of each camera. The key human joints mentioned include: the tip of the nose, left / right eye, left / right ear, left / right shoulder, left / right elbow, left / right wrist, left / right hip, left / right knee, and left / right ankle, totaling 17 joint points; The physiological constraint loss function is as follows: ； in, It is an assembly of human bones. A set of human joint angles To estimate the first [number] joint using the three-dimensional coordinates of key human joints The length of a bone, This is based on the average bone length calculated from population statistics. To estimate the first [number] joint using the three-dimensional coordinates of key human joints The degree of each joint angle, , For the first The maximum and minimum range of motion of each joint angle; To artificially set an order-of-magnitude balance factor; The geometric constraint loss function is: ;in, It is a collection of key joints in the human body. To use the camera The projection obtained Two-dimensional coordinates of key joints in an individual's body For the first Three-dimensional coordinates of key joints in an individual's body Let be the projection function for the c-th camera.

2. The multi-view, cross-modal, three-dimensional human pose estimation system for rehabilitation scenarios according to claim 1, characterized in that, The multi-view visual acquisition module (11) consists of three fixed cameras arranged in a 120-degree circular array, distributed around the periphery of the monitoring area for rehabilitation patients; the inertial sensing module (12) includes a three-axis accelerometer, a three-axis angular velocity sensor, and a three-axis magnetometer, totaling five, which are worn on the left wrist, right wrist, left ankle, right ankle, and fifth lumbar vertebra, respectively; the multi-view visual acquisition module (11) and the inertial sensing module (12) are respectively connected to the time synchronization unit (2).

3. The multi-view, cross-modal, three-dimensional human pose estimation system for rehabilitation scenarios according to claim 1, characterized in that, The feature extraction and fusion unit (3) includes: The visual feature extraction module (31) is a convolutional neural network (CNN) with three components, which are used to extract visual feature vectors from three viewpoints. The IMU feature extraction module (32) is a 1D-CNN with five modules, which are used to extract the IMU feature vectors of five IMU time-series motion data. The three-view feature fusion module (33) is a multi-head attention mechanism used to fuse three visual feature vectors and output multi-view visual feature vectors; the input end is connected to three independent visual feature extraction modules (31) respectively, and the output end is connected to the feature fusion module (35); The IMU feature splicing module (34) is a Concat connection layer used to splice five IMU feature vectors and output motion feature vectors; the input end is connected to five independent IMU feature extraction modules (32) respectively, and the output end is connected to the feature fusion module (35); The feature fusion module (35) is a multi-layer perceptual network used to fuse multi-view visual feature vectors and motion feature vectors to output high-dimensional fused features.

4. A method for estimating multi-view, cross-modal, three-dimensional human pose in a rehabilitation scenario, applied to the multi-view, cross-modal, three-dimensional human pose estimation system for rehabilitation scenarios as described in any one of claims 1 to 3, characterized in that: Includes the following steps: S1: Data Acquisition: Multimodal data, including three-view image data and five IMU time-series motion data, are simultaneously acquired through three fixed cameras arranged in a 120-degree circular array and five inertial sensors worn on key parts of the human body. S2: Time synchronization: Hardware-level time synchronization is performed on the acquired multi-view video stream and inertial measurement unit data to ensure that all data are aligned on the time axis, and the synchronized data is sent to the feature extraction and fusion unit (3). S3: Feature extraction and fusion: The feature extraction and fusion unit (3) performs feature extraction and fusion on the three-view image data and the time-series motion data of 5 IMUs in the multimodal data to obtain high-dimensional fused features; S4: Self-supervised contrastive learning optimization: By constructing a loss function that is consistent across perspectives, modalities and time sequences, a self-supervised contrastive learning framework is built, which then feeds back to optimize the network parameters in the feature extraction and fusion unit (3); S5: 3D pose decoding: The high-dimensional fusion features extracted by the feature extraction and fusion unit (3) are decoded into the 3D coordinates of key human joints using a multilayer perceptual network; S6: Post-processing optimization: Based on human physiological constraints and geometric constraints, a loss function is constructed to further optimize the network parameters in the feature extraction and fusion unit (3); S7: Outputs optimized real-time 3D coordinates of key human joints to aid in the analysis of patient rehabilitation.

5. The multi-view, cross-modal, three-dimensional human pose estimation method for rehabilitation scenarios according to claim 4, characterized in that, The working principle of the network architecture in step S3 is as follows: S301: The visual feature extraction module (31) extracts visual feature vectors for the image data of each viewpoint; the IMU feature extraction module (32) extracts IMU feature vectors for each IMU temporal motion data. S302: Use the IMU feature splicing module (34) to splice the five IMU feature vectors to obtain the motion feature vector; S303: The three visual feature vectors are fused using the three-view feature fusion module (33) to obtain a multi-view visual feature vector; S304: Feature fusion module (35) fuses multi-view visual feature vectors and motion feature vectors to obtain high-dimensional fused features.

6. The multi-view, cross-modal, three-dimensional human pose estimation method for rehabilitation scenarios according to claim 4, characterized in that, The specific steps of S4 are as follows: S401: Construct datasets for positive and negative samples, specifically: (a) For cross-view contrastive learning: combine three visual feature vectors at the same time point in pairs as a positive sample, and combine any two visual feature vectors at different times as a negative sample; (b) For cross-modal contrastive learning: combine multi-view visual feature vectors and motion feature vectors at the same time point as a positive sample, and combine multi-view visual feature vectors and motion feature vectors at different times as a negative sample; (c) For temporal consistency learning: combine two high-dimensional fusion features at adjacent times as a positive sample, and combine two high-dimensional fusion features at non-adjacent times as a negative sample. S402: Based on the corresponding dataset, a joint loss function is constructed using the cross-view contrast loss function, the cross-modal contrast loss function, and the temporal consistency loss function to train the network parameters in the visual feature extraction module (31), the IMU feature extraction module (32), the three-view feature fusion module (33), and the feature fusion module (35).

7. The multi-view, cross-modal, three-dimensional human pose estimation method for rehabilitation scenarios according to claim 4, characterized in that, The self-supervised contrastive learning optimization described in step S4 and the post-processing optimization described in step S6 are both offline training modes, that is, multimodal data of rehabilitation patients are collected in advance for a period of time to train the network parameters; during online real-time analysis, steps S4 and S6 are called every once in a while to fine-tune the network parameters.

8. A method for estimating multi-view, cross-modal, three-dimensional human pose in a rehabilitation scenario, applied to the multi-view, cross-modal, three-dimensional human pose estimation system for rehabilitation scenarios as described in any one of claims 1 to 3, characterized in that: By weighting and combining all loss functions to form a composite objective function, the network parameters in the feature extraction and fusion unit (3) can be optimized in just one training session, including the following steps: S1: Data Acquisition: Multimodal data, including three-view image data and five IMU time-series motion data, are simultaneously acquired through three fixed cameras arranged in a 120-degree circular array and five inertial sensors worn on key parts of the human body. S2: Time synchronization: Hardware-level time synchronization is performed on the acquired multi-view video stream and inertial measurement unit data to ensure that all data are aligned on the time axis, and the synchronized data is sent to the feature extraction and fusion unit (3). S3: Feature extraction and fusion: The feature extraction and fusion unit (3) performs feature extraction and fusion on the three-view image data and the time-series motion data of 5 IMUs in the multimodal data to obtain high-dimensional fused features; S4: 3D pose decoding: The high-dimensional fusion features extracted by the feature extraction and fusion unit (3) are decoded into the 3D coordinates of key human joints using a multilayer perceptual network; S5: Network parameter training: Construct a joint loss function from cross-viewpoint, cross-modal, temporal consistency, human physiological constraints, and geometric constraints, and use offline collected data to train the network parameters in the feature extraction and fusion unit (3); S6: Execute steps S1~S4 using the trained network to output the three-dimensional coordinates of key human joints in real time online, which can be used to assist in the analysis of the patient's rehabilitation.

9. An electronic device, characterized in that, The electronic device includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the multi-view cross-modal three-dimensional human pose estimation method for rehabilitation scenarios as described in any one of claims 4 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the multi-view cross-modal three-dimensional human pose estimation method for rehabilitation scenarios as described in any one of claims 4 to 8.

Images