Monocular vision and sparse IMU-based rehabilitation action whole body attitude estimation method and system

Abstract

The invention provides a monocular vision and sparse IMU rehabilitation action whole body posture estimation method and system, and the method comprises the steps: synchronously collecting video data and inertial data of human body rehabilitation actions through a monocular RGB camera and a plurality of IMUs, and cutting and zooming an image to a preset resolution; extracting a key point thermodynamic diagram from continuous N frames of images by using a sliding window and a residual neural network, and calculating 2D key point pixel coordinates of each frame; splicing the N frames of 2D key point pixel coordinates, the rotation matrix of the IMU and the acceleration signal into an input sequence; cross-modal time sequence modeling is carried out on an input sequence through time Transform, and after high-dimensional features are extracted, weighted average is carried out through a convolutional layer, and 3D relative key point coordinates of the last frame are output through a regression head. According to the method, by fusing monocular vision and sparse IMU cross-modal data, the problem of visual information loss caused by limb self-shielding is effectively solved, and the defect that a traditional pure vision method is insufficient in precision in rehabilitation actions is overcome.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and sensor fusion, and in particular to a method and system for estimating whole-body posture of rehabilitation movements using monocular vision and sparse IMU. Background Art

[0002] With the increasing aging of the population and the growing demand for sports rehabilitation, rehabilitation assessment technology based on human posture analysis has received increasing attention. In clinical rehabilitation scenarios, accurate whole-body posture estimation is of great significance for the evaluation of movement norms, quantification of rehabilitation progress, and personalized training guidance. In existing technologies, vision solutions based on monocular RGB cameras are widely used due to their non-invasiveness and low cost advantages. However, when processing complex rehabilitation movements (such as kneeling dynamic frog stretch and single-leg book opening movements), due to frequent limb self-occlusion, key point tracking fails and the posture estimation accuracy is significantly reduced.

[0003] To alleviate the occlusion problem, some studies have attempted to fuse inertial measurement unit (IMU) data to supplement visual information. However, traditional multimodal solutions usually require the deployment of dense sensors (such as more than 10 IMUs), which not only increases the burden on users, but also affects the natural execution of rehabilitation movements. In addition, existing methods mostly rely on single-frame data fusion or simple time series modeling, which makes it difficult to capture the continuous time series correlation of dynamic movements, resulting in jumps or lags in the estimation results, and cannot meet clinical high-precision requirements. For example, Chinese patent CN202110880873.6 discloses a real-time motion capture method based on monocular RGB images and inverse kinematics algorithms. Although it achieves lightweight deployment on mobile terminals, it does not fuse IMU data, is not robust enough in occlusion scenarios, and lacks a cross-modal time series modeling mechanism, making it difficult to adapt to the dynamic changes of complex rehabilitation movements.

[0004] Current industry research focuses on independent optimization of visual or inertial data, but research on deep fusion of multimodal signals and modeling of long-term dependencies remains unresolved. Therefore, a whole-body pose estimation method is urgently needed that can reduce the complexity of sensor deployment, effectively integrate visual and inertial data, and maintain temporal continuity to meet the urgent need for high-precision, high-availability technologies in the rehabilitation medicine field. Summary of the Invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for estimating the whole body posture of rehabilitation movements using monocular vision and sparse IMU.

[0006] According to the present invention, a method for estimating the whole body posture of rehabilitation movements using monocular vision and sparse IMU is provided, the method comprising the following steps:

[0007] Step S1: synchronously collect video data and inertial data of human rehabilitation movements through a monocular RGB camera and multiple inertial measurement units (IMUs), crop and scale the images to a preset resolution, and convert the IMU signals to the camera's three-dimensional coordinate system;

[0008] Step S2: Use sliding window and residual neural network to extract key point heat map from N consecutive frames of images and calculate the 2D key point pixel coordinates of each frame;

[0009] Step S3: Concatenate the 2D keypoint pixel coordinates of N frames with the IMU rotation matrix and acceleration signal into an input sequence, and introduce a learnable position encoding to capture temporal dependencies;

[0010] Step S4: Perform cross-modal temporal modeling on the input sequence through the temporal transformer. After extracting high-dimensional features, the convolutional layer performs weighted averaging and the regression head outputs the 3D relative key point coordinates of the last frame, with the pelvis as the root node.

[0011] Preferably, the number of frames N of the sliding window in step S2 is a dynamically adjusted value, and the range is set to 8 to 16 frames.

[0012] Preferably, the position encoding in step S3 adopts a hybrid encoding method based on a sine function and a learnable parameter.

[0013] Preferably, the number of the inertial measurement units (IMUs) is 6, which are respectively deployed at preset positions of the head, two forearms, abdomen and two calves.

[0014] Preferably, the regression head consists of two fully connected layers, the activation function adopts LeakyReLU, and the last layer outputs the offset of the 3D key point coordinates.

[0015] The present invention also provides a rehabilitation movement whole body posture estimation system based on monocular vision and sparse IMU, the system comprising the following modules:

[0016] Module M1: Synchronously collects video and inertial data of human rehabilitation movements through a monocular RGB camera and multiple inertial measurement units (IMUs), crops and scales the images to a preset resolution, and converts the IMU signals into the camera's 3D coordinate system.

[0017] Module M2: Use sliding windows and residual neural networks to extract key point heat maps from N consecutive frames of images and calculate the 2D key point pixel coordinates of each frame;

[0018] Module M3: concatenates the 2D keypoint pixel coordinates of N frames with the IMU rotation matrix and acceleration signals into an input sequence, and introduces a learnable position encoding to capture temporal dependencies;

[0019] Module M4: Performs cross-modal temporal modeling on the input sequence through the temporal transformer, extracts high-dimensional features, performs weighted averaging through the convolutional layer, and outputs the 3D relative key point coordinates of the last frame through the regression head, with the pelvis as the root node.

[0020] Preferably, the number of frames N of the sliding window in the module M2 is a dynamically adjusted value, and the range is set to 8 to 16 frames.

[0021] Preferably, the position coding in the module M3 adopts a hybrid coding method based on a sine function and a learnable parameter.

[0022] Preferably, the number of the inertial measurement units (IMUs) is 6, which are respectively deployed at preset positions of the head, two forearms, abdomen and two calves.

[0023] Preferably, the regression head consists of two fully connected layers, the activation function adopts LeakyReLU, and the last layer outputs the offset of the 3D key point coordinates.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. By fusing monocular vision with sparse cross-modal IMU data, this invention effectively solves the problem of missing visual information due to limb self-occlusion, overcoming the lack of accuracy of traditional purely visual methods in rehabilitation movements. The system can still achieve continuous and high-precision whole-body posture capture in difficult rehabilitation movements such as kneeling dynamic frog stretches and single-leg book openings.

[0026] 2. The present invention only requires 6 IMU sensors to be deployed at key parts of the human body to achieve full-body posture estimation, avoiding the cumbersomeness of densely deploying sensors in traditional IMU solutions, significantly reducing hardware costs and usage barriers, and improving system practicality;

[0027] 3. This invention models the temporal dependency between visual and IMU signals through the temporal Transformer architecture, solving the posture jump problem that is prone to occur in single-frame estimation, ensuring the continuity and stability of posture estimation in complex action sequences, and providing reliable data support for the quantitative evaluation of clinical rehabilitation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0029] Figure 1 It is a schematic diagram of the process of the present invention;

[0030] Figure 2 It is a structural schematic diagram of the present invention.

[0031] in:

[0032] First wearable IMU sensor 1 Fourth wearable IMU sensor 4

[0033] Second wearable IMU sensor 2 Fifth wearable IMU sensor 5

[0034] Third wearable IMU sensor 3 Sixth wearable IMU sensor 6

[0035] RGB Camera 7 DETAILED DESCRIPTION

[0036] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0037] Example 1:

[0038] Reference Figure 1 and Figure 2 According to a method for estimating whole-body posture of rehabilitation movements using monocular vision and sparse IMU provided by the present invention, the method comprises the following steps:

[0039] Step S1: synchronously collect video data and inertial data of human rehabilitation movements through a monocular RGB camera 7 and multiple inertial measurement units (IMUs), crop and scale the images to a preset resolution, and convert the IMU signals into a camera three-dimensional coordinate system; the number of inertial measurement units (IMUs) is 6, which are respectively deployed at preset positions of the head, two forearms, abdomen, and two calves, namely, a first wearable IMU sensor 1, a second wearable IMU sensor 2, a third wearable IMU sensor 3, a fourth wearable IMU sensor 4, a fifth wearable IMU sensor 5, and a sixth wearable IMU sensor 6; the first wearable IMU sensor 1 is deployed on the head; the second wearable IMU sensor 2 is deployed on the abdomen; the third wearable IMU sensor 3 is deployed on the right forearm; the fourth wearable IMU sensor 4 is deployed on the left forearm; the fifth wearable IMU sensor 5 is deployed on the right calf; and the sixth wearable IMU sensor 6 is deployed on the left calf;

[0040] Step S2: Use a sliding window and residual neural network to extract key point heat maps from N consecutive frames of images and calculate the 2D key point pixel coordinates of each frame; the number of frames N of the sliding window is a dynamically adjusted value, and the range is set to 8 to 16 frames.

[0041] Step S3: Concatenate the 2D keypoint pixel coordinates of N frames with the IMU's rotation matrix and acceleration signal into an input sequence, and introduce a learnable position encoding to capture temporal dependencies; the position encoding adopts a hybrid encoding method based on a sine function and learnable parameters.

[0042] Step S4: The temporal transformer performs cross-modal temporal modeling on the input sequence. After extracting high-dimensional features, the convolutional layer performs weighted averaging, and the regression head outputs the 3D relative keypoint coordinates of the last frame, with the pelvis as the root node. The regression head consists of two fully connected layers, using LeakyReLU as the activation function. The final layer outputs the offset of the 3D keypoint coordinates.

[0043] The present invention also provides a whole-body posture estimation system for rehabilitation movements using monocular vision and sparse IMU. The whole-body posture estimation system for rehabilitation movements using monocular vision and sparse IMU can be implemented by executing the process steps of the whole-body posture estimation method for rehabilitation movements using monocular vision and sparse IMU. That is, those skilled in the art can understand the whole-body posture estimation method for rehabilitation movements using monocular vision and sparse IMU as an optimal implementation of the whole-body posture estimation system for rehabilitation movements using monocular vision and sparse IMU.

[0044] Example 2:

[0045] The present invention also provides a rehabilitation movement whole body posture estimation system based on monocular vision and sparse IMU, the system comprising the following modules:

[0046] Module M1: Synchronously collects video and inertial data of human rehabilitation movements through a monocular RGB camera 7 and multiple inertial measurement units (IMUs), crops and scales the images to a preset resolution, and converts the IMU signals into the camera's three-dimensional coordinate system. There are six IMUs, deployed at preset positions on the head, two forearms, abdomen, and two calves.

[0047] Module M2: Use a sliding window and residual neural network to extract key point heat maps from N consecutive frames of images and calculate the 2D key point pixel coordinates of each frame; the number of frames N in the sliding window is a dynamically adjusted value, ranging from 8 to 16 frames.

[0048] Module M3: Concatenates the 2D keypoint pixel coordinates of N frames with the IMU's rotation matrix and acceleration signals into an input sequence, and introduces a learnable position encoding to capture temporal dependencies. The position encoding uses a hybrid encoding method based on a sine function and learnable parameters.

[0049] Module M4: A temporal transformer is used to perform cross-modal temporal modeling on the input sequence. After extracting high-dimensional features, the convolutional layer performs weighted averaging, and the regression head outputs the 3D relative keypoint coordinates of the last frame, with the pelvis as the root node. The regression head consists of two fully connected layers, using LeakyReLU as the activation function. The final layer outputs the offset of the 3D keypoint coordinates.

[0050] Example 3:

[0051] This paper proposes a whole-body pose estimation algorithm based on a monocular RGB camera (7) and sparse (6) inertial measurement units (IMUs). By fusing visual and IMU timing signals through a temporal Transformer architecture, the algorithm solves the problem of insufficient pose estimation accuracy caused by self-occlusion in complex rehabilitation movements. The algorithm includes the following steps:

[0052] Timing alignment: The camera and IMU sampling rate are both 60Hz, and they are synchronized through clapping. During clapping, the IMU acceleration reaches its peak at the moment the two hands touch, and the image clarity changes from blurry to sharp. (Image sharpness is represented by the variance of the Laplace filter result in OpenCV, and the sharpening frame is identified through threshold detection.) Align the timestamp of the IMU acceleration peak with the timestamp of the camera frame that sharpened. You can then perform another clapping motion to verify synchronization. If the difference between the IMU and camera timestamps during the second clap is less than 0.04 seconds, synchronization is considered successful.

[0053] Feature interaction: Each IMU provides 12 values ​​at each time step (9 from the rotation matrix and 3 from the acceleration vector), so the 6 IMUs provide a total of 72 values. The IMU signals of N frames are combined with the 2D keypoint coordinates (J keypoints, that is, J is the number of predefined human keypoints) to form the input sequence X∈R N×(2J+72) , where N is the time series length of the input image and IMU sequence, that is, the length of the sliding window, and R represents the real number domain. X can be regarded as the data x of N frames. i , i=1,2...N. i Through a shared linear embedding matrix E∈R (2J+72)×C Mapped into the latent space, where C is the embedding dimension, the embedded feature sequence Z∈R is obtained N×C .

[0054] In order to introduce temporal information, the learnable position encoding matrix E pos ∈R N×C Add it element by element with Z to get the final input of Transformer encoder Z0=Z+E pos =[x1E,x2E,…,x L E]+Epos .

[0055] The Transformer encoder consists of a stack of L layers, whose structure is given the embedded feature Z0∈R N×C The case can be expressed as follows:

[0056] Z′ l =MSA(LN(Z l-1 ))+Z l-1 , l=1,2,…,L

[0057] Z = MLP(LN(Z′ l ))+Z′ l , l=1,2,…,L

[0058] Y=LN(Z L )

[0059] Where LN(·) represents the layer normalization operation (the same as in ViT), MSA represents the multi-head self-attention layer, which has the same structure as in the classic Transformer model. MLP represents a multilayer perceptron. l-1 、Z' l 、Z l Used to represent the intermediate state of the model, Z l-1 : The feature representation of the input of the first layer of the Transformer encoder is the result of the output of the previous layer, Z' l : represents the intermediate feature representation after layer normalization and multi-head self-attention mechanism processing in the lth layer of the Transformer encoder. Z l : Feature representation of the output of the lth layer of the Transformer encoder.

[0060] The output of the Transformer encoder is Y∈R N×C , and then pass through the convolution layer and regression head to output the relative coordinates of the 3D key points of the last frame.

[0061] In the above process, the introduction of learnable position encoding will enable the model to improve its ability to understand time series, and the Transformer's Attention mechanism will implicitly model the spatiotemporal dependency between IMU and visual information (i.e., 2D key point coordinates), thereby realizing feature interaction between the two.

[0062] Step 1: Data collection and preprocessing;

[0063] Step 1.1: Use 7 monocular RGB cameras and 6 IMUs (located on the head, two forearms, abdomen, and two calves) to synchronously collect video data and inertial data of the human body while performing rehabilitation exercises.

[0064] Step 1.2: Crop and scale the image to 256×256 pixels and convert the IMU signal to the camera's 3D coordinate system.

[0065] In this study, the six IMU modules share the same world coordinate system, which is achieved by aligning and unifying the geomagnetic measurements of each IMU's built-in magnetometer. Each IMU can obtain: the rotation matrix from the IMU carrier coordinate system to the world coordinate system and the acceleration a of the IMU in the carrier coordinate system world .

[0066] Before the experiment begins, one of the IMUs is fixed to the ArUcomarker. The ArUcomarker is recognized by the camera and the calibration algorithm is run to obtain the rotation matrix of the ArUcomarker coordinate system relative to the camera coordinate system:

[0067] According to the fixed pasting relationship between IMU and ArUcomarker (for example, the rotation matrices of the two coordinate systems are aligned when pasting), the rotation matrix from the IMU coordinate system to the marker coordinate system can be calculated in advance:

[0068] Combining the above two, the rotation matrix of the IMU world coordinate system relative to the camera coordinate system can be derived:

[0069]

[0070] Thus, the rotation matrix from the IMU carrier coordinate system to the camera coordinate system and the acceleration of the IMU in the camera coordinate system are obtained:

[0071]

[0072] Step 2: 2D key point extraction;

[0073] Step 2.1: Use a sliding window and a residual neural network to extract key point heat maps of N frames from N consecutive frames of images.

[0074] Step 2.2: Calculate the pixel coordinates of the 2D key points of each frame using the heat map. (At this point, the results of the last N-1 frames can be stored for the next inference. The next inference only needs to calculate the heat map of the last frame in the sliding window to save computing power.)

[0075] Step 3: Fusion of vision and IMU signals;

[0076] Step 3.1: Concatenate the 2D keypoint pixel coordinates of N frames with the IMU rotation matrix and acceleration signal into an input sequence.

[0077] Step 3.2: Introduce learnable positional encodings to the input sequence to ensure that the model captures the temporal dependencies between frames.

[0078] Step 4: 3D keypoint regression;

[0079] The temporal transformer extracts high-dimensional features of the input sequence, learns the relationship between 2D key points and IMU signals, and after weighted averaging by the convolutional layer, the regression head outputs the 3D relative key point coordinates of the last frame (with the pelvis as the root node).

[0080] When posture recognition algorithms are used for rehabilitation movement guidance and interaction, a common approach is to calculate the key joint angles corresponding to the current movement using 3D keypoint coordinates. For example, in the McGill crunch, the angles of the neck and spine are very important, and consulting a rehabilitation therapist can provide angle references for these joints. The angles of these two joints are calculated using the key points that constitute them. The time series angles are filtered. When the angles of these two joints experience the following transitions: less than the minimum angle reference value, greater than the maximum angle reference value, and then less than the minimum angle reference value, a movement cycle is considered complete, thus achieving movement counting. If the angle remains within the minimum angle reference or within the maximum angle reference, a correction will be issued.

[0081] In addition to guiding interaction by calculating joint angles, joint angular velocity and angular acceleration can also be calculated to evaluate movement smoothness and responsiveness. The curve of joint angle changes over time can be recorded, and the smoothness of the curve can be observed to evaluate movement coordination and muscle control. In certain movements where the torso is almost stationary, such as the dead bug pedaling, the displacement amplitude of the root node can be observed to evaluate balance and core control.

[0082] By fusing monocular vision with sparse IMU signals, this method addresses the issue of reduced pose estimation accuracy in rehabilitation scenarios caused by severe self-occlusion in existing purely visual approaches, while also avoiding the inconvenience of densely deployed sensors required by traditional IMU solutions. By employing cross-modal temporal modeling with a temporal transformer, the jumps associated with single-frame estimation are avoided, enabling continuous, high-precision whole-body pose estimation for complex rehabilitation movements (such as kneeling dynamic frog stretches and single-leg book-opening maneuvers), providing reliable technical support for clinical rehabilitation assessment and guidance.

[0083] Those skilled in the art may understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0084] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0085] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A method for estimating whole-body posture of rehabilitation movements using monocular vision and sparse IMU, characterized by: The method comprises the following steps: Step S1: synchronously collect video data and inertial data of human rehabilitation movements through a monocular RGB camera and multiple inertial measurement units (IMUs), crop and scale the images to a preset resolution, and convert the IMU signals to the camera's three-dimensional coordinate system; Step S2: Use sliding window and residual neural network to extract key point heat map from N consecutive frames of images and calculate the 2D key point pixel coordinates of each frame; Step S3: Concatenate the 2D keypoint pixel coordinates of N frames with the IMU rotation matrix and acceleration signal into an input sequence, and introduce a learnable position encoding to capture temporal dependencies; Step S4: Perform cross-modal temporal modeling on the input sequence through the temporal transformer. After extracting high-dimensional features, the convolutional layer performs weighted averaging and the regression head outputs the 3D relative key point coordinates of the last frame, with the pelvis as the root node.

2. The method for estimating whole-body posture of rehabilitation movements using monocular vision and sparse IMU according to claim 1, characterized in that: The number of frames N of the sliding window in step S2 is a dynamically adjusted value, and the range is set to 8 to 16 frames.

3. The method for estimating whole-body posture of rehabilitation movements using monocular vision and sparse IMU according to claim 1, characterized in that: The position encoding in step S3 adopts a hybrid encoding method based on a sine function and a learnable parameter.

4. The method for estimating whole-body posture of rehabilitation movements using monocular vision and sparse IMU according to claim 1, wherein: The number of the inertial measurement units (IMUs) is 6, which are respectively deployed at preset positions on the head, two forearms, abdomen and two calves.

5. The method for estimating whole-body posture of rehabilitation movements using monocular vision and sparse IMU according to claim 1, characterized in that: The regression head consists of two fully connected layers, the activation function uses LeakyReLU, and the last layer outputs the offset of the 3D key point coordinates.

6. A whole-body posture estimation system for rehabilitation movements based on monocular vision and sparse IMU, characterized by: The system includes the following modules: Module M1: Synchronously collects video and inertial data of human rehabilitation movements through a monocular RGB camera and multiple inertial measurement units (IMUs), crops and scales the images to a preset resolution, and converts the IMU signals into the camera's 3D coordinate system. Module M2: Use sliding windows and residual neural networks to extract key point heat maps from N consecutive frames of images and calculate the 2D key point pixel coordinates of each frame; Module M3: concatenates the 2D keypoint pixel coordinates of N frames with the IMU rotation matrix and acceleration signals into an input sequence, and introduces a learnable position encoding to capture temporal dependencies; Module M4: Performs cross-modal temporal modeling on the input sequence through the temporal transformer, extracts high-dimensional features, performs weighted averaging through the convolutional layer, and outputs the 3D relative key point coordinates of the last frame through the regression head, with the pelvis as the root node.

7. The rehabilitation movement whole body posture estimation system based on monocular vision and sparse IMU according to claim 6, characterized in that: The number of frames N of the sliding window in the module M2 is a dynamically adjusted value, and the range is set to 8 to 16 frames.

8. The rehabilitation movement whole body posture estimation system based on monocular vision and sparse IMU according to claim 6, characterized in that: The position coding in the module M3 adopts a hybrid coding method based on a sine function and a learnable parameter.

9. The monocular vision and sparse IMU rehabilitation movement whole body posture estimation system according to claim 6, characterized in that: The number of the inertial measurement units (IMUs) is 6, which are respectively deployed at preset positions on the head, two forearms, abdomen and two calves.

10. The whole body posture estimation system for rehabilitation movement based on monocular vision and sparse IMU according to claim 6, characterized in that: The regression head consists of two fully connected layers, the activation function uses LeakyReLU, and the last layer outputs the offset of the 3D key point coordinates.

Images