A human posture estimation and motion evaluation integrated system in joint rehabilitation training

Abstract

The application relates to a human posture estimation and motion evaluation integrated system in joint rehabilitation training and belongs to the technical field of computer vision and intelligent rehabilitation medicine. The system comprises a data acquisition and preprocessing module, a BlazePose posture extraction module, a TCFormer motion recognition module and a TCDTW multidimensional motion evaluation module which are sequentially connected. The application aims to solve the problems of low efficiency and strong subjectivity of existing rehabilitation evaluation methods, and the problems of unstable posture extraction, insufficient time sequence modeling capability and single evaluation dimension of existing visual methods. The application realizes robust human key point extraction by introducing a lightweight BlazePose model, designs a TCFormer network integrating a Transformer and a Conformer to enhance the spatiotemporal feature modeling capability, and proposes a TCDTW algorithm integrating multidimensional features to realize fine motion alignment and scoring. The application can realize accurate recognition and objective and comprehensive quantitative evaluation of rehabilitation motions, and improves the intelligent level of rehabilitation training.

Description

Technical Field

[0001] This invention relates to the technical field of computer vision and intelligent rehabilitation medicine, specifically to a joint rehabilitation training movement recognition and evaluation system based on improved deep learning and multidimensional dynamic time warping. Background Technology

[0002] The standardization of joint rehabilitation exercises is crucial to the effectiveness of treatment. Currently, traditional rehabilitation assessments mainly rely on the subjective observation and experience of rehabilitation therapists, which suffers from low efficiency, strong subjectivity, and inability to meet the needs of large-scale telemedicine.

[0003] While existing vision-based intelligent rehabilitation methods have made some progress, they still face key technical bottlenecks: First, posture data extraction is unstable in unstructured environments (such as changes in lighting or occlusion); second, traditional convolutional neural networks have shortcomings in handling long-term dependencies and extracting local temporal features, making it difficult to effectively model complex continuous movements; finally, the movement evaluation mechanism lacks multi-dimensional fusion, and existing methods only perform coarse matching through key point displacements, ignoring motion angles, speed changes, and multi-joint collaborative features, making it difficult to comprehensively reflect the quality of the movement. Summary of the Invention

[0004] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a joint rehabilitation training movement recognition and evaluation system based on improved deep learning and multidimensional dynamic time warping. To address the problem of traditional high-precision pose estimation algorithms' heavy reliance on computational resources, this solution innovatively introduces BlazePose as the core of pose extraction, achieving efficient and robust extraction of 33 key points. To address the insufficient temporal modeling capability, this solution designs the TCFormer lightweight recognition network, integrating the global modeling capability of Transformer and the local perception advantage of Conformer, and introduces a multi-head self-attention mechanism to enhance focus on keyframes, significantly improving movement recognition accuracy. To address the problem of a single dimension in movement evaluation, this solution proposes the TCTDTW multidimensional dynamic time warping algorithm, integrating spatial, angular, dynamic, and rhythmic features for flexible alignment and quantified scoring, achieving objectivity and refinement in movement evaluation.

[0005] The technical solution adopted by the present invention is as follows: The present invention provides a joint rehabilitation training movement recognition and evaluation system based on improved deep learning and multidimensional dynamic time warping, including a data acquisition and preprocessing module, a BlazePose posture extraction module, a TCFormer movement recognition module and a TCDTW multidimensional movement evaluation module.

[0006] The data acquisition and preprocessing module specifically obtains raw video data by acquiring rehabilitation training video streams.

[0007] The BlazePose pose extraction module specifically obtains a standardized skeletal keypoint sequence through region detection, keypoint regression, and data standardization.

[0008] The TCFormer action recognition module specifically extracts and classifies action spatiotemporal features by constructing a hybrid network that integrates Transformer and Conformer, thereby obtaining action recognition results.

[0009] The TCDTW multidimensional action evaluation module specifically extracts multidimensional features and uses the TCDTW algorithm for alignment and weighted fusion to obtain multidimensional action evaluation results. Attached Figure Description

[0010] Figure 1 The accompanying figure is an abstract of an integrated system for human posture estimation and motion assessment in joint rehabilitation training provided by this invention.

[0011] Figure 2 This is a flowchart illustrating the BlazePose pose extraction module.

[0012] Figure 3 This is a schematic diagram of the network structure of the TCFormer action recognition module.

[0013] Figure 4 This is a flowchart illustrating the TCDTW multidimensional motion assessment module. Detailed Implementation

[0014] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0015] Example 1, see Figure 1 The present invention provides a joint rehabilitation training movement recognition and evaluation system based on improved deep learning and multidimensional dynamic time warping, including a data acquisition and preprocessing module, a BlazePose posture extraction module, a TCFormer movement recognition module and a TCDTW multidimensional movement evaluation module.

[0016] The data acquisition and preprocessing module is used to acquire rehabilitation training videos and perform noise reduction and format conversion.

[0017] The BlazePose pose extraction module receives video data and extracts the three-dimensional coordinates of 33 key points of the human body in real time, and outputs a standardized skeletal key point sequence.

[0018] The TCFormer motion recognition module receives a sequence of skeletal key points to identify the type of rehabilitation motion performed by the user (such as elbow flexion and extension, shoulder rotation, etc.).

[0019] Example 2, see Figure 2 This embodiment is based on the above embodiment, and the BlazePose pose extraction module specifically includes the following steps:

[0020] Region detection and localization, using lightweight networks to predict human ROI, can be formally described as follows:

[0021] In the formula, For the current frame image, This refers to key information from the previous frame. This is the detection function.

[0022] Keypoint regression prediction: Predicting the coordinates of 33 keypoints within the ROI. and visibility .

[0023] Data standardization involves normalizing the coordinates and calculating derived features, such as joint angles. :

[0024]

[0025] In the formula, , These are the vectors of adjacent bones.

[0026] Example 3, see Figure 3 As this embodiment is based on the above embodiment, the TCFormer action recognition module specifically includes the following steps:

[0027] Construct a hybrid network structure that combines a Transformer encoder with a Conformer module.

[0028] Multi-head self-attention enhancement (MHSA) is used to calculate temporal correlations, as shown in the following formula:

[0029] .

[0030] In the formula, , , These are query, key, and value matrices, respectively. This is the scaling factor. A multi-head mechanism is used to monitor different subspace features such as motion velocity changes and spatial displacement in parallel.

[0031] Local feature extraction utilizes the convolutional module in Conformer to capture local details:

[0032]

[0033] In the formula, Using the input features, short-term action transition features are extracted through one-dimensional convolution.

[0034] Example 4, see Figure 4 This embodiment is based on the above embodiment, and the TCTDTW multidimensional motion assessment module specifically includes the following steps:

[0035] Multidimensional feature extraction, extracting spatial features Angular features Dynamic characteristics and rhythmic features .

[0036] Dynamic Time Warping (DTW) alignment calculates the minimum path distance for each feature dimension:

[0037]

[0038] In the formula, For user sequences, For standard template sequences, For distance measurement, This represents the path weight.

[0039] Weighted fusion scoring to calculate comprehensive evaluation indicators :

[0040]

[0041] In the formula, Weight the importance of different feature dimensions (e.g., increase the weight of angle features in joint range of motion training). This represents the normalized distance for the corresponding dimension.

[0042] By performing the above operations, this invention not only solves the problem of high computational resource consumption in traditional methods, but also significantly improves the accuracy of rehabilitation movement recognition and the objectivity of assessment through the innovative combination of TCFormer and TCDTW, enabling it to adapt to the rehabilitation training needs in complex and dynamic scenarios.

Claims

1. The invention is characterized by comprising a data acquisition and preprocessing module, a BlazePose pose extraction module, a TCFormer motion recognition module, and a TCDTW multidimensional motion evaluation module; The data acquisition and preprocessing module specifically extracts rehabilitation training video streams from open-source datasets and performs data cleaning and standardization to obtain standard rehabilitation action sequence data. The BlazePose pose extraction module specifically locates the human body region using a lightweight detection model, extracts the 3D coordinates of 33 key points of the human body using a keypoint regression network, and calculates the visibility score to obtain a standardized skeletal keypoint sequence. The TCFormer action recognition module specifically constructs a lightweight recognition network that integrates Transformer and Conformer, inputs the standardized skeletal keypoint sequence into the model, and uses self-attention mechanism and local convolution to capture spatiotemporal features to obtain action category and confidence. The TCDTW multidimensional action evaluation module specifically extracts multidimensional action features, introduces a time-aligned composite dynamic time warping algorithm (TCDTW), realizes the alignment and scoring of user actions with standard templates, and obtains multidimensional action evaluation results.

2. The joint rehabilitation training movement recognition and evaluation system based on improved deep learning and multidimensional dynamic time warping according to claim 1, characterized in that: The BlazePose pose extraction module specifically includes the following steps: Region detection and localization, specifically using a lightweight BlazeFace model to detect the input video frame, quickly locate the Region of Interest (ROI) of the human body, and predict the ROI range of the current frame based on the key point positions of the previous frame; Key point regression prediction, specifically within the ROI region, predicting the three-dimensional coordinates (x, y, z) of 33 key points of the human body through a key point regression network, where the z coordinate is obtained through implicit depth estimation, and simultaneously predicting the visibility score of each key point; Data standardization processing, specifically normalizing the extracted key point coordinates, calculating joint angles, angular velocities, and relative distance derived features, and constructing a standardized skeletal key point sequence containing multi-dimensional semantic information.

3. The joint rehabilitation training movement recognition and evaluation system based on improved deep learning and multidimensional dynamic time warping according to claim 1, characterized in that: The TCFormer action recognition module specifically includes the following steps: A TCFormer action recognition model is established, which includes an embedding layer, a Transformer encoder module, a Conformer convolution module, and a classification output layer. Feature embedding and mapping specifically involves mapping the standardized skeletal keypoint sequence into a low-dimensional feature vector through an embedding layer; Global and local feature extraction: Specifically, the Transformer encoder module extracts long-distance dependencies in the action sequence to model the overall rhythm of the action; in parallel, the Conformer convolution module extracts local convolutional context information to capture short-term pose changes and key transitions. Multi-head self-attention enhancement specifically involves introducing a multi-head self-attention mechanism (MHSA) to dynamically calculate the correlation weights between temporal frames and focus on key joints and changing regions. The action classification output specifically involves inputting the extracted spatiotemporal features into the classification output layer and outputting the action category probability through the Softmax function.

4. The joint rehabilitation training movement recognition and evaluation system based on improved deep learning and multidimensional dynamic time warping according to claim 1, characterized in that: The TCTDTW multidimensional motion assessment module specifically includes the following steps: multidimensional feature extraction, specifically extracting spatial features, angular features, dynamic features, and rhythmic features from the skeletal keypoint sequence; the spatial features include the three-dimensional coordinates and trajectory length of key joints; the angular features include the angle between skeletal nodes and the degree of mobility; the dynamic features include angular velocity, linear velocity, and acceleration; and the rhythmic features include the motion cycle and smoothness index; multidimensional dynamic alignment, specifically constructing a dual-channel multidimensional time series and using the TCTDTW algorithm to independently perform time warping on multiple feature dimensions to eliminate temporal misalignment between user actions and standard templates; weighted fusion scoring, specifically weighting and fusing the matching distance of each dimension according to the importance weight of features to calculate a comprehensive similarity index; and quantitative feedback generation, specifically generating multidimensional motion scores and rehabilitation guidance suggestions based on the comprehensive similarity index, combined with the percentage of accurate keypoints and root mean square error.

5. The joint rehabilitation training movement recognition and evaluation system based on improved deep learning and multidimensional dynamic time warping according to claim 3, characterized in that: The integration of the Transformer encoder module and the Conformer convolution module is as follows: a convolutional layer is embedded in the self-attention structure of the Transformer, and global feature aggregation and local feature extraction are alternately performed through residual connections and layer normalization mechanisms to form a hierarchical spatiotemporal feature representation.

Images