A method for limb rehabilitation movement recognition and evaluation based on deep learning

Abstract

The present invention belongs to the field of computer application technologies and relates to a method for recognizing and evaluating limb rehabilitation movements based on deep learning. This method collects video data and skeleton data of limb functional rehabilitation movements, constructs training sets and validation sets for the video data and skeleton data after preprocessing; constructs a hybrid model of SE-ResBlock3D and LSTM, and the hybrid model realizes the recognition and classification of limb functional rehabilitation movements by sequentially passing the input video data through a convolutional layer, a sequence of 4 residual blocks, and an LSTM module; constructs a deep learning network SGP-Net, and the deep learning network SGP-Net sequentially passes the input preprocessed skeleton data through an STGCN module, a GAT module, and a PMTC module to extract and evaluate the spatial and temporal features of limb functional rehabilitation movements, so as to realize the quality evaluation of limb functional rehabilitation movements.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning, and particularly relates to a method for recognizing and evaluating limb rehabilitation movements based on deep learning. Background Art

[0002] Currently, the well-known methods and devices for action recognition and action quality assessment mainly have three major deficiencies: (1) Most of them are applied to healthy people, and there are few methods and devices for action recognition and action quality assessment for patients with limb dysfunction; (2) The application scenarios mainly include ordinary life scenarios (such as standing, walking, squatting, etc.) and sports scenarios (such as diving competitions, dancing, etc.), lacking application scenarios for the rehabilitation training of patients with limb functions; (3) Most of the existing methods are based on single-modal data of videos, while this method combines multi-modal information of RGB vision and skeleton model data, significantly improving the algorithm performance and accuracy. In contrast, this method can provide a finer-grained evaluation effect for rehabilitation movements with subtle differences.

[0003] The paper (Mourchid, Youssef, and Rim Slama. "D-STGCNT: A Dense Spatio-Temporal Graph Conv-GRU Network based on transformer for assessment of patient physical rehabilitation." Computers in Biology and Medicine 165(2023):107420.) proposed a new deep learning model - Dense Spatio-Temporal Graph Conv-GRU Network with Transformer (D-STGCNT) for automatically evaluating the quality of unsupervised patient physical rehabilitation exercises. This model combines a modified version of the STGCN and Transformer architectures to effectively process spatio-temporal data, especially human skeleton data. D-STGCNT uses dense connections and GRU mechanisms to quickly process large 3D skeleton inputs and effectively model time dynamics. The attention mechanism of the Transformer encoder focuses on the relevant parts of the input sequence, which is suitable for evaluating rehabilitation exercises. Evaluations on the KIM ORE and UI-PRMD datasets show that this method is superior to existing methods in terms of accuracy and computing time, achieving faster and more accurate learning and evaluation. The rehabilitation actions in the datasets used are few, unable to meet the needs of actual limb motor function rehabilitation, and thus leading to difficulties in covering various types of patients with limb dysfunction in the application scope of the model.

[0004] A rehabilitation action recognition method and system based on multi-modal information fusion (CN 117137435 A) When the user is performing rehabilitation training, the IMU acquisition device is used to continuously acquire the position data of 17 human key nodes of the user's body, and the RGB-D camera is used to continuously acquire the multi-action image data of the user; the multi-modal data alignment algorithm is used to align the data in space and time to obtain the multi-modal data after space-time alignment; based on the multi-modal data after space-time alignment, the action recognition algorithm of the lightweight modal screening decision network is used to recognize the rehabilitation actions of the user, and the data collected by the IMU is aligned with the data collected by the RGB-D camera in space and time, providing a reliable standard and basic data for the subsequent rehabilitation action recognition and the quality evaluation of rehabilitation actions.

[0005] The biggest drawback of this method is that it is difficult to guarantee the accuracy of the IMU. The IMU sensor may be affected by integration errors during long-term operation, resulting in drift of the position data. This drift may accumulate over time, affecting the long-term tracking and evaluation of actions.

[0006] A rehabilitation action evaluation method based on Transformer (CN 117671787 A) proposes a rehabilitation action quality evaluation method based on Transformer, which effectively solves the problem of difficult learning of the temporal relationship between long feature sequences. This method first designs a prediction model based on Transformer and trains the model using the public dataset KIMORE dataset. Then, the KinectV2 camera is used to collect skeleton node data, and the skeleton node sequence of the rehabilitation action to be evaluated is input into the trained prediction model for prediction, and the final result is output. The disadvantages of this method are as follows: ① It uses the relatively old Kinect V2 depth camera instead of the latest Kinect Azure DK depth camera, which is insufficient in terms of resolution, number of skeleton points, etc.; ② It does not fully consider the motion relationship between skeleton nodes, does not use ST-GCN or related algorithms to fully extract the motion relationship between skeleton nodes, and only uses the Transformer algorithm for action quality evaluation; ③ It only uses single-modal data and does not use multi-modal data, making it difficult to deeply obtain the action features behind limb movements and limiting the accuracy of limb movement quality evaluation.

[0007] A fitness action recognition method, device and related equipment based on human pose estimation (CN202211531164.8) proposes a fitness action recognition method and its device, computer equipment and storage medium based on human pose estimation. The method mainly includes the following steps: ① Video stream processing and human key point extraction: Extract human key point data from the video stream, detect and estimate the pose of the human body in the video frame through an object detection algorithm and a human pose tracking algorithm (using an improved BlazeDark algorithm), obtain the coordinates of human key points in a preset quantity and area, and perform serialization and centering processing. ② Action recognition: Input the extracted human key point data into a trained fitness action classification network (a stacked neural network of a multi-layer neural network MLP and a long short-term memory neural network LSTM) for action recognition, and obtain an action classification result, including recognized action data and corresponding categories. ③ Similarity calculation: Calculate the similarity between the recognized action data and the standard action data of the corresponding category in a preset standard fitness action dictionary to obtain a similarity value. ④ Action quality evaluation: Based on the similarity value, perform fitness action quality evaluation to obtain a quality evaluation result. The normalization process of the similarity value may be included during the evaluation process. This patent uses single-modal data, which is difficult to ensure the classification and evaluation accuracy, and lacks the ability to finely evaluate complex actions. For example, this patent uses similarity calculation for action quality evaluation, which is difficult to fully capture the subtle differences of actions, resulting in insufficiently fine evaluation results. In addition, the adaptability to dynamic changes is poor. The actions of users will have dynamic changes and irregularities, which may cause the action recognition model and similarity calculation method to perform poorly when facing non-standard actions, affecting the accuracy of fitness action quality evaluation.

[0008] Human action recognition technology (HAR) has important application values in fields such as rehabilitation, sports analysis, and security monitoring. Traditional HAR methods mainly rely on handcrafted features and classical machine learning techniques. These methods usually require rich domain knowledge and have limited generalization ability in different scenarios. With the development of deep learning technology, the research focus has shifted to using neural networks to automatically learn temporal and spatial features from raw videos to improve the accuracy and robustness of action recognition.

[0009] However, existing deep learning methods still have some problems when dealing with complex video data, such as insufficient feature extraction and inadequate capture of time dependence. In addition, existing methods usually only focus on action classification and ignore the evaluation of action quality. This is particularly disadvantageous for application scenarios such as medical rehabilitation that require precise evaluation of action quality. Therefore, there is an urgent need for a new method that can efficiently classify actions and accurately evaluate action quality. Summary of the Invention

[0010] The object of the present invention is to overcome the deficiencies in the prior art and provide a method for recognizing and evaluating limb rehabilitation movements based on deep learning.

[0011] To achieve the object of the present invention, the following technical solutions will be adopted for implementation.

[0012] A method for recognizing and evaluating limb rehabilitation movements based on deep learning includes the following steps:

[0013] S1. Preprocess the video data and skeleton data of the limb function rehabilitation movements collected, and construct a training set and a validation set for the video data and the skeleton data;

[0014] S2. Train and validate the constructed hybrid model with the training set and the validation set of the video data, and save the optimized hybrid model. The hybrid model includes a 3D convolutional neural network, at least two continuously arranged residual sequences, a fully connected layer, an LSTM layer, a fully connected layer, and a SoftMax function connected in sequence, and adjacent two residual sequences are connected through a downsampling convolutional layer, which is used for video feature extraction and classification to obtain video features of correct actions;

[0015] S3. Train and validate the constructed deep learning network SGP-Net with the training set and the validation set of the skeleton data, and save the optimized deep learning network SGP-Net. The deep learning network SGP-Net is composed of an STGCN module containing a GAT module and a PMTC module arranged in sequence, which is used to extract the spatial and temporal features for evaluating limb function rehabilitation movements; wherein: the STGCN module containing the GAT module is used to capture temporal features and retain the structural characteristics of spatial features, and dynamically calculate the attention weights between nodes through the GAT module to adaptively adjust the connection weights between nodes; the PMTC module includes a dynamic convolution kernel generator, a multi-scale convolution operation, a spatio-temporal joint attention mechanism, a temporal modeling module, and a temporal convolution module in sequence, which is used to extract and analyze the temporal dynamic features in the action sequence;

[0016] S4. Input the video features of correct actions obtained in step S2 and the spatial and temporal features extracted in step S3 into the action quality evaluation module for quality evaluation, and obtain personalized guidance and suggestions through the quality evaluation report.

[0017] As a preferred solution of the present invention, the deep learning network SGP-Net is composed of an STGCN module containing a GAT module and a PMTC module arranged in sequence; wherein:

[0018] The STGCN module containing the GAT module is used to capture temporal features while retaining the structural characteristics of spatial features. By dynamically calculating the attention weights between nodes, the connection weights between nodes are adaptively adjusted.

[0019] The STGCN module containing the GAT module includes a preliminary convolution and a skip localization mechanism for two-level graph convolution, which perform feature extraction operations for the first hop and the second hop respectively. Among them:

[0020] Preliminary convolution: The input features are processed using a (9,1) convolution kernel and a ReLU activation function to capture basic spatio-temporal features, and they are merged with the original input through a concatenation operation.

[0021] Skip localization mechanism: First, a (1,1) convolution operation is performed on the features merged by the preliminary convolution. Then, a ConvLSTM2D convolutional layer is used to capture temporal features. Next, the GAT module is used to dynamically calculate the attention weights between nodes, and the connection weights between nodes are adaptively adjusted.

[0022] The PMTC module includes a dynamic convolution kernel generator, multi-scale convolution operations, a spatio-temporal joint attention mechanism, a temporal modeling module, and a temporal convolution module. Among them:

[0023] Dynamic convolution kernel generator: According to the temporal features input by the STGCN module, convolution kernels with different odd dilation rates are generated.

[0024] Multi-scale convolution operations: In parallel with convolution kernels with different odd dilation rates, multi-level temporal feature extraction is performed on the input features to form different feature maps.

[0025] Spatio-temporal joint attention mechanism: It is used to perform weighted processing on the high-dimensional feature map formed by splicing different feature maps to form multi-scale features.

[0026] Temporal modeling module: It is used to capture both the temporal and spatial features of the multi-scale feature vectors integrated and compressed by the fully connected layer, and perform bidirectional processing on the time series information of the multi-scale feature vectors.

[0027] Temporal convolution module: Through convolution kernels with different even dilation rates, the changing trends in the action sequence are captured at different time scales.

[0028] As a preferred embodiment of the present invention, the limb function rehabilitation movements are Brunnstrom grading movements, including 6 upper limb movements and 7 lower limb movements, where: the 6 upper limb movements include basket-carrying posture, touching the lumbar spine, 90° arm flexion, 90° arm abduction, 180° arm flexion, and finger-to-nose test; the 7 lower limb movements include bilateral knee flexion, bilateral ankle dorsiflexion, bilateral heel sliding, standing knee flexion, bilateral ankle plantar flexion, hip abduction with knee extended, and bilateral foot side-sliding.

[0029] As a preferred embodiment of the present invention, the preprocessing of the video data includes the following steps:

[0030] S31. Unify the video frame rate to 30 frames per second;

[0031] S32. Scale the pixels of the original video data to 270x240;

[0032] S33. Perform frame extraction on the RGB and Depth videos;

[0033] S34. Align the time of each frame with the joint point data to maintain the spatio-temporal consistency of the data;

[0034] S35. View and record the start and end times of each action, and extract the frame sequence images containing the complete action;

[0035] S36. Use the Kalman filter to correct the noise of the data to improve the data quality;

[0036] S37. Perform downsampling on the extracted image sequence, that is, retain one frame image every 10 frames, which can fully retain the complete semantic information of the action;

[0037] S38. Perform data augmentation processing using horizontal flipping and noise reduction processing;

[0038] S39. Randomly crop the resolution of the video frame to 224x224, and normalize all pixel values to the range of 0-1;

[0039] S310. Unify the input data format of the hybrid model of SE-ResBlock3D and LSTM to (3, 24, 224, 224).

[0040] As a preferred embodiment of the present invention, the skeleton data is the three-dimensional space coordinate data of 32 human joint points, and the preprocessing of the three-dimensional space coordinate data of 32 human joint points includes the following steps:

[0041] S41. After aligning the three-dimensional space coordinate data of 32 joint points with the video frame in time, use every 100 frames as an input;

[0042] S42. When there are multiple segments of input for an action data and the last segment is less than 100 frames, it is padded from the first frame of the action data to 100 frames.

[0043] S43. According to the parent - child node correspondence table of 32 human body joint points given on the Azure Kinect official website, the serial numbers of these 32 joint points are linked to form a graph for use in a graph convolutional network.

[0044] S44. The input data format of the SGP - Net model is unified as (100, 32, 3).

[0045] As a preferred solution of the present invention, the 3D convolutional neural network is composed of a 3D convolutional layer, a batch normalization layer, and a ReLU activation function; wherein: the 3D convolutional layer has 32 convolutional kernels, the size of the convolutional kernels is (1, 7, 7), the stride is (1, 2, 2), and padding is performed in the height and width directions to maintain the size of the feature map.

[0046] As a preferred solution of the present invention, adjacent two of the at least two continuously arranged residual sequences are connected by a down - sampling convolutional layer. Each residual sequence contains at least three residual blocks, and each residual block is composed of two continuously arranged 3D convolutional layers, and an SE module is set after each residual block.

[0047] As a preferred solution of the present invention, the LSTM is used to capture long - term dependencies.

[0048] As a preferred solution of the present invention, the different odd dilation rates are respectively d = 3, d = 5, d = 7, d = 9, d = 11, and the quantity of each is 16.

[0049] As a preferred solution of the present invention, the different even dilation rates are respectively d = 2, d = 4, d = 8, and the quantity of each is 32.

[0050] Beneficial effects: The present invention aims to achieve a more comprehensive and accurate assessment of the motor rehabilitation of hemiplegic patients through deep learning and multi - modal fusion analysis. Among them, we focus on two core issues: First, how to effectively fuse the RGB video and skeleton data obtained by a depth camera to improve the information richness and accuracy of rehabilitation action discrimination and evaluation; Second, we focus on using advanced action quality assessment methods to analyze the quality of rehabilitation patients during movement execution, especially paying attention to the differences between individuals during the rehabilitation stage. Finally, solving the above two problems will help improve the accuracy and scientific nature of the assessment. Description of the Drawings

[0051] Figure 1 It is a structural diagram of the SE - R3D - LSTM module described in the present invention;

[0052] Figure 2 It is the structural diagram of the SGP-Net module described in the present invention;

[0053] Figure 3 It is the structural diagram of the PMTC module described in the present invention;

[0054] Figure 4 It is the accuracy graph of the hybrid model described in the present invention;

[0055] Figure 5 It is the loss graph of the hybrid model described in the present invention;

[0056] Figure 6 It is the confusion matrix graph of the hybrid model described in the present invention;

[0057] Figure 7 It is the prediction result graph of the STGCN model described in the present invention;

[0058] Figure 8 [[ID=2,6]]It is the prediction result graph of the SGP-Net network described in the present invention;

[0059] Figure 9 It is the flowchart of the method described in the present invention;

[0060] Figure 10 It is the original video data processing block diagram in the method described in the present invention;

[0061] Figure 11 It is the skeleton data processing block diagram in the method described in the present invention. Detailed implementation manners

[0062] For the convenience of those skilled in the art to understand, the present invention will be further described below in conjunction with embodiments and the accompanying drawings. The content mentioned in the embodiments does not limit the present invention.

[0063] As an embodiment of the present invention, as Figures 1 to 11 shown, a method for limb rehabilitation action recognition and evaluation based on deep learning includes the following steps:

[0064] I. Dataset description and preprocessing:

[0065] We collected a total of 13 different actions of 48 volunteers, and selected the Brunnstrom staging actions as shown in Table 1 below, including 6 upper limb and 7 lower limb actions. The age range of the volunteers is mainly from 30 to 70 years old, and the gender ratio is balanced.

[0066] Table 1

[0067]

[0068] In the present invention, the execution processes of all volunteers' actions were accurately recorded, including the start and end times of the actions, and a frame sequence image containing the complete action was extracted. To increase the distinguishability between frames, the present invention performed downsampling on the extracted image sequence, that is, one frame was retained every 10 frames. Under the condition that the device frame rate is 30 frames per second (FPS), the complete semantic information of the action can be fully retained through this downsampling operation. Finally, 575 action sequence images were sorted out, and the sequence length of each action was between 9 and 30 frames. The data set was divided into a training set and a validation set, where 85% of the data was used for model training, and the remaining 15% was used to verify the model performance to ensure the generalization ability of the model.

[0069] In the early data processing stage, the present invention first scaled the pixels of the original video data to 270x240 and performed various data augmentation processes, such as horizontal flipping and noise reduction processing, so as to generate more robust training and test data. To further optimize the matching between the data and the input of the neural network, the resolution of the video frames was randomly cropped to 224x224, and all pixel values were normalized to the range of 0-1. It should be noted that the number of frames in the video sequence was uniformly adjusted to 24 frames after interval selection. For sequences with less than 24 frames, they were padded to 24 frames by copying the last frame to ensure that all input data had the same shape and size.

[0070] Finally, the input data format of the model was unified to (3, 24, 224, 224). This processing not only ensured the consistency and standardization of data input, but also improved the generalization ability of the model in different environments. These data processing steps laid a solid foundation for the subsequent training and verification of the model, enabling the present invention to make full use of deep learning technology to automatically learn effective features from video data, thereby significantly improving the accuracy and robustness of human action recognition.

[0071] II. Limb Function Rehabilitation Action Recognition and Classification Module

[0072] The present invention proposes a new method for human action recognition and classification using a hybrid model that combines SE-ResBlock3D and LSTM. The overall architecture of the model is as Figure 1 shown. This model combines a 3D convolutional neural network (CNN), a residual block, an SE (Squeeze-and-Excitation) module, and LSTM to achieve efficient video feature extraction and classification.

[0073] First, the video data is input into a 3D convolutional layer with 32 filters, a convolutional kernel size of (1, 7, 7), a stride of (1, 2, 2), and padding in the height and width directions to maintain the size of the feature map. A batch normalization layer and a ReLU activation function are followed after the convolutional layer to increase non-linearity. Then, the data passes through the first residual block sequence, which contains three Residual Blocks, and each Residual Block consists of two 3D convolutional layers and an SE module. The residual block is designed such that the input can have skip connections to alleviate the vanishing gradient problem. Similarly, the feature map passes through the second and third residual block sequences in turn, each sequence contains multiple Residual Blocks, and the number of channels is 64 and 128 respectively. And after each residual block sequence, the feature map passes through a downsampling convolutional layer to further reduce the spatial size and increase the number of channels. Finally, the feature map passes through the fourth residual block sequence, which contains three Residual Blocks with 256 channels. Then, the feature map undergoes global average pooling to reduce the spatial dimension to 1.

[0074] Meanwhile, the feature is compressed to 256 dimensions through a 1D convolutional layer, and the feature tensor is transposed so that the frame number dimension is in the front to meet the input requirements of the LSTM, which is used to capture long-term dependencies in the video. The feature is input into a three-layer LSTM network, and the hidden state of the last time step of its output is taken. After passing through the ReLU activation function, it enters the fully connected layer for classification to obtain the final probability distribution of each action category.

[0075] SE-ResBlock3D is the main module for us to extract spatial features. This module combines 3D convolution, residual connection, and SE, and through the synergistic effect of these components, it enhances the feature extraction ability. Specifically, an SE module is added after the output of each residual block to recalculate the channel weights and weight the feature map. This method not only enhances the model's attention to key features but also demonstrates excellent computational efficiency and robustness, and can provide a significant performance improvement in computer vision tasks such as video classification and action recognition, with broad compatibility and application potential.

[0076] III. Spatial and Temporal Feature Extraction and Quality Assessment Module for Limb Function Rehabilitation Movements

[0077] The action quality assessment module of the present invention constructs an innovative deep learning model architecture SGP-Net based on STGCN, GAT, and PMTC for efficiently extracting and evaluating the spatial and temporal features of human actions. By deeply integrating these advanced neural network modules, the model realizes the accurate quality assessment of complex human actions and has broad application potential, especially in the fields of rehabilitation medicine and sports analysis.

[0078] The network structure of SGP-Net, as Figure 2 shown. During the implementation process, first, the STGCN module is used to extract features in the spatial and temporal dimensions from the input action video data. The STGCN module adopts a skip localization mechanism of initial convolution and two-level graph convolution to perform the first-hop and second-hop feature extraction operations respectively. In the initial convolution, the model uses a (9,1) convolution kernel with 64 convolution kernels. The input features are processed through the ReLU activation function to capture the basic spatio-temporal features, and they are merged with the original input through a concatenation operation to provide richer feature inputs for subsequent skip connections.

[0079] In each level of skip localization, the model first performs a (1,1) convolution operation on the input features with 64 convolution kernels, mainly for feature channel fusion and compression. Immediately afterwards, the ConvLSTM2D convolutional layer is used to further capture temporal features with 32 convolution kernels. ConvLSTM2D combines the convolutional operation with the temporal dependence modeling ability of LSTM, which can maintain the structural nature of features in the spatial dimension while capturing dynamic changes in the temporal dimension. The output of this layer is further processed by the GAT module. The GAT module dynamically calculates the attention weights between nodes. Through the LeakyReLU activation function and Softmax normalization processing, the connection weights between nodes are adaptively adjusted, making the feature propagation process more focused on important nodes and suppressing irrelevant noise.

[0080] Specifically, the GAT module performs a linear transformation on each node feature, calculates the similarity between nodes through a dot product operation, and normalizes the attention coefficients for dynamic weighted feature propagation. This mechanism ensures that the model can adaptively select and aggregate features in the complex relationships between nodes, significantly enhancing the efficiency and accuracy of feature extraction in different scenarios. The output features of the two-level skip connections are merged through a concatenation operation to form a rich feature set as the input of the subsequent PMTC module.

[0081] The design of the PMTC module aims to solve the extraction and analysis of temporal dynamic features in complex action sequences. As Figure 3 shown, the PMTC module combines dynamic convolution kernel generation, multi-scale convolution operations, and spatio-temporal joint attention mechanisms, greatly enhancing the model's feature capture ability at different time scales, especially suitable for complex action modeling of long time series.

[0082] First, the Dynamic Kernel Generator in the PMTC module generates convolution kernels with different dilation rates according to the input temporal features. The dilation rates are d = 3, d = 5, d = 7, d = 9, d = 11 respectively, ensuring that features can be effectively captured on multiple temporal scales. The size of the convolution kernel is (3,1), and its design purpose is to ensure that the receptive field in the temporal dimension is wide enough while reducing the interference of the spatial dimension on temporal modeling. By dynamically generating convolution kernels, the PMTC module can adaptively adjust the perception range of the convolution kernel according to the input data, ensuring that both short-term and long-term temporal features can be effectively captured.

[0083] Next, the system performs multi-level temporal feature extraction on the input data through parallel multi-scale convolution operations. Each convolutional layer performs convolution operations using independent dilation rates, ensuring that the model can process short-term and long-term temporal changes simultaneously on multiple temporal scales. The convolved features are non-linearly mapped through the ReLU activation function, and Dropout is used to prevent the model from overfitting. The feature maps output by each convolutional layer are merged through the Concatenate operation to form a high-dimensional feature representation, ensuring that multi-scale features are retained and providing input for the subsequent spatio-temporal joint attention mechanism.

[0084] After completing multi-scale convolution, a spatio-temporal joint attention mechanism is introduced. This mechanism weights the concatenated feature maps to ensure that the model can focus on the most important features at different times and scales. The calculation of the attention weights is achieved through spatio-temporal joint calculation, dynamically adjusting the weight relationship between features, and ensuring that the model focuses on the most representative temporal dynamic features. This adaptive weighting method can not only highlight the key changes in the action but also effectively reduce the interference of irrelevant features, improving the robustness and accuracy of the model.

[0085] Next, the multi-scale features weighted by the spatio-temporal joint attention mechanism enter the fully connected layer (FC) for further integration and compression. The fully connected layer compresses the multi-dimensional features into a more compact vector representation through non-linear mapping, ensuring that the features can better represent the key action changes in the time series and providing more effective input for the subsequent temporal modeling module.

[0086] Subsequently, the feature vectors processed by the fully connected layer are input into the ConvLSTM2D and bidirectional LSTM (Bi-LSTM) modules. The ConvLSTM2D module further enhances the model's ability to model spatio-temporal dependencies by simultaneously capturing temporal and spatial features, and can accurately extract the dynamic changes of complex actions. The Bi-LSTM module enhances the model's ability to capture long-term dependencies by processing time series information bidirectionally, ensuring that key changes occurring in the action sequence can be comprehensively identified and extracted. The bidirectional architecture of this module allows the model to process forward and backward information in the action sequence simultaneously, thus ensuring high accuracy and robustness in action recognition and evaluation.

[0087] After the temporal modeling is completed, the features are input into the Temporal Convolutional Network (TCN) module. The TCN module uses multiple convolutional kernels with different dilation rates (d = 2, d = 4, d = 8 respectively) to capture the changing trends in the action sequence at different time scales. Through multi-scale temporal modeling, the TCN module can effectively extract long-term dependencies, thereby ensuring that the model has strong temporal dynamic analysis capabilities in the action quality assessment of long sequences. Finally, the output features of the TCN module provide highly accurate temporal dynamic analysis for the comprehensive assessment of action quality, ensuring the accuracy and reliability of the action quality assessment results.

[0088] After the PMTC module completes feature extraction, the next step in the model's processing is to input the multi-scale features output by the PMTC into a (1,1) convolutional layer for further feature compression and adjustment. At this time, the model integrates the high-dimensional features through this small convolutional kernel, compresses them into a more compact feature vector, ensuring the computational efficiency of the subsequent stage and retaining the key information. Immediately afterwards, these processed feature vectors are passed to the last module of the model - the LSTM module. The LSTM is used to capture long-term dependencies in the video sequence. Through the structure of the LSTM, the model can effectively identify the long-term features and complex temporal dynamic changes of actions in the video sequence. This is crucial for the quality assessment of human actions because the smoothness and coordination of actions often require comprehensive analysis through long-term sequence information. The LSTM module adopts a stacked layer design to ensure that the temporal information of complex action sequences can be extracted and integrated layer by layer. Finally, the output of the LSTM generates a high-dimensional feature vector containing the dynamic features of the complete action sequence, which provides detailed data support for the subsequent evaluation tasks, ensuring that when conducting action quality assessment, it can comprehensively reflect key indicators such as the smoothness, accuracy, and consistency of actions.

[0089] The technical route of the present invention realizes a comprehensive evaluation of human actions through the powerful spatio-temporal feature extraction of the STGCN module, the adaptive attention weighting of the GAT module, and the multi-scale dynamic modeling of the PMTC module. Combining the time series processing ability of the LSTM module, the model can efficiently process complex backgrounds and diverse action scenarios, and is widely applied in fields such as rehabilitation therapy and sports analysis, providing strong support for the development of industry technologies.

[0090] A more accurate algorithm for limb function rehabilitation action recognition and action quality assessment has significant advantages in limb movement rehabilitation and home rehabilitation. It can optimize through personalized training plans, provide real-time feedback and correction, help patients improve the effect of rehabilitation training, avoid secondary injuries caused by improper actions, and quantify the rehabilitation progress, facilitating rehabilitation doctors to evaluate the treatment effect. In terms of home rehabilitation, the accurate recognition algorithm ensures that patients can also obtain high-quality rehabilitation guidance at home, reducing treatment costs. At the same time, through remote monitoring and support, it improves the compliance and rehabilitation experience of patients, making rehabilitation more reliable and efficient.

[0091] During the training and validation process, the proposed model's validation accuracy and loss graphs on the validation set are as shown in Figure 4 and Figure 5 The validation accuracy of the proposed method reaches 97.70% at the 2000th epoch. The validation loss drops from the initial 0.3121 to 0.0001 at the end of the iteration. The results show that the proposed method is very effective, with extremely low training and validation losses and high accuracy during the training and validation processes.

[0092] According to the experimental results, all categories perform well. For further analysis, we calculate the recall rate, accuracy, precision, and F1 score of the proposed model on the test set, as shown in Table 2.

[0093] Table 2

[0094]

[0095] The confusion matrix shows the comparison between the prediction results and the actual results of the hybrid model. We can clearly see in which categories the model performs well and in which categories there are misclassifications. As shown in Figure 6 , the confusion matrix shows accurate predictions along the diagonal. Deeper colors indicate higher classification accuracy of the proposed model for the relevant classes, while lighter colors indicate samples with misclassifications. When testing 13 types of human action classifications, the model achieved excellent results in all performance evaluation metrics.

[0096] The experimental results show that our method has achieved high accuracy and robustness in the 13-class action classification task, verifying its effectiveness in processing complex video data. The evaluation results of the confusion matrix further prove the excellent performance of the model in each category, especially its outstanding ability to distinguish similar actions. At the same time, the accuracy of distinguishing between class 1 and class 4 actions still needs to be further improved. In addition, our dataset covers different ages and genders, ensuring the wide applicability and generalization ability of the model and demonstrating great potential in practical applications. Generally speaking, the model proposed in the present invention not only has significant innovations in theory but also shows great potential in practical applications.

[0097] IV. Evaluation of Model Prediction Performance

[0098] In the action quality evaluation of the present invention, a graph attention network (GAT) and a spatio-multi-scale temporal convolution (PMTC) module are introduced and fused into a new network structure, SGP-Net. Compared with the traditional baseline model (STGCN), the performance on the test set has been significantly improved. In the present invention, we use four evaluation metrics, namely the mean absolute error (MAD), root mean square error (RMS), mean absolute percentage error (MAPE), and mean square error (MSE), to comprehensively evaluate the prediction performance of the model from different perspectives.

[0099]

[0100] MAD measures the average level of the absolute error between the predicted value and the actual value, reflecting the average size of the prediction error. MAD calculates the average of the absolute values of the differences between the predicted value and the true value, so it can directly reflect the accuracy of the prediction, and its unit is the same as the original data.

[0101]

[0102] RMS is used to evaluate the magnitude of the difference between the predicted value and the actual value, which is obtained by calculating the square root of the mean of the sum of the squared errors. RMS can magnify larger errors, and the smaller its value, the better the prediction effect of the model.

[0103]

[0104] MAPE is an evaluation of the percentage of the prediction error relative to the actual value, reflecting the proportional size of the prediction error relative to the actual value. MAPE is especially suitable for comparing data of different magnitudes. By normalizing the error to a percentage, it is convenient to compare the errors between different datasets. The smaller the MAPE, the lower the relative error of the model and the better the prediction effect.

[0105]

[0106] The MSE measures the mean squared size of the prediction error by calculating the square of the difference between the predicted value and the actual value and then averaging it. The MSE can amplify larger prediction errors, is sensitive to outliers, and is suitable for model optimization and improvement. The smaller the MSE value, the higher the accuracy of the model prediction.

[0107] These evaluation metrics can comprehensively measure the accuracy, robustness, and consistency of model prediction, providing a multi-dimensional reference basis for model performance evaluation.

[0108] As shown in Table 3 of the test set results, after introducing the GAT and PMTC modules, the MAD decreased from 0.03540 of the baseline model to 0.02727, the RMS decreased significantly from 0.09582 to 0.07257, the MAPE decreased significantly from 0.40777 to 0.31336, and the MSE decreased from 0.00918 to 0.00526. The comprehensive optimization of these metrics indicates that the present invention has significantly improved the evaluation accuracy and generalization ability of the model for action quality.

[0109] The significant reduction in MAD and RMS reflects a substantial improvement in the prediction accuracy of the model. In particular, the reduction in RMS indicates that the overall distribution of prediction errors is more concentrated, reducing the impact of large deviations. The substantial decrease in MSE further confirms the effectiveness of the model in reducing large errors, demonstrating the powerful ability of the model in processing complex action sequences after introducing the new module. The reduction in MAPE indicates that the improved model has significantly enhanced its ability to handle proportional errors, meaning that the model can more accurately identify and evaluate action quality and performs better in dealing with diverse action patterns.

[0110] After introducing the GAT module, the model enhances its attention to key features by adaptively adjusting the weights between nodes, thus maintaining a high recognition accuracy in complex scenarios. The dynamic weighting mechanism of GAT effectively suppresses the propagation of noise features, enabling the model to still maintain a robust performance on unseen test data. The PMTC module further enhances the model's sensitivity to temporal changes by accurately modeling the temporal dynamic features, enabling the model to not only capture static feature information but also track the dynamic changes during the execution of actions. This ability to extract dynamic features enables the model to accurately evaluate the quality of actions in complex time series data, especially in tasks that require differentiating action details and fluency, where it performs excellently.

[0111] Overall, after introducing the GAT and PMTC modules and integrating them with the baseline model into the new network architecture SGP-Net, the significant performance improvement on the test set fully demonstrates the innovation and practicality of this technical route. This improvement not only significantly enhances the evaluation accuracy of the model but also strengthens the robustness and generalization ability of the model, enabling it to handle complex and variable action patterns in practical applications and providing valuable technical support for fields such as rehabilitation medicine and motion analysis, such as Figure 7 and Figure 8 shown

[0112] Table 3

[0113]

[0114] The present invention provides a system and method based on multimodal deep learning for human action classification and quality assessment, which is particularly applicable to rehabilitation therapy and motion analysis. Through the acquisition and fusion of multimodal data and advanced deep learning algorithms, the system realizes efficient and accurate classification and quality assessment of human actions. The specific implementation process is as Figure 9 shown, where the video data processing module is as Figure 10 shown, and the skeleton data processing module is as Figure 11 shown

Claims

1. A method for limb rehabilitation motion recognition and evaluation based on deep learning, characterized in that: It includes the following steps: S1. Preprocess the video data and skeleton data of the limb function rehabilitation actions collected, and construct the training set and validation set of the video data and skeleton data; S2. Train and validate the constructed hybrid model with the training set and validation set of the video data, and save the optimized hybrid model. The hybrid model includes a 3D convolutional neural network, at least two consecutively arranged residual sequences, a fully connected layer, an LSTM layer, a fully connected layer, and a SoftMax function connected in sequence. And two adjacent residual sequences are connected through a downsampling convolutional layer, which is used for video feature extraction and classification to obtain the video features of correct actions; S3. Train and validate the constructed deep learning network SGP-Net with the training set and validation set of the skeleton data, and save the optimized deep learning network SGP-Net. The deep learning network SGP-Net is composed of an STGCN module containing a GAT module and a PMTC module arranged in sequence, which is used to extract the spatial and temporal features for evaluating the limb function rehabilitation actions. Among them: the STGCN module containing the GAT module is used to capture the temporal features and retain the structural characteristics of the spatial features at the same time, and dynamically calculate the attention weights between nodes through the GAT module to adaptively adjust the connection weights between nodes; the PMTC module includes a dynamic convolution kernel generator, a multi-scale convolution operation, a spatio-temporal joint attention mechanism, a temporal modeling module, and a temporal convolution module in sequence, which is used to extract and analyze the temporal dynamic features in the action sequence; S4. Input the video features obtained in step S2 and the spatial and temporal features extracted in step S3 into the action quality evaluation module for quality evaluation, and obtain personalized guidance and suggestions through the quality evaluation report.

2. The method for limb rehabilitation motion recognition and evaluation based on deep learning according to claim 1, wherein: The STGCN module containing the GAT module includes a preliminary convolution and a two-level graph convolution jump localization mechanism, which respectively perform the feature extraction operations of the first jump and the second jump. Among them: Preliminary convolution, processes the input features using a (9,1) convolution kernel and a ReLU activation function, captures the basic spatio-temporal features, and merges them with the original input through a splicing operation; The jump localization mechanism first performs a (1,1) convolution operation on the features merged by the preliminary convolution, then uses a ConvLSTM2D convolutional layer to capture the temporal features, and then dynamically calculates the attention weights between nodes through the GAT module to adaptively adjust the connection weights between nodes.

3. The method for limb rehabilitation motion recognition and evaluation based on deep learning according to claim 1, wherein: The dynamic convolution kernel generator generates convolution kernels with different odd dilation rates according to the temporal features input by the STGCN module; the multi-scale convolution operation is parallel to the convolution kernels with different odd dilation rates to perform multi-level temporal feature extraction on the input features to form different feature maps; where: the different odd dilation rates are d = 3, d = 5, d = 7, d = 9, d = 11 respectively, and the quantity of each is 16; the spatio-temporal joint attention mechanism is used to perform weighted processing on the high-dimensional feature map formed by splicing different feature maps to form multi-scale features; the temporal modeling module is used to capture the temporal and spatial features of the multi-scale feature vectors integrated and compressed by the fully connected layer at the same time, and perform bidirectional processing on the time series information of the multi-scale feature vectors; the temporal convolution module captures the change trend in the action sequence at different time scales through convolution kernels with different even dilation rates; where: the different even dilation rates are d = 2, d = 4, d = 8 respectively, and the quantity of each is 32.

4. A method for limb rehabilitation motion recognition and evaluation based on deep learning according to claim 1, characterized in that The limb function rehabilitation actions are Brunnstrom grading actions, including 6 upper limb actions and 7 lower limb actions, where: the 6 upper limb actions include basket-carrying posture, touching the lumbar spine, arm flexion at 90°, arm abduction at 90°, arm flexion at 180°, and finger-nose test; the 7 lower limb actions include bilateral knee flexion, bilateral ankle dorsiflexion, bilateral heel sliding, standing knee flexion, bilateral ankle plantar flexion, hip abduction with knee extension, and bilateral foot side sliding.

5. A method for limb rehabilitation motion recognition and evaluation based on deep learning according to claim 1, characterized in that, The preprocessing of the video data includes the following steps: S31: Unify the video frame rate to 30 frames per second; S32: Scale the pixels of the original video data to 270x240; S33: Perform frame extraction on the RGB and Depth videos; S34: Align each frame with the joint point data in time to maintain the spatio-temporal consistency of the data; S35: View and record the start and end moments of each action, and extract the frame sequence images containing the complete action; S36: Use the Kalman filter to correct the noise of the data to improve the data quality; S37: Perform downsampling processing on the extracted image sequence, that is, retain one frame image every 10 frames, which can fully retain the complete semantic information of the action; S38: Perform data augmentation processing using horizontal flipping and noise reduction processing; S39: Randomly crop the resolution of the video frame to 224x224, and normalize all pixel values to the range of 0-1; S310: Unify the input data format of the hybrid model of SE-ResBlock3D and LSTM to (3, 24, 224, 224).

6. A method for limb rehabilitation motion recognition and evaluation based on deep learning according to claim 1, characterized in that The skeleton data is the three-dimensional spatial coordinate data of 32 human joint points. The preprocessing of the three-dimensional spatial coordinate data of 32 human joint points includes the following steps: S41: After aligning the three-dimensional spatial coordinate data of 32 joint points with the video frame in time, take every 100 frames as an input; S42: One action data has multiple segments of input. When the last segment is less than 100 frames, fill it up to 100 frames starting from the first frame of this action data. S43. Connect the sequence numbers of these 32 joint points into a graph according to the parent-child node correspondence table of 32 human joint points given on the official website of Azure Kinect for use in the graph convolutional network; S44. The input data format of the SGP-Net model is unified as (100, 32, 3).

7. A method for limb rehabilitation motion recognition and evaluation based on deep learning according to claim 1, characterized in that The 3D convolutional neural network is composed of a 3D convolutional layer, a batch normalization layer, and a ReLU activation function; among them: the 3D convolutional layer has 32 convolutional kernels, the size of the convolutional kernels is (1, 7, 7), the stride is (1, 2, 2), and padding is performed in the height and width directions to maintain the size of the feature map.

8. A method for limb rehabilitation motion recognition and evaluation based on deep learning according to claim 1, characterized in that Adjacent two of the at least two continuously arranged residual sequences are connected by a downsampling convolutional layer, each residual sequence contains at least three residual blocks, each residual block is composed of two continuously arranged 3D convolutional layers, and an SE module is arranged after each residual block.

9. A method for limb rehabilitation motion recognition and evaluation based on deep learning according to claim 1, characterized in that The LSTM is used to capture long-term dependencies.

Images