Rehabilitation training system based on key point recognition model and graph neural network

Abstract

The invention discloses a rehabilitation training system based on a key point recognition model and a graph neural network, and belongs to the field of deep learning neural networks. According to the system, a YOLOv8-Pose attitude recognition model, STGCN space-time diagram convolution and a double-layer full-convolution twin network are fused. The YOLOv8-Pose posture recognition model is used for accurately detecting the posture of the human body and providing basic data for action analysis of the posture of the human body. Based on STGCN space-time diagram convolution after space-time attention improvement and a double-layer full-convolution twin network, action recognition is carried out according to the features of human body postures, the correctness of rehabilitation actions is evaluated, and therefore the rehabilitation progress of a patient is accurately monitored and evaluated.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning neural networks, and specifically relates to a rehabilitation training system DL-REHABIDETACT based on a key point recognition model and a graph neural network. Background Art

[0002] With the accelerating aging of the population, traditional rehabilitation diagnosis methods, which rely on extensive manpower and time, are no longer able to meet the needs of modern society. Therefore, high-efficiency and high-precision AI-based rehabilitation diagnosis has become a hot topic in medical research. The application of AI in rehabilitation diagnosis not only improves diagnostic efficiency and accuracy, but also enhances the generalization capabilities of models, improving their interpretability and credibility, thereby promoting continuous improvement and optimization.

[0003] Against this backdrop, deep learning AI models, particularly keypoint recognition technology and spatiotemporal sequence extraction models such as OpenPose and YOLOv8-Pose, as well as ST-GCN (Spatiotemporal Graph Convolutional Network), have matured amidst the rapid development of computer vision. These technologies provide a foundation for AI-based self-diagnosis of rehabilitation conditions. These technologies can accurately detect human posture and extract the spatiotemporal characteristics of movements, enabling accurate analysis of patients' rehabilitation movements.

[0004] Based on these technologies, an innovative model construction idea is proposed: first, the YOLOv8-Pose posture detection model is used for posture detection to obtain the key point information of the human skeleton; then, the improved ST-GCN spatiotemporal graph convolutional network is used to extract motion features and capture the dynamic changes of motion; finally, a two-layer fully convolutional network is used to judge the correctness of the motion feature recognition results, thereby realizing automatic assessment of the patient's recovery status.

[0005] This integrated model not only ensures data quality and improves model performance, but also enhances model generalization, improves model interpretability, and increases credibility, providing patients with a more scientific and autonomous rehabilitation training experience. Through AI technology, patients can gain real-time insights into the effectiveness of rehabilitation training, while doctors can more effectively monitor and adjust rehabilitation plans, achieving digital, visual, and scientific rehabilitation training. As AI technology matures in the control field, it demonstrates tremendous development potential and broad application prospects in rehabilitation medicine. Summary of the Invention

[0006] This invention focuses on applying deep learning AI models to patients' motor and behavioral rehabilitation, aiming to contribute to the field of rehabilitation diagnosis through scientific research innovation, and thus promote the development of health care.

[0007] This paper proposes an integrated model construction strategy that combines the YOLOv8-Pose posture recognition model, the STGCN spatiotemporal graph convolution, and a two-layer fully convolutional Siamese network. First, the YOLOv8-Pose posture recognition model is used to accurately detect human posture, providing basic data for motion analysis of human posture. Then, the STGCN spatiotemporal graph convolution, improved based on spatiotemporal attention, and the two-layer fully convolutional Siamese network perform action recognition based on these characteristics of human posture and evaluate the correctness of rehabilitation movements, thereby accurately monitoring and evaluating the patient's rehabilitation progress.

[0008] Furthermore, the YOLOv8 in the YOLOv8-Pose pose recognition model introduces new features and improvements to obtain YOLOv8-Pose based on the previous YOLO version to further improve performance and flexibility. The YOLOv8-Pose is a model in the YOLOv8 series that is specifically used for pose estimation tasks. It can identify key points of the human body in an image, which can represent various parts of the human body, such as joints, landmarks or other significant features. In the official pre-trained YOLOv8-Pose model, there are 17 key points, each of which represents a different part of the human body, such as the nose (one key point), eyes (two key points), ears (two key points), shoulders (two key points), elbows (two key points), wrists (two key points), hips (two key points), knees (two key points) and ankles (two key points).

[0009] Furthermore, the key point coordinate shape of the original output in the YOLOv8-Pose posture recognition model is (1,17,2), which is set in a list of length N. 1 represents one frame, 17 represents 17 key points, and 2 represents the x-coordinate and y-coordinate. Afterwards, the output is post-processed. In order to ensure the uniformity of the input of the YOLOv8-Pose posture recognition model, the longest target number of frames in the model video is set during training, and the length of the model video is the final length during inference. When the number of frames exceeds the set number of frames, the excess part is cut off. When the number of frames is insufficient, the frames are supplemented backward based on the key points of the last frame. Afterwards, the key points of all frames are spliced ​​from the 0th dimension to form a key point sequence, and the shape of the output key point sequence is (M,17,2).

[0010] Furthermore, to improve the processing efficiency of the present invention, an optimization strategy is implemented, specifically targeting the time-consuming skeleton recognition step. During dataset creation, a larger YOLOv8x-Pose model is used to perform keypoint recognition to ensure accuracy. However, during the inference phase, a more lightweight YOLOv8n-Pose model is used for skeleton recognition to speed up processing. This design enables the system to achieve fast response while maintaining high accuracy.

[0011] However, whether in the training or inference process, only one person appears completely in the video, which only involves single-person detection. There is no person tracking, and it is impossible to determine the ownership of the skeleton.

[0012] Furthermore, the normalization and coordinate point deletion algorithm MODL is used to standardize and lightweight the obtained skeleton, reducing performance usage while improving the model detection effect.

[0013] The normalization algorithm is described as follows: In actual applications, the distance between the human body and the lens will continue to change when moving, and the sizes of different pictures will also cause the pixel units of the x-axis and y-axis to be inconsistent. In addition, differences in height and weight between different individuals will also affect the consistency of the coordinates. In order to simplify the processing flow and improve performance, the coordinates need to be normalized. The specific operation is: for a picture with a width of w and a height of h, there are 13 coordinate points (xi, yi). The normalization process includes dividing all xi coordinate values ​​by the picture width w, and dividing all yi coordinate values ​​by the picture height h. After this processing, the coordinate points will no longer depend on the specific size of the picture, thereby achieving standardization. The formula is as follows:

[0014]

[0015] That is to say, the normalization of the skeleton coordinates is completed.

[0016] The coordinate point removal algorithm is described as follows: During research, we discovered that not all identified keypoints contribute substantially to the algorithm's final performance. In fact, some keypoints may be redundant or weakly relevant to the core objective, negatively impacting the algorithm's efficiency and accuracy. Therefore, we decided to take a more streamlined approach to optimize the algorithm by directly removing unnecessary keypoints.

[0017] Initially, 17 potential keypoints (the ones mentioned above) were extracted from the data. While these keypoints might theoretically be helpful for the algorithm, in practice, some were found to be less effective than expected. To improve algorithm efficiency and reduce unnecessary computation, the decision was made to filter these keypoints. Through a series of tests and evaluations, it was determined that only 13 keypoints (specifically, nose, shoulders, elbows, wrists, hips, knees, and ankles) were truly necessary for the model. Consequently, these additional keypoints were removed, improving model performance.

[0018] In practice, we first deleted the extra coordinates, reducing the coordinate point dimension from 17 to 13. At the same time, we modified the skeleton graph connection edges in the model to ensure that the model can accurately capture the relationship between each coordinate point when processing data, thereby laying a solid foundation for subsequent classification tasks.

[0019] STGCN (Spatio-Temporal Graph Convolutional Networks) is a deep learning framework specifically designed for processing spatiotemporal sequence data. It was initially used in the field of traffic prediction. By combining graph convolution and gated temporal convolution, it effectively captures spatial and temporal dependencies, thereby improving prediction accuracy. A two-layer fully convolutional network is designed at the final connection to output the judgment results.

[0020] The composition structure is as follows:

[0021] Spatiotemporal convolutional block: This block consists of a sandwich structure consisting of two gated temporal convolutional layers and a spatial graph convolutional layer. The gated temporal convolutional layer contains one-dimensional convolution and a gated linear unit (GLU), while the spatial graph convolutional layer is used to convolve graph-structured data to extract spatial features.

[0022] Stacked structure: Multiple spatiotemporal convolution blocks are stacked, with two layers stacked to fully extract spatiotemporal features.

[0023] Attention module: Through the attention mechanism, the model's attention to specific features is strengthened.

[0024] Convolutional layer: The network contains two convolutional layers. On the two branches of the twin network, the convolutional layer performs convolution on the input image to extract image features and complete classification at the same time.

[0025] The STGCN uses the relationships between skeleton nodes as edges and takes a time-series image sequence as input. It performs convolution operations on the graph structure data through a spatial graph convolution layer to extract spatial features. It then uses a gated temporal convolution layer to process the time series information and capture temporal dynamics. Multiple spatiotemporal convolution blocks are stacked to fully extract spatiotemporal features. Finally, the output layer passes through a temporal convolution layer and a fully connected layer for further processing.

[0026] A two-layer fully convolutional twin network fuses two images. In each branch, the image first passes through a convolutional layer, where the convolution kernel slides across the image to perform a convolution operation, extracting low-level features such as edges and textures. It then passes through a nonlinear activation function to enhance the network's nonlinear representation capabilities. Finally, the features extracted from both branches are fused and classified to measure the similarity between the two images. The final output is used for tasks such as image similarity determination and image matching.

[0027] This system makes the following improvements to STGCN to improve its performance and accuracy in processing spatiotemporal sequence data:

[0028] 1. Enhanced Graph Convolutional Layer: This introduces an improved graph convolutional layer that better captures complex relationships between nodes. By optimizing the graph construction method, the model's adaptability to dynamic changes is improved.

[0029] 2. Multi-scale temporal convolution: In the temporal convolution part, a multi-scale convolution strategy is adopted, which enables the model to capture both short-term and long-term temporal dependencies. This method effectively improves the model's ability to learn features at different time scales.

[0030] 3. Adaptive Attention Mechanism: We designed an adaptive attention mechanism that dynamically adjusts attention weights based on the characteristics of the input data. This mechanism makes the model more flexible and efficient when processing different types of spatiotemporal sequences.

[0031] 4. Spatial-temporal Cross-Attention Mechanism: This paper proposes an attention mechanism that combines spatial and temporal information. It is widely used in tasks such as video analysis, action recognition, and time series prediction. By dynamically assigning different weights, it captures key spatial regions and key time steps in the data, thereby improving the model's ability to model spatiotemporal features.

[0032] Through the above improvements, the STGCN model performs well in multiple spatiotemporal series prediction tasks, significantly improving the accuracy and stability of predictions. These innovations will provide new ideas and methods for the research and application of spatiotemporal series data.

[0033] The Double-Layer Fully Convolutional Network (DFC) is a deep learning network designed for image classification. Its core is to replace traditional fully connected layers with two fully convolutional layers, thereby reducing the classification task from the entire image level to the pixel level. This design not only improves the model's performance but also enhances its ability to capture local features in the image.

[0034] The optimization function selected is the cross entropy function. The formula is as follows:

[0035]

[0036] The optimizer used was AdamW. The learning rate was set to 1e-3. AdamW is an improvement on the Adam optimizer, incorporating a weight decay mechanism to prevent model overfitting. It combines the advantages of the Adagrad and RMSProp optimizers and can adaptively adjust the learning rate of each parameter. The learning rate determines the step size of the parameter update during each iteration.

[0037] Self-made video action feature dataset;

[0038] This model was retrained based on a custom dataset. To reduce performance overhead during training, the captured videos were converted into keypoint sequences and saved as npy files. The processed dataset consists of 824 keypoint sequences and their corresponding labels. The labels are 0 and 1, with 0 representing correct and 1 representing incorrect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Flowchart of the method of the present invention.

[0040] Figure 2 This is an example of yolov8-pose recognition effect.

[0041] Figure 3 It is a graphical interface for patent applications. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below with reference to specific implementation examples and the accompanying drawings.

[0043] First, let's take a look at the appendix. train.py is the runtime file used for training and can be used directly for training. predict.py is used for inference, which is used to obtain the model for inference. qtapp.py is the graphical interface file, which provides a graphical display of predict.py. normalize.py is used for normalization and removing redundant points. videotonpy.py converts the video into the input .npy format for dataset creation.

[0044] The specific steps include:

[0045] Step 1: Based on yolov8-pose posture recognition, extract the character's skeleton diagram and obtain the coordinates of the character's key nodes.

[0046] After inputting a model video and a sequence of video frames to be detected, the pre-trained yolov8-pose algorithm generates the coordinates of the person in each frame. This patented post-processing code can be used to convert this into a sequence of key points in the person's skeleton. This ensures that only one person appears in the video. This patent only covers single-person detection and does not involve tracking and identification.

[0047] Step 2: Process the obtained coordinate points using coordinate deletion and coordinate normalization.

[0048] After normalizing the obtained skeleton key point sequence, the redundant points of the face are deleted.

[0049] Step 3: Input the deleted coordinate points into the STGCN+ double-layer fully convolutional twin network to obtain the judgment result.

[0050] Inputting the resulting skeleton coordinate point sequence into the STGCN yields corresponding motion features with a dimension of (64, 256, 1, 1). The motion feature data from both videos is then fed into a two-layer fully convolutional network. The two-layer fully convolutional network processes and classifies the input motion features. Based on the model output, it determines whether the video matches the standard motion, thereby assisting patients in determining their own motion behavior.

[0051] So far, the specific implementation process of the present invention has been described.

Claims

1. A rehabilitation training system based on a key point recognition model and a graph neural network, characterized in that: It includes the YOLOv8-Pose posture recognition model, STGCN spatiotemporal graph convolution and double-layer fully convolutional twin network; first, the YOLOv8-Pose posture recognition model is used to accurately detect human posture and provide basic data for human posture motion analysis; then, the improved STGCN spatiotemporal graph convolution and double-layer fully convolutional twin network based on spatiotemporal attention perform motion recognition based on these features of human posture and evaluate the correctness of rehabilitation movements, thereby achieving accurate monitoring and evaluation of rehabilitation progress.

2. A rehabilitation training system based on a key point recognition model and a graph neural network according to claim 1, characterized in that: The YOLOv8-Pose pose recognition model is used for pose estimation tasks and can identify human key points in images, with a total of 17 key points.

3. A rehabilitation training system based on a key point recognition model and a graph neural network according to claim 1, characterized in that: The key point coordinate shape of the original output in the YOLOv8-Pose posture recognition model is (1,17,2), which is set in a list of length N; 1 is a frame, 17 is 17 key points, and 2 is the x coordinate and y coordinate; then, the output is post-processed; for the uniformity of the input of the YOLOv8-Pose posture recognition model, the longest target number of frames in the model video is set during training, and the length of the model video is the final length during inference; when the number of frames exceeds the set number of frames, the excess part is cut off; when the number of frames is insufficient, the frames are supplemented backward based on the key points of the last frame; then, the key points of all frames are spliced ​​from the 0th dimension to form a key point sequence, and the shape of the output key point sequence is (M,17,2).

4. A rehabilitation training system based on a key point recognition model and a graph neural network according to claim 1, characterized in that: In the preparation of the dataset for skeleton recognition among 17 key points, a lightweight YOLOv8n-Pose model is used for skeleton recognition.

5. The rehabilitation training system based on key point recognition model and graph neural network according to claim 1, characterized in that: The normalization and coordinate point deletion algorithm MODL is used to standardize and lightweight the obtained skeleton, reducing performance usage while improving model detection results; The normalization algorithm is described as follows: In practical applications, the distance between the human body and the camera changes continuously during movement. Different image sizes also lead to inconsistent pixel units on the x- and y-axes. Differences in height and weight among individuals can affect coordinate consistency. Specifically, for an image with a width of w and a height of h, there are 13 coordinate points (xi, yi). The normalization process involves dividing all xi coordinate values ​​by the image width w and all yi coordinate values ​​by the image height h. This process eliminates the need for coordinates to be standardized, as shown in the following formula: That is to say, the normalization of the skeleton coordinates is completed.

6. A rehabilitation training system based on a key point recognition model and a graph neural network according to claim 5, characterized in that: The coordinate point deletion algorithm is described as follows: by deleting extra coordinates, the coordinate point dimension is reduced from 17 to 13; at the same time, the skeleton graph connection edges are modified in the model to ensure that the model can accurately capture the relationship between each coordinate point when processing data, thereby laying a solid foundation for subsequent classification tasks.

7. The rehabilitation training system based on key point recognition model and graph neural network according to claim 5, characterized in that: STGCN is used to process spatiotemporal sequence data and was initially used in the field of traffic prediction. By combining graph convolution and gated temporal convolution, it can effectively capture spatial and temporal dependencies, thereby improving prediction accuracy. A two-layer fully convolutional network is designed at the end to output the judgment results. The composition structure is as follows: Spatiotemporal convolutional block: It consists of a "sandwich" structure consisting of two gated temporal convolutional layers and one spatial graph convolutional layer. The gated temporal convolutional layer contains one-dimensional convolution and gated linear unit (GLU). The spatial graph convolutional layer is used to convolve graph structure data to extract spatial features. Stacked structure: multiple spatiotemporal convolution blocks are stacked, with two layers stacked to fully extract spatiotemporal features; Attention module: Through the attention mechanism, the model's attention to specific features is strengthened; Convolutional layer: The network contains two convolutional layers. On the two branches of the twin network, the convolutional layer performs convolution on the input image to extract image features and complete classification at the same time.

8. The rehabilitation training system based on key point recognition model and graph neural network according to claim 7, characterized in that: The STGCN part takes the relationship between skeleton nodes as edges and takes the image sequence in time series as input; The spatial graph convolution layer performs convolution operations on the graph structure data to extract spatial features. The gated temporal convolution layer is used to process time series information and capture temporal dynamics. After stacking multiple spatiotemporal convolution blocks, the spatiotemporal features are fully extracted. Finally, the output layer is further processed by the temporal convolution layer and the fully connected layer. The two-layer fully convolutional twin network will fuse the two images. In each branch, the image first passes through the convolution layer, where the convolution kernel slides over the image to perform convolution operations and extract the low-level features of the image. Then, it passes through a nonlinear activation function to increase the nonlinear expression capability of the network. Finally, the features extracted from the two branches are fused and classified to measure the similarity between the two images, obtaining the final output result, which is used to determine whether the images are similar and perform image matching tasks.

9. The rehabilitation training system based on key point recognition model and graph neural network according to claim 8, characterized in that: The two-layer fully convolutional network is a deep learning network designed for image classification. It uses two fully convolutional layers to replace the traditional fully connected layers, thereby refining the classification task from the entire image level to the pixel level. The optimization function selects the cross entropy function; the formula is as follows: The optimizer uses adamw; The learning rate is set to 1e-3; AdamW is an improvement on the Adam optimizer, adding a weight decay mechanism to prevent model overfitting; Self-made video action feature dataset.

10. The rehabilitation training system based on key point recognition model and graph neural network according to claim 1, characterized in that: The specific steps include: Step 1: Based on yolov8-pose posture recognition, extract the character's skeleton diagram and obtain the coordinates of the character's key nodes; After inputting the model video and the video frame sequence to be detected, the pre-trained yolov8-pose will give the character coordinate points of each frame; it is converted into a sequence of character skeleton key points through post-processing code; Step 2: Process the obtained coordinate points by using coordinate deletion and coordinate normalization; After normalizing the obtained skeleton key point sequence, the redundant points of the face are deleted; Step 3: Input the deleted coordinate points into the STGCN+ double-layer fully convolutional twin network to obtain the judgment result; Inputting the obtained skeleton coordinate point sequence into STGCN will obtain the corresponding action features with a dimension of (64, 256, 1, 1); the action feature data of the two videos will be further input into the two-layer fully convolutional network; the two-layer fully convolutional network will process and classify the input action features, and output whether the action in the video meets the standard, thereby assisting in judging the own action behavior.

Images