Intelligent rehabilitation training evaluation method based on multi-scale spatio-temporal graph convolution network

Abstract

The present application relates to the technical field of training rehabilitation evaluation, and particularly relates to an intelligent rehabilitation training evaluation method based on a multi-scale spatio-temporal graph convolution network, which comprises: inputting a human posture sequence to be evaluated into a trained rehabilitation evaluation model to output a rehabilitation evaluation prediction result; the processing steps of the rehabilitation evaluation model comprise: extracting joint modal feature data and skeletal modal feature data of each frame of data in the human posture sequence; capturing global dependency and short-term time sequence association in the joint modal feature sequence and the skeletal modal feature sequence respectively through a plurality of multi-scale spatio-temporal modeling modules connected in cascade to obtain joint modal features and skeletal modal features; inputting the joint modal features and the skeletal modal features after fusion into two fully connected layers for processing to obtain a first prediction result and a second prediction result and to sum the first prediction result and the second prediction result. The present application can improve the accuracy and comprehensiveness of rehabilitation training evaluation and significantly reduce training and deployment costs.

Description

Technical Field

[0001] This invention relates to the field of training and rehabilitation assessment technology, specifically to an intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks. Background Technology

[0002] Rehabilitation training is crucial for patients with neurological or musculoskeletal disorders to regain physical function. Because effective rehabilitation requires long-term, consistent practice, home-based rehabilitation has become a practical and easily implemented approach. However, without timely professional guidance, patients are prone to making mistakes during home rehabilitation exercises. Therefore, automated rehabilitation training assessment plays a key role by providing objective, real-time feedback, and this area has received increasing research attention.

[0003] Early rehabilitation training assessment methods relied primarily on wearable sensors, which were inconvenient to use. In contrast, vision-based methods, requiring only cameras and intelligent algorithms, offer a simple and easy-to-implement solution. These methods typically use human posture sequences as input and employ various vision algorithms to automate training and assessment. Some methods treat it as a regression problem, generating continuous scores to reflect movement quality; while others use classification models to directly determine the correctness of the training movements represented by the skeletal sequence.

[0004] Existing paradigms for classification-based rehabilitation training assessment mainly include two types: 1) Classification-based training and testing (ESTT); 2) Unified training and classification testing (UTEST). For example... Figure 1 As shown in (a), the ESTT paradigm requires independent training and testing of the model for each training action, increasing computational costs. Figure 1 As shown in (b), although the UTEST paradigm integrates data from all action types during the training phase, testing still requires separate testing for each training action type. This necessitates establishing independent deployment schemes for different training action types, increasing the complexity of the evaluation process. Specifically, Pnorm achieves viewpoint-independent skeletal action recognition by jointly modeling spatiotemporal features; Zhang et al. proposed a neural network that can adaptively adjust the virtual viewpoint to enhance recognition robustness; Zheng et al. combined a spatiotemporal graph convolutional network (ST-GCN) with rotation-invariant descriptors to improve the adaptability to viewpoint changes; and Karlov et al. used a supervised contrastive learning framework to train a unified ST-GCN model across action types.

[0005] Despite the progress made by the existing solutions, there are still significant limitations: 1) Existing paradigms result in high model training and deployment costs. ESTT requires building multiple models for different action types, and UTEST still requires deployment by action type. 2) Existing methods lack joint-skeletal multimodal fusion, making it difficult to optimize the representation of human movement patterns. 3) Existing methods lack multi-scale temporal modeling, focusing only on global dependencies while ignoring short-term temporal correlations. Summary of the Invention

[0006] To address the shortcomings of the existing technologies, the technical problem this invention aims to solve is: how to provide an intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks, which achieves model training based on the Absolutely Unified Training and Testing Paradigm (AUTT); integrates complementary information from joints and bones through a bimodal fusion network to comprehensively understand movement patterns; synchronously models global dependencies and short-term temporal correlations through multi-scale spatiotemporal modeling; and maintains semantic consistency while enhancing data diversity through a label-guided temporal fusion strategy (LGTM), thereby improving the accuracy and comprehensiveness of rehabilitation training assessment and significantly reducing training and deployment costs.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] Intelligent rehabilitation training assessment methods based on multi-scale spatiotemporal graph convolutional networks include:

[0009] S1: Obtain the human pose sequence to be evaluated;

[0010] S2: Input the human posture sequence to be evaluated into the trained rehabilitation assessment model and output the corresponding rehabilitation assessment prediction results;

[0011] The steps involved in training a rehabilitation assessment model include:

[0012] S201: Obtain a dataset containing human pose sequences and their labels for various training action types, and divide the dataset into a training set and a test set;

[0013] S202: Use the human posture sequences of all training movement types in the training set as input for unified training of the rehabilitation assessment model;

[0014] S203: Extract joint modal feature data and skeletal modal feature data from each frame of the human posture sequence to obtain joint modal feature sequence and skeletal modal feature sequence;

[0015] S204: By cascading several multi-scale spatiotemporal modeling modules, global dependencies and short-term temporal correlations in the joint modal feature sequence are captured synchronously to obtain joint modal features;

[0016] S205: By cascading several multi-scale spatiotemporal modeling modules, global dependencies and short-term temporal correlations in the skeletal modal feature sequence are captured simultaneously to obtain skeletal modal features;

[0017] S206: After fusing and average pooling the joint modal features and skeletal modal features, the data is input into two fully connected layers for processing to obtain the first prediction result and the second prediction result.

[0018] S207: Calculate the loss function based on the first and second prediction results, and optimize the parameters of the rehabilitation assessment model;

[0019] S208: Repeat steps S202 to S207 to iteratively train the rehabilitation assessment model until it converges or reaches the maximum number of iterations;

[0020] S209: Use human posture sequences of all training action types in the test set to uniformly test the trained rehabilitation assessment model; during the test, sum the first and second prediction results output by the two fully connected layers to obtain the rehabilitation assessment prediction result output by the rehabilitation assessment model.

[0021] Preferably, in step S2, during the training process, the joint modality feature sequence and the skeletal modality feature sequence are augmented using a label-guided temporal fusion strategy;

[0022] The data augmentation process based on a label-guided time-series fusion strategy includes the following steps:

[0023] S2031: Obtain the included A dataset of samples and their corresponding tags Each sample includes Frame data, each frame includes There are 1 joint point, each joint point is composed of 1 joint point. 3D coordinate representation;

[0024] S2032: Organize the dataset according to the labels The samples in the data are divided into the correct subset of samples. and error sample subset Two subsets;

[0025] The formula is expressed as:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] In the formula: and Representing datasets respectively Correct and incorrect samples; Indicates the first The label of each sample This indicates that it is correct. The representation is incorrect;

[0031] S2033: For the correct sample subset Perform time-series fusion to achieve data augmentation: from the correct sample subset Two samples were randomly selected from the middle. and Temporal downsampling is performed using bilinear interpolation; the downsampled sequences are then concatenated to form an augmented correct sample subset. ;

[0032] The formula is expressed as:

[0033] ;

[0034] In the formula: This indicates a time-series downsampling operation implemented using bilinear interpolation; Represents the time series length after downsampling, and its value is randomly sampled from a beta distribution; symbol This indicates splicing along the time dimension; Indicates from 1 to The index of the random permutation sequence;

[0035] S2034: For the subset of erroneous samples Performing temporal fusion to achieve data augmentation: from a subset of erroneous samples Two samples were randomly selected from the middle. and Temporal downsampling is performed using bilinear interpolation; the downsampled sequences are then concatenated to form an augmented error sample subset. ;

[0036] The formula is expressed as:

[0037] ;

[0038] S2035: Based on augmented correct sample subsets and augmented error sample subset Building an augmented dataset ;

[0039] The formula is expressed as:

[0040] ;

[0041] .

[0042] Preferably, in step S204, the multi-scale spatiotemporal modeling module includes three cascaded spatiotemporal feature extraction units;

[0043] The processing steps for each spatiotemporal feature extraction unit include:

[0044] S2041: The spatial feature sequence is obtained by performing topological perceptual modeling on the input through the spatial graph convolution module;

[0045] S2042: After adding the spatial feature sequence to the input of the multi-scale spatiotemporal modeling module element by element, the sequence is then processed through the recombination operation layer to obtain the small-scale recombined feature sequence.

[0046] S2043: Temporal dynamic features are extracted from the recombined feature sequence at a small time scale using the temporal graph convolution module to obtain the temporal dynamic feature sequence;

[0047] S2044: The temporal dynamic feature sequence is scaled back by the recovery operation layer to obtain the recovered feature sequence;

[0048] S2045: The output of the spatiotemporal feature extraction unit is obtained by adding the recovered feature sequence to the input of the multi-scale spatiotemporal modeling module element by element.

[0049] Preferably, in step S2041, the spatial graph convolution module includes three branches; the first and second branches each include cascaded 1×1 convolutional layers and temporal pooling layers; the third branch includes a 1×1 convolutional layer.

[0050] The outputs of the first and second branches are input together into the correlation modeling module for correlation modeling, and then input into the cascaded linear projection layer and topology refinement module.

[0051] The outputs of the topology graph refinement module and the third branch are fused to obtain the spatial feature sequence output by the spatial graph convolution module.

[0052] Preferably, in step S2041, the processing steps of the spatial graph convolution module include:

[0053] S20411: Sequence of human poses The first in Frame data Input spatial graph convolution module; where Representing data The number of key points, Representing data The number of channels;

[0054] S20412: Transfer data Input the third branch to obtain intermediate features ;in This indicates a feature extraction operation performed through a 1×1 convolutional layer.

[0055] S20413: Transfer data After each joint in the model is linearly projected through the first and second branches, it is input into the correlation modeling module to perform channel-by-channel joint correlation modeling, thus obtaining the correlation modeling results. ;

[0056] The formula is expressed as:

[0057] ;

[0058] In the formula: , Representing data The joints in; , Indicates the joint , Perform linear projection to reduce feature dimensionality; This represents the activation function that introduces a nonlinear projection;

[0059] S20414: Correlation Modeling Results Using a Linear Projection Layer Perform a linear projection to obtain the correlation tensor. Correlation tensor vectors in Indicates joint With joints The final channel-by-channel correction relationship between them;

[0060] The formula is expressed as:

[0061] ;

[0062] In the formula: Indicates a linear projection layer;

[0063] S20415: Relevance tensor refined using the topology graph refinement module The data is then refined to obtain a channel-by-channel topology map. ;

[0064] The formula is expressed as:

[0065] ;

[0066] In the formula: This represents the learnable scaling factor, used to adjust the thinning intensity; This indicates element-wise addition implemented via a broadcast mechanism; An adjacency matrix representing the topological structure of a sequence of human poses;

[0067] S20416: The intermediate feature output from the third branch Channel-by-channel topology map output by the topology refinement module Perform channel-by-channel feature fusion to generate output data. Spatial features ; where the channel-by-channel topology diagram Each channel in the matrix corresponds to a refined topology matrix. , Using topological matrix Independent fusion intermediate features The spatial features are obtained by stitching together the feature data of each channel and all the fusion results. ;

[0068] The formula is expressed as:

[0069] ;

[0070] In the formula: , express The Middle Feature data of each channel; symbols Indicates a splicing operation;

[0071] S20417: Human posture sequence Each frame of data is processed through steps S20411 to S20416 to obtain the spatial features of each frame of data; the sequence composed of all spatial features is used as the spatial feature sequence output by the spatial graph convolution module.

[0072] Preferably, in step S2042, the recombination operation layer is used to divide the input feature sequence into finer scales along the time dimension through recombination operations;

[0073] The formula is expressed as:

[0074] ;

[0075] In the formula: Indicates a restructuring operation; Indicates the first A time scale; The number of frames representing the feature sequence; This indicates the number of key points in each frame of data; This indicates the number of channels in each frame of data.

[0076] Preferably, in step S2043, the time-map convolution module includes four branches;

[0077] The first, second, and third branches each consist of cascaded 1×1 convolutional layers and ReLU activation functions; the fourth branch consists of a 1×1 convolutional layer.

[0078] The output of the first branch is connected to a max pooling layer; the outputs of the second and third branches are each connected to a dynamic temporal graph convolutional layer.

[0079] After element-wise summing of the outputs of the max pooling layer, the two dynamic temporal graph convolutional layers, and the fourth branch, a 1×1 convolutional layer is used for feature fusion to obtain the temporal dynamic feature sequence output by the temporal graph convolutional module.

[0080] Preferably, in step S2043, the processing steps of the dynamic temporal graph convolutional layer in the temporal graph convolution module include:

[0081] S20431: The feature sequence As input to the dynamic temporal graph convolutional layer; where Indicates time Corresponding features Indicates the number of frames;

[0082] S20432: Generating timing based on sampling strategy of dynamic temporal graph convolutional layer sampling location set ;

[0083] The sampling strategy of dynamic temporal graph convolutional layers is expressed as follows:

[0084] ;

[0085] In the formula: Indicates the first The first frame centered on the convolution operation Each frame position; Indicates the feeling field; Indicates feeling wild The first One trainable offset parameter;

[0086] S20433: Time-based one-dimensional convolution sampling location set For feature sequences At any moment Perform feature aggregation along the time dimension to obtain time-aggregated features. ;

[0087] The formula is expressed as:

[0088] ;

[0089] In the formula: Indicates the kernel size; Indicates the first convolution kernel One weight parameter; This represents a sampling function implemented through linear interpolation; Indicates time Corresponding features;

[0090] S20434: For the characteristic sequence At each time step, S20432 to S20433 are executed to obtain the corresponding temporal aggregated features; the sequence composed of the temporal aggregated features of all time steps is used as the dynamic temporal graph convolutional sequence output by the dynamic temporal graph convolutional layer. .

[0091] Preferably, in step S2044, the recovery operation layer is used to restore the input feature sequence to its original scale through a recovery operation;

[0092] The formula is expressed as:

[0093] ;

[0094] In the formula: Indicates a recovery operation; Indicates the first A time scale; The number of frames representing the feature sequence; This indicates the number of key points in each frame of data; This indicates the number of channels in each frame of data.

[0095] Preferably, in step S207, the loss function is calculated using the following formula. :

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] In the formula: , These represent the loss functions corresponding to the two fully connected layers; , Each represents containing Sample small batch dataset The first and second prediction results output by two fully connected layers; Indicates the number of categories; Indicates to The probability vector obtained after softmax normalization is the first... One element; This indicates the label after label smoothing. One component; Indicates sample Unique hot-coded real labels; Indicates an indicator function, when The value is 1 if the condition is met, and 0 otherwise. This is the smoothing coefficient.

[0101] Compared with existing technologies, the intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks in this invention has the following advantages:

[0102] Existing classification-based training and testing (ESTT) methods require training a separate model for each training action type during training. This not only hinders the model from optimizing parameters using the complete training set, thus limiting performance, but also significantly increases training computational costs and testing complexity. Existing unified training and classification-based testing (UTEST) methods, while performing unified training on the training set, still conduct testing separately for each training action type, requiring independent deployment schemes for different training action types and increasing the complexity of the evaluation process. To address these issues, this invention proposes... Figure 1 (c) shows the training of the rehabilitation assessment model (MSST-GCN) based on the Absolutely Uniform Training and Testing Paradigm (AUTT). During training, the model is uniformly trained on a training set containing all training movement types and then uniformly tested on a test set containing all training movement types. This end-to-end approach maintains the rigor of the assessment without requiring customized processes for each training movement type. It is not only more in line with the actual application scenarios of rehabilitation training assessment, but also reduces the complexity of model deployment, significantly reduces training and deployment costs, and maintains high flexibility across movement types.

[0103] In the rehabilitation assessment model (MSST-GCN) proposed in this invention, the input data undergoes augmentation processing, and features are extracted from joint and skeletal modalities through two branches. These extracted features are then fused and dimensionality-reduced before being input into two independent classifiers. The outputs are then integrated to generate the final assessment prediction. This model, through a dual-modal fusion network, integrates complementary information from joints and bones to comprehensively understand movement patterns, thereby improving the accuracy and comprehensiveness of rehabilitation training assessment.

[0104] Multi-scale feature extraction techniques have been extensively studied in the field of computer vision, typically employing hierarchical network architectures to expand the receptive field and capture spatial correlations in image data. However, existing multi-scale methods are not directly applicable to rehabilitation training assessment because human posture sequences only exhibit inter-frame temporal correlations and are composed of discrete joints in the spatial dimension. To address these issues, this invention proposes a Multi-Scale Spatiotemporal Modeling Module (MSSTMB). MSSTMB simultaneously models global dependencies and short-term temporal correlations. This dual focus is crucial for rehabilitation assessment: global dependencies track macroscopic evolutionary patterns such as endurance performance and stage transitions, while short-term correlations capture microscopic movement patterns such as joint stability and movement fluency. This enables MSSTMB to achieve accurate and comprehensive assessment of training movements, thereby further improving the accuracy and comprehensiveness of rehabilitation training assessment.

[0105] To address the problem of scarce training data and enhance the robustness of models to temporal noise, this invention proposes a label-guided temporal fusion strategy (LGTM). This strategy enhances data diversity while maintaining semantic consistency, alleviating the problem of scarce training data in rehabilitation training assessment tasks and improving the robustness of models to temporal noise in human posture sequences.

[0106] Extensive experiments on the IRDS, UI-PRMD, and REHAB24-6 datasets demonstrate that MSST-GCN achieves excellent performance. These results highlight the potential of MSST-GCN as a reliable computational foundation for scalable, automated home rehabilitation systems. Attached Figure Description

[0107] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0108] Figure 1 To compare different rehabilitation training assessment paradigms.

[0109] Figure 2 This is a network structure diagram of the rehabilitation assessment model (MSST-GCN) and its components.

[0110] Figure 3 For parameters Evaluation of the methods.

[0111] Figure 4 For parameters Evaluation of the methods.

[0112] Figure 5 The ROC and PR curves in the AUTT paradigm on the IRDS dataset.

[0113] Figure 6The ROC and PR curves in the AUTT paradigm are shown on the UI-PRMD (Kinect) dataset.

[0114] Figure 7 The ROC and PR curves in the AUTT paradigm are shown on the UI-PRMD(Vicon) dataset.

[0115] Figure 8 The ROC and PR curves in the AUTT paradigm on the REHAB24-6 dataset.

[0116] Figure 9 This is a visualization of the multi-scale spatiotemporal graph convolutional network on the REHAB24-6 dataset. Correct samples are represented by uniform joint size and color; for incorrect samples, larger joint size and darker color indicate a higher contribution to the model's prediction. Detailed Implementation

[0117] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0118] The following detailed explanation illustrates the specific implementation methods:

[0119] Example:

[0120] Early rehabilitation training assessment systems primarily rely on wearable sensors for data collection. To improve ease of use, many researchers have turned their attention to vision-based methods. Sequences of participant postures acquired from standard cameras using Kinect devices or posture estimation models typically serve as input data for these vision-based methods. Current vision-based assessment methods are mainly implemented using regression or classification frameworks.

[0121] This invention focuses on a classification-based method for assessing rehabilitation training. Related research includes: Li et al. achieved viewpoint-independent skeletal movement recognition by jointly modeling spatiotemporal features; to improve recognition robustness, Zhang et al. proposed an adaptive neural network that can dynamically adjust the virtual viewpoint; and Zheng et al. enhanced viewpoint robustness by combining an ST-GCN baseline model with rotation-invariant descriptors. It should be noted that the above methods all employ the ESTT paradigm, resulting in high training and deployment costs. Although Karlov et al. trained a unified ST-GCN model across rehabilitation training movements using a supervised contrastive learning framework, their testing phase still employed a classification scheme, which inadvertently increased deployment costs.

[0122] To address the aforementioned problems, this embodiment discloses an intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks, including:

[0123] S1: Obtain the human pose sequence to be evaluated;

[0124] In this embodiment, a human pose sequence is generated by extracting human pose features from a video image sequence using a human pose estimation model.

[0125] S2: Input the human posture sequence to be evaluated into the trained rehabilitation assessment model (MSST-GCN) and output the corresponding rehabilitation assessment prediction results;

[0126] like Figure 2 As shown, the processing steps for training the rehabilitation assessment model include:

[0127] S201: Obtain a dataset containing human pose sequences and their labels for various training action types, and divide the dataset into a training set and a test set;

[0128] S202: Use the human posture sequences of all training movement types in the training set as input for unified training of the rehabilitation assessment model;

[0129] S203: Extract joint modal feature data and skeletal modal feature data from each frame of the human posture sequence to obtain joint modal feature sequence and skeletal modal feature sequence;

[0130] S204: By cascading several multi-scale spatiotemporal modeling modules (MSSTMB), global dependencies and short-term temporal correlations in the joint modal feature sequence are captured simultaneously to obtain joint modal features;

[0131] S205: By cascading several multi-scale spatiotemporal modeling modules (MSSTMB), global dependencies and short-term temporal correlations in the skeletal modal feature sequence are captured simultaneously to obtain skeletal modal features;

[0132] S206: After fusing and average pooling the joint modal features and skeletal modal features, the data is input into two fully connected layers for processing to obtain the first prediction result and the second prediction result.

[0133] S207: Calculate the loss function based on the first and second prediction results, and optimize the parameters of the rehabilitation assessment model;

[0134] S208: Repeat steps S202 to S207 to iteratively train the rehabilitation assessment model until it converges or reaches the maximum number of iterations;

[0135] S209: Use human posture sequences of all training action types in the test set to uniformly test the trained rehabilitation assessment model; during the test, sum the first and second prediction results output by the two fully connected layers to obtain the rehabilitation assessment prediction result output by the rehabilitation assessment model.

[0136] During the inference process, the human posture sequence to be evaluated is input into the trained rehabilitation assessment model: joint modal feature data and skeletal modal feature data are extracted from each frame of the human posture sequence to obtain joint modal feature sequences and skeletal modal feature sequences; global dependencies and short-term temporal correlations in the joint modal feature sequences are captured synchronously through several cascaded multi-scale spatiotemporal modeling modules to obtain joint modal features; global dependencies and short-term temporal correlations in the skeletal modal feature sequences are captured synchronously through several cascaded multi-scale spatiotemporal modeling modules to obtain skeletal modal features; after fusing and average pooling the joint modal features and skeletal modal features, they are input into two fully connected layers for processing to obtain the first prediction result and the second prediction result; the first prediction result and the second prediction result are summed to obtain the rehabilitation assessment prediction result output by the rehabilitation assessment model.

[0137] To better illustrate the technical solution of the present invention, this embodiment is described in the following parts.

[0138] I. Problem Definition

[0139] Given a rehabilitation training dataset ,in A set representing sequences of human postures. A set representing real labels. The goal of rehabilitation training assessment is to analyze human postural sequences. The exercise is classified into two categories: incorrect or correct. According to clinical practice, incorrect training movements are considered positive examples. Correct training movements are considered negative examples (), while correct training movements are considered negative examples (). ). Represents the dataset The number of samples, each human pose sequence Include Frame data, each frame of data is composed of It consists of 10 joints, each joint is composed of 10 joints. 3D coordinate representation.

[0140] in Represents the frame index (time). ), To add self-connected adjacency matrices, Represents the diagonal elements of the degree matrix. The weight matrix is ​​a learnable matrix. This represents a non-linear activation function. It's important to note the spatial features of each frame. It is calculated independently.

[0141] II. Data Augmentation

[0142] To address the scarcity of training data and enhance the model's robustness to temporal noise, this invention proposes a temporal fusion strategy that augments the training set by enhancing human pose sequences. However, directly performing cross-sample temporal fusion within a mini-batch may result in semantically ambiguous data. For example, correctly fused samples... With error samples Fusion can lead to label conflicts. To address this issue, this invention proposes a label-guided temporal fusion strategy (LGTM) to ensure semantic consistency during the data augmentation process.

[0143] Specifically, during the training process, a label-guided temporal fusion strategy is used to augment the joint modality feature sequences and the skeletal modality feature sequences.

[0144] The data augmentation process based on a label-guided time-series fusion strategy includes the following steps:

[0145] S2031: Obtain the contents A (mini-batch) dataset of samples and their corresponding tags Each sample includes Frame data, each frame includes There are 10 key points, each key point is composed of 10 key points. 3D coordinate representation;

[0146] S2032: Analyze the dataset according to its labels The samples in the data are divided into the correct subset of samples. and error sample subset Two subsets;

[0147] The formula is expressed as:

[0148] ;

[0149] ;

[0150] ;

[0151] ;

[0152] In the formula: and Representing datasets respectively Correct and incorrect samples; Indicates the first The label of each sample This indicates that it is correct. The representation is incorrect;

[0153] S2033: For the correct sample subset Perform time-series fusion to achieve data augmentation: from the correct sample subset Two samples were randomly selected from the middle. and Temporal downsampling is performed using bilinear interpolation; the downsampled sequences are then concatenated to form an augmented correct sample subset. ;

[0154] The formula is expressed as:

[0155] ;

[0156] In the formula: This indicates a time-series downsampling operation implemented using bilinear interpolation; Represents the time series length after downsampling, and its value is randomly sampled from a beta distribution; symbol This indicates splicing along the time dimension; Indicates from 1 to The index of the random permutation sequence;

[0157] S2034: For the subset of erroneous samples Performing temporal fusion to achieve data augmentation: from a subset of erroneous samples Two samples were randomly selected from the middle. and Temporal downsampling is performed using bilinear interpolation; the downsampled sequences are then concatenated to form an augmented error sample subset. ;

[0158] The formula is expressed as:

[0159] ;

[0160] S2035: Based on augmented correct sample subsets and augmented error sample subset Constructing augmented (mini-batch) training data ;

[0161] The formula is expressed as:

[0162] ;

[0163] .

[0164] Ultimately, and Combining the data yields augmented (mini-batch) training data. .

[0165] This invention proposes a label-guided temporal fusion strategy (LGTM), which enhances data diversity while maintaining semantic consistency, alleviates the problem of scarce training data in rehabilitation training assessment tasks, and improves the robustness of the model to temporal noise in human posture sequences.

[0166] III. Multi-scale Spatiotemporal Modeling Module

[0167] The Multi-Scale Spatiotemporal Modeling Module (MSSTMB) is a core component of MSST-GCN. Unlike existing methods that only capture inter-frame global dependencies, MSSTMB can simultaneously model global dependencies and short-term temporal correlations. This dual focus is crucial for rehabilitation assessment: global dependencies track macroscopic evolutionary patterns such as endurance performance and stage transitions, while short-term correlations capture microscopic movement patterns such as joint stability and movement fluency. Therefore, MSSTMB enables accurate and comprehensive assessment of training movements.

[0168] MSSTMB consists of three cascaded spatiotemporal feature extraction units, with the number of feature channels in the three spatiotemporal feature extraction units increasing in the order of 64, 128, and 256.

[0169] The processing steps for each spatiotemporal feature extraction unit include:

[0170] S2041: The input is topologically modeled using a spatial graph convolutional module (SGCM) to obtain a spatial feature sequence;

[0171] S2042: After adding the spatial feature sequence to the input of the multi-scale spatiotemporal modeling module element by element, the sequence is then processed through the recombination operation layer to obtain the small-scale recombined feature sequence.

[0172] The recombination operation layer is used to divide the input feature sequence into finer scales along the time dimension through recombination operations; this fine-grained division enables the subsequent temporal map convolutional module (TGCM) to capture inter-frame correlation details that are crucial for recognizing micro motion patterns.

[0173] The formula is expressed as:

[0174] ;

[0175] In the formula: Indicates a restructuring operation; Indicates the first A time scale; The number of frames representing the feature sequence; This indicates the number of key points in each frame of data; This indicates the number of channels in each frame of data.

[0176] S2043: Temporal dynamic features are extracted from the recombined feature sequence at a small time scale using the temporal graph convolution module to obtain the temporal dynamic feature sequence; after the SGCM completes the topology-aware feature extraction, the TGCM processes the generated overall feature sequence to model global dependencies.

[0177] S2044: The temporal dynamic feature sequence is scaled back by the recovery operation layer to obtain the recovered feature sequence;

[0178] The recovery operation layer is used to restore the input feature sequence to its original scale through recovery operations;

[0179] The formula is expressed as:

[0180] ;

[0181] In the formula: Indicates a recovery operation; Indicates the first A time scale; The number of frames representing the feature sequence; This indicates the number of key points in each frame of data; This indicates the number of channels in each frame of data.

[0182] S2045: The output of the spatiotemporal feature extraction unit is obtained by adding the recovered feature sequence to the input of the multi-scale spatiotemporal modeling module element by element.

[0183] like Figure 2 As shown in (b), each MSSTMB contains three parallel-operating units that operate simultaneously on a specific short timescale. Processing is performed at the full-sequence scale. As the network depth decreases layer by layer, MSSTMB is able to simultaneously capture multi-scale short-term associations and global dependencies.

[0184] Multi-scale feature extraction techniques have been extensively studied in the field of computer vision. These methods typically employ hierarchical network architectures to expand the receptive field, aiming to capture the inherent spatial correlations of image data. However, such designs are not suitable for rehabilitation training assessment tasks because skeletal sequences lack continuous spatial structure and only exhibit inter-frame temporal correlations. In contrast, the MSSTMB proposed in this invention introduces a multi-scale feature extraction mechanism through a recombination operation. This mechanism obtains different receptive fields in the temporal dimension by progressively dividing the temporal domain, thus naturally aligning with the characteristics of rehabilitation training data.

[0185] IV. Spatial Graph Convolution Module

[0186] The Spatial Graph Convolutional Module (SGCM) within the Multi-Scale Spatiotemporal Modeling Module (MSSTMB) is responsible for achieving topology-aware feature aggregation across skeletal joints. This requires leveraging biomechanical adjacency relationships in the skeletal topology graph to guide the modeling of inter-joint relationships. Standard GCN employs a channel-independent topology propagation mechanism, forcing all feature channels to share the same adjacency matrix. This architectural constraint struggles to adapt to the heterogeneity of kinematic patterns encoded across different channels, limiting its ability to capture limb-specific motor features in rehabilitation assessment.

[0187] To overcome these limitations, this invention introduces Channel Thinning Graph Convolution (CTR-GC) to construct SGCM.

[0188] Given an input sequence of human poses The Middle Frame data Spatial features output by the spatial graph convolution module The definition is as follows:

[0189] ;

[0190] In the formula: This indicates the feature extraction operation performed through the convolutional layer; This represents a correlation modeling operation; An adjacency matrix representing the topological structure of a sequence of human poses; This indicates a refinement operation on the adjacency matrix; This indicates channel-wise feature fusion. To achieve channel-wise feature fusion, a correlation modeling function is used. First, in each channel Learn the inter-joint correlations and generate channel-level correlation tensors. ; and then utilize For adjacency matrix Further refinement yields a channel-level topology graph. .

[0191] like Figure 2As shown in (c), the spatial graph convolution module includes three branches;

[0192] The first and second branches both consist of cascaded 1×1 convolutional layers and temporal pooling layers; the third branch consists of 1×1 convolutional layers.

[0193] The outputs of the first and second branches are input together into the correlation modeling module for correlation modeling, and then input into the cascaded linear projection layer and topology refinement module.

[0194] The outputs of the topology graph refinement module and the third branch are fused to obtain the spatial feature sequence output by the spatial graph convolution module.

[0195] Specifically, the processing steps of the spatial graph convolution module include:

[0196] S20411: Sequence of human poses The first in Frame data Input spatial graph convolution module; where Representing data The number of key points, Representing data The number of channels;

[0197] S20412: Transfer data Input the third branch to obtain intermediate features ;in This indicates a feature extraction operation performed through a 1×1 convolutional layer.

[0198] S20413: Transfer data After each joint in the model is linearly projected through the first and second branches, it is input into the correlation modeling module to perform channel-by-channel joint correlation modeling, thus obtaining the correlation modeling results. ;

[0199] The formula is expressed as:

[0200] ;

[0201] In the formula: , Representing data The joints in; , Indicates the joint , Perform linear projection to reduce feature dimensionality; This represents the activation function that introduces a nonlinear projection;

[0202] Where the formula calculate and The Euclidean distance along the channel dimension is calculated, and this distance is nonlinearly projected to obtain the node. and The joint relationships between them.

[0203] S20414: Correlation Modeling Results Using a Linear Projection Layer Perform a linear projection to obtain the correlation tensor. Correlation tensor vectors in Indicates joint With joints The final channel-by-channel correction relationship between them;

[0204] The formula is expressed as:

[0205] ;

[0206] In the formula: This represents a linear projection layer; to recover the dimension of the correlation results, a 1×1 convolutional layer is used. Perform linear projection.

[0207] S20415: Relevance tensor refined using the topology graph refinement module The data is then refined to obtain a channel-by-channel topology map. ;

[0208] The formula is expressed as:

[0209] ;

[0210] In the formula: This represents the learnable scaling factor, used to adjust the thinning intensity; This indicates element-wise addition implemented via a broadcast mechanism; An adjacency matrix representing the topological structure of a sequence of human poses;

[0211] S20416: The intermediate feature output from the third branch Channel-by-channel topology map output by the topology refinement module Perform channel-by-channel feature fusion to generate output data. Spatial features ; where the channel-by-channel topology diagram Each channel in the matrix corresponds to a refined topology matrix. , Using topological matrix Independent fusion intermediate features The spatial features are obtained by stitching together the feature data of each channel and all the fusion results. ;

[0212] The formula is expressed as:

[0213] ;

[0214] In the formula: , express The Middle Feature data of each channel; symbols Indicates a splicing operation;

[0215] S20417: Human posture sequence Each frame of data is processed through steps S20411 to S20416 to obtain the spatial features of each frame of data; the sequence composed of all spatial features is used as the spatial feature sequence output by the spatial graph convolution module.

[0216] V. Time-Graph Convolution Module

[0217] To meet the multi-scale temporal feature extraction requirements of the Multi-Scale Spatiotemporal Modeling Module (MSSTMB), this invention constructs a Temporal Graph Convolutional Module (TGCM) based on existing research. Unlike the original design, this invention employs element-wise summation followed by 1×1 convolution for feature fusion (replacing the concatenation operation), aiming to improve feature representation capabilities through spatial information integration and cross-channel interactive learning. Figure 2 As shown in (d), TGCM consists of four branches: one branch performs max pooling, two branches are equipped with Dynamic Temporal Graph Convolution (DTGC) components with different dilation rates, and the last branch is a residual connection. The inputs of all branches are channel-compressed through different 1×1 convolutional layers to reduce computational cost.

[0218] like Figure 2 As shown in (d), the time-graph convolution module includes four branches;

[0219] The first, second, and third branches each consist of cascaded 1×1 convolutional layers and ReLU activation functions; the fourth branch consists of a 1×1 convolutional layer.

[0220] The output of the first branch is connected to a max pooling layer; the outputs of the second and third branches are each connected to a dynamic temporal graph convolutional layer.

[0221] After element-wise summing of the outputs of the max pooling layer, the two dynamic temporal graph convolutional layers, and the fourth branch, a 1×1 convolutional layer is used for feature fusion to obtain the temporal dynamic feature sequence output by the temporal graph convolutional module.

[0222] In rehabilitation training assessment tasks, human pose sequences are obtained by sampling continuous motion in physical space at a fixed frame rate. This discrete sampling process inevitably leads to the loss of inter-frame physical kinematic information. Correspondingly, Static Temporal Graph Convolution (STGC) uses discrete fixed convolution kernel windows, which exacerbates this information loss when modeling skeletal sequences. To mitigate this problem, this invention employs a DTGC component to extract temporal features from skeletal sequences.

[0223] Given a feature sequence Specific frame index STGC uses the following kernel window to determine the frame positions involved in temporal convolution:

[0224] ;

[0225] In the formula: Indicates the first The first frame centered on the convolution operation Each frame position; The receptive field is defined by the kernel size. Representative at The first part of the intrinsic participation in the temporal convolution The position offset. For a convolution kernel of size 3, its corresponding receptive field is... In this configuration, STGC uses a fixed-position discretization method to define the sampling positions of temporal convolutions, which contradicts the goal of mitigating information loss.

[0226] In contrast, DTGC employs a dynamic, continuous receptive field, enabling more effective temporal modeling of discretely sampled human pose sequences. Assuming a dynamic, continuous receptive field... Inside Each sampling location participates in temporal convolution. First, in Uniform sampling within the range Each position offset is initialized to a trainable parameter. Then, the fixed position offset within the kernel window is... Replacing the above trainable positional parameters, the sampling strategy for DTGC is as follows:

[0227] ;

[0228] In the formula: Indicates feeling wild The first There are 1 trainable offset parameter.

[0229] In the formula kernel window, the sampling position The values ​​are real numbers and are dynamically optimized during training. This continuous sampling strategy, combined with linear interpolation for feature extraction, enables DTGC to mitigate the information loss caused by fixed discretization sampling of bone sequences.

[0230] Specifically, the processing steps of the dynamic temporal graph convolutional layer in the temporal graph convolution module include:

[0231] S20431: The feature sequence As input to the dynamic temporal graph convolutional layer; where Indicates time Corresponding features Indicates the number of frames;

[0232] S20432: Generating timing based on sampling strategy of dynamic temporal graph convolutional layer sampling location set ;

[0233] The sampling strategy of dynamic temporal graph convolutional layers is expressed as follows:

[0234] ;

[0235] In the formula: Indicates the first The first frame centered on the convolution operation Each frame position; The receptive field is defined by the kernel size. Indicates feeling wild The first One trainable offset parameter;

[0236] S20433: Time-based one-dimensional convolution sampling location set For feature sequences At any moment Perform feature aggregation along the time dimension to obtain time-aggregated features. ;

[0237] The formula is expressed as:

[0238] ;

[0239] In the formula: Indicates the kernel size; Indicates the first convolution kernel One weight parameter; This represents a sampling function implemented through linear interpolation; Indicates time Corresponding features;

[0240] S20434: For characteristic sequences At each time step, S20432 to S20433 are executed to obtain the corresponding temporal aggregated features; the sequence composed of the temporal aggregated features of all time steps is used as the dynamic temporal graph convolutional sequence output by the dynamic temporal graph convolutional layer. .

[0241] In STGC, the kernel size is controlled by hyperparameters to form a fixed, discrete receptive field; while in DTGC, the kernel size is only used to initialize the learnable sampling locations within the receptive field. These location parameters are dynamically adjusted during model training, thus forming a dynamically continuous receptive field. This mechanism enables DTGC to effectively mitigate the information loss problem when sampling discrete skeletal sequences.

[0242] VI. Unified Training and Testing Paradigm

[0243] To achieve effective training and testing of the model, the dataset is... Divided into training set With test set ,in and Considering the existence of various training movement types, and This can be further expressed as and ,in Represents a specific type of training movement. This represents the total number of training movement types.

[0244] Existing methods are mainly divided into two categories: 1) Class-Specific Training and Testing (ESTT); 2) Unified Training and Class-Specific Testing (UTEST). The ESTT method models each training action type separately; specifically, for a given type... The model in Training and The above test requires repeating this process for all training action types. This not only hinders the model from optimizing parameters using the complete training set, thus limiting performance, but also significantly increases training computational cost and testing complexity. While the UTEST method... While unified training is implemented, the testing phase is still conducted separately according to the type of training movement, which requires the establishment of independent deployment plans for different types of training movements, increasing the complexity of the evaluation process.

[0245] To address the aforementioned problems, this invention proposes a model based on the Absolutely Unified Training and Testing Paradigm (AUTT), which... After unified training, directly in A unified test is implemented. This end-to-end solution maintains the rigor of the evaluation while eliminating the need to customize processes according to the type of training movement. For example... Figure 3As shown, MSST-GCN is equipped with two fully connected layers as predictors to enhance representation capabilities. Let mini-batch training samples... The output after two fully connected layers is and The label smooth cross-entropy (LSCE) loss function is used to guide training.

[0246] Taking the first fully connected layer as an example, its LSCE loss function is defined as:

[0247] ;

[0248] In the formula: This indicates the number of classification categories in rehabilitation training assessment tasks. The value is 2; Indicates to The probability vector obtained after softmax normalization is the first... One element; This indicates the label after label smoothing. Each component.

[0249] ;

[0250] In the formula: It is a sample Unique hot-coded real labels; It is an indicator function, when The value is 1 if the condition is met, and 0 otherwise. This is the smoothing coefficient.

[0251] Similarly, the LSCE loss function of the second fully connected layer is defined as:

[0252] ;

[0253] In the formula: , Each represents containing Sample small batch dataset The first and second prediction results are output through two fully connected layers.

[0254] The final loss function can be defined as:

[0255] ;

[0256] During the training phase, the model uses the formulaic loss function as a guide. All samples are optimized to learn more generalizable parameters from data containing multiple training action types. During the testing phase, the outputs of the two fully connected layers are summed to generate the final prediction. This invention... The data from all training movement types are tested uniformly, which is more in line with the actual application scenario of rehabilitation training assessment, while reducing the complexity of model deployment.

[0257] VII. Experimental Instructions

[0258] To better illustrate the advantages of the technical solution of the present invention, the following experiment is disclosed in this embodiment.

[0259] 1. Dataset

[0260] This embodiment evaluates the effectiveness of the proposed MSST-GCN on three rehabilitation training datasets, including IRDS, UI-PRMD, and REHAB24-6.

[0261] The IRDS dataset contains nine training 3D skeleton sequences, acquired at 30fps from 15 patients and 14 healthy subjects using a Kinect sensor. Each sample is labeled with the training motion type and its correctness. Following established settings, 1,942 samples were used for training and 647 for testing.

[0262] The UI-PRMD dataset contains 10 training exercises performed by 10 healthy subjects, each exercise repeated 10 times in both correct and incorrect ways. It consists of two independent subsets: Kinect captures (30fps, 1,500 training / 500 test samples) and Vicon captures (100fps, 1,484 training / 495 test samples).

[0263] The REHAB24-6 dataset contains 1,072 samples of six rehabilitation training exercises performed by 10 subjects, acquired at 30fps in a laboratory setting using two RGB cameras and 16 motion capture cameras. All training exercises included both correct and incorrect versions performed under the supervision of a physical therapist. To maintain consistency with the previous two datasets, this invention uses their 3D skeletal data, containing 804 training samples and 268 test samples. This dataset presents challenges due to its multi-viewpoint nature, varying lighting, complex backgrounds, and the occasional presence of irrelevant individuals.

[0264] 2. Evaluation Indicators

[0265] This experiment uses classification accuracy and F1 score to evaluate model performance: accuracy measures overall prediction correctness, while the F1 score comprehensively considers the precision and recall of the positive class. Furthermore, receiver operating characteristic (ROC) curves and precision-recall (PR) curves are used for qualitative analysis of the model.

[0266] 3. Implementation details

[0267] Combination Figure 2As shown, each branch of MSST-GCN contains three MSSTMB modules with output channels of 64, 128, and 256, respectively. For 30 fps datasets (IRDS, UI-PRMD (Kinect), REHAB24-6), the scale parameters... ( The parameters were set to 10, 6, and 3 respectively; for the UI-PRMD (Vicon) dataset at 100 fps, the corresponding parameters were set to 20, 10, and 5. The model was trained for 100 epochs using the Adam optimizer, with a weight decay rate of 0.0005, a learning rate of 0.00035, and a smoothing factor in the loss function. To ensure a fair comparison, we followed the AUTT paradigm for data sampling, centralized processing, and a 3:1 training / test split.

[0268] 4. Parameter Analysis

[0269] 4.1 Parameters

[0270] Hyperparameters in the loss function As a smoothing factor, it is used to suppress overconfidence in predictions and enhance generalization ability. Through the... A grid search was performed on the interval (step size 0.05) and it was found that when Performance is optimal when the value is increased to 0.1; performance decreases beyond this value. Figure 3 The model is in The best results were achieved at the specified time (e.g., 96.60% accuracy on the IRDS dataset and 96.36% F1 score on the REHAB24-6 dataset). Therefore, subsequent experiments were all set to... .

[0271] 4.2 Parameters

[0272] parameter ( This experiment controls the scale partitioning in MSSTMB. To determine the optimal value, parameter search experiments were conducted on all datasets. Figure 4 The grid pattern indicates the optimal combination. For the 30 fps dataset, the optimal setting for IRDS is (10,6,3), for UI-PRMD (Kinect) it is (10,6,3) / (10,6,2) / (10,5,3) / (10,5,2), and for REHAB24-6 it is (10,6,3) / (10,6,2) / (10,5,3). To maintain consistency, these datasets uniformly use the (10,6,3) configuration. For the 100 fps UI-PRMD (Vicon) dataset, the optimal configuration is (20,10,5).

[0273] 5. Experimental Results

[0274] 5.1 Performance Comparison under the AUTT Paradigm

[0275] This experiment compares and evaluates the proposed MSST-GCN with several representative methods under the Absolutely Unified Training and Testing (AUTT) paradigm, including Pnorm, VANet, ST-GCN, ST-GCN with rotation invariance (RI), SCL, and SCL with RI. Based on open-source code, this experiment implements AUTT versions of Pnorm, VANet, ST-GCN, and ST-GCN with RI. For SCL and its RI version, whose code is not publicly available, this experiment first reproduces the results according to the original description, and then implements its AUTT version. Table 1 summarizes the quantitative comparison results under the AUTT paradigm (all metrics are presented as percentages). Figure 5 , Figure 6 , Figure 7 , Figure 8 The model's discriminative ability is then qualitatively evaluated using ROC and PR curves.

[0276] Table 1 Performance comparison under the AUTT paradigm (all metrics are presented as percentages (%))

[0277]

[0278] As shown in Table 1, MSST-GCN achieves state-of-the-art performance on all datasets. On the IRDS dataset, its F1 score reaches 91.54%, slightly outperforming ST-GCN with RI. More significantly, it achieves accuracy / F1 score of 98.80% / 98.79% and 99.19% / 99.19% on the UI-PRMD (Kinect) and UI-PRMD (Vicon) datasets, respectively. On the challenging REHAB24-6 dataset, MSST-GCN maintains strong robustness, outperforming all comparable methods with 96.64% accuracy and 96.36% F1 score. Qualitative analysis shows the ROC and PR curves (…). Figures 5-8 This further validated the discriminative ability and robustness of MSST-GCN—its ROC curve approached the upper left corner, and its PR curve approached the upper right corner, indicating that the model can reliably identify the repetition of correct / incorrect actions under different training action types and environments.

[0279] To further analyze the results, the experimental comparison method has the following limitations: the lack of joint-skeletal multimodal fusion leads to insufficient representation of motion patterns; it fails to capture multi-scale temporal features, focusing only on global dependencies while ignoring short-term temporal correlations; and it does not address the problem of data scarcity or improve robustness to temporal noise, thus limiting its practical application performance.

[0280] The superior performance of MSST-GCN can be attributed to three main contributions: First, the bimodal network integrates joint and skeletal features to enhance discriminative capabilities; second, MSSTMB simultaneously captures global dependencies (reflecting macroscopic processes such as endurance and phase transitions) and short-term temporal correlations (encoding microscopic patterns such as joint stability and movement smoothness); and finally, LGTM enhances the temporal diversity of training data, alleviating data scarcity and improving robustness to temporal noise. These designs enable it to exhibit powerful performance and generalization capabilities in diverse rehabilitation scenarios.

[0281] Table 2 Performance comparison of the ESTT paradigm on the REHAB24-6 dataset (all metrics are presented as percentages (%))

[0282]

[0283] Table 3 Performance comparison of the UTEST paradigm on the REHAB24-6 dataset (all metrics are presented as percentages (%))

[0284]

[0285] 5.2 Performance Comparison between ESTT and UTEST Paradigms

[0286] In addition to the AUTT paradigm, for fair comparison, this experiment further evaluates MSST-GCN and the contrasting methods under the Classified Training and Testing (ESTT) and Unified Training and Classified Testing (UTEST) paradigms. The experiments are conducted on the challenging REHAB24-6 dataset.

[0287] In the ESTT paradigm, each training action type is trained and tested independently, without the need for cross-type generalization ability, thus most methods perform well. As shown in Table 2, the model of this invention achieves 99.30% accuracy and 99.34% F1 score on REHAB24-6, surpassing VANet by 1.80% accuracy and 1.94% F1 score, respectively. However, ESTT requires training an independent model for each class, resulting in the highest training and deployment costs. In the UTEST paradigm, a single model is trained using all types of data but tested separately by class, resulting in moderate training costs. As shown in Table 3, MSST-GCN still maintains the best average accuracy and F1 score in the UTEST paradigm. Combining Tables 1 to 3, it can be seen that AUTT is a challenging paradigm due to the need for unified training and testing across classes. Under all paradigms, MSST-GCN consistently outperforms existing methods, demonstrating strong generalization ability and comprehensive performance advantages.

[0288] 6. Visual Analysis

[0289] Visualization is crucial for rehabilitation training assessment, providing intuitive feedback that helps patients and clinicians understand movement quality. To evaluate the contribution of each joint in the classification of incorrect movements, this experiment calculated the gradient of the model output relative to the input skeletal sequence and normalized the importance of each joint in each frame for visualization. Figure 9 The visualization examples shown are from the REHAB24-6 dataset: the push-up sample at the top is classified as a correct movement by the model, with all joints rendered in blue with a fixed size; the squat sample at the bottom is identified as an incorrect movement, with joint size and color depth reflecting their classification contribution (larger size and darker color indicate a more significant contribution). Meanwhile... Figure 9 The lower half shows that the knee and ankle joints play a dominant role in the assessment, which is highly consistent with the biomechanical characteristics of the squatting movement.

[0290] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. An intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks, characterized in that, include: S1: Obtain the human pose sequence to be evaluated; S2: Input the human posture sequence to be evaluated into the trained rehabilitation assessment model and output the corresponding rehabilitation assessment prediction results; The steps involved in training a rehabilitation assessment model include: S201: Obtain a dataset containing human pose sequences and their labels for various training action types, and divide the dataset into a training set and a test set; S202: Use the human posture sequences of all training movement types in the training set as input for unified training of the rehabilitation assessment model; S203: Extract joint modal feature data and skeletal modal feature data from each frame of the human posture sequence to obtain joint modal feature sequence and skeletal modal feature sequence; S204: By cascading several multi-scale spatiotemporal modeling modules, global dependencies and short-term temporal correlations in the joint modal feature sequence are captured synchronously to obtain joint modal features; In step S204, the multi-scale spatiotemporal modeling module includes three cascaded spatiotemporal feature extraction units; The processing steps for each spatiotemporal feature extraction unit include: S2041: The spatial feature sequence is obtained by performing topological perceptual modeling on the input through the spatial graph convolution module; S2042: After adding the spatial feature sequence to the input of the multi-scale spatiotemporal modeling module element by element, the sequence is then processed through the recombination operation layer to obtain the small-scale recombined feature sequence. S2043: Temporal dynamic features are extracted from the recombined feature sequence at a small time scale using the temporal graph convolution module to obtain the temporal dynamic feature sequence; S2044: The temporal dynamic feature sequence is scaled back by the recovery operation layer to obtain the recovered feature sequence; S2045: The output of the spatiotemporal feature extraction unit is obtained by adding the recovered feature sequence to the input of the multi-scale spatiotemporal modeling module element by element. S205: By cascading several multi-scale spatiotemporal modeling modules, global dependencies and short-term temporal correlations in the skeletal modal feature sequence are captured simultaneously to obtain skeletal modal features; S206: After fusing and average pooling the joint modal features and skeletal modal features, the data is input into two fully connected layers for processing to obtain the first prediction result and the second prediction result. S207: Calculate the loss function based on the first and second prediction results, and optimize the parameters of the rehabilitation assessment model; S208: Repeat steps S202 to S207 to iteratively train the rehabilitation assessment model until it converges or reaches the maximum number of iterations; S209: Use human posture sequences of all training action types in the test set to uniformly test the trained rehabilitation assessment model; during the test, sum the first and second prediction results output by the two fully connected layers to obtain the rehabilitation assessment prediction result output by the rehabilitation assessment model.

2. The intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks as described in claim 1, characterized in that: In step S2, during the training process, a label-guided temporal fusion strategy is used to augment the joint modality feature sequences and the skeletal modality feature sequences. The data augmentation process based on a label-guided time-series fusion strategy includes the following steps: S2031: Obtain the contents A dataset of samples and their corresponding tags Each sample includes Frame data, each frame includes There are 10 key points, each key point is composed of 10 key points. 3D coordinate representation; S2032: Organize the dataset according to the labels The samples in the data are divided into the correct subset of samples. and error sample subset Two subsets; The formula is expressed as: ； ； ； ； In the formula: and Representing datasets respectively Correct and incorrect samples; Indicates the first The label of each sample, This indicates that it is correct. The representation is incorrect; S2033: For the correct sample subset Perform time-series fusion to achieve data augmentation: from the correct sample subset Two samples were randomly selected from the middle. and Temporal downsampling is performed using bilinear interpolation; the downsampled sequences are then concatenated to form an augmented correct sample subset. ; The formula is expressed as: ； In the formula: This indicates a time-series downsampling operation implemented using bilinear interpolation; Represents the time series length after downsampling, and its value is randomly sampled from a beta distribution; symbol This indicates splicing along the time dimension; Indicates from 1 to The index of the random permutation sequence; S2034: For the subset of erroneous samples Performing temporal fusion to achieve data augmentation: from a subset of erroneous samples Two samples were randomly selected from the middle. and Temporal downsampling is performed using bilinear interpolation; the downsampled sequences are then concatenated to form an augmented error sample subset. ; The formula is expressed as: ； S2035: Based on augmented correct sample subsets and augmented error sample subset Building an augmented dataset ; The formula is expressed as: ； 。 3. The intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks as described in claim 1, characterized in that: In step S2041, the spatial graph convolution module includes three branches; the first and second branches both include cascaded 1×1 convolutional layers and temporal pooling layers; the third branch includes a 1×1 convolutional layer. The outputs of the first and second branches are input together into the correlation modeling module for correlation modeling, and then input into the cascaded linear projection layer and topology refinement module. The outputs of the topology graph refinement module and the third branch are fused to obtain the spatial feature sequence output by the spatial graph convolution module.

4. The intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks as described in claim 3, characterized in that: In step S2041, the processing steps of the spatial graph convolution module include: S20411: Sequence of human poses The first in Frame data Input spatial graph convolution module; where Representing data The number of key points, Representing data The number of channels; S20412: Transfer data Input the third branch to obtain intermediate features ;in This indicates a feature extraction operation performed through a 1×1 convolutional layer. S20413: Transfer data After each joint in the model is linearly projected through the first and second branches, it is input into the correlation modeling module to perform channel-by-channel joint correlation modeling, thus obtaining the correlation modeling results. ; The formula is expressed as: ； In the formula: , Representing data The joints in; , Indicates the joint , Perform linear projection to reduce feature dimensionality; This represents the activation function that introduces a nonlinear projection; S20414: Correlation Modeling Results Using a Linear Projection Layer Perform a linear projection to obtain the correlation tensor. Correlation tensor vectors in Indicates joint With joints The final channel-by-channel correction relationship between them; The formula is expressed as: ； In the formula: Indicates a linear projection layer; S20415: Relevance tensor refined using the topology graph refinement module The data is then refined to obtain a channel-by-channel topology map. ; The formula is expressed as: ； In the formula: This represents the learnable scaling factor, used to adjust the thinning intensity; This indicates element-wise addition implemented via a broadcast mechanism; An adjacency matrix representing the topological structure of a sequence of human poses; S20416: The intermediate feature output from the third branch Channel-by-channel topology map output by the topology refinement module Perform channel-by-channel feature fusion to generate output data. Spatial features ; where the channel-by-channel topology diagram Each channel in the matrix corresponds to a refined topology matrix. , Using topological matrix Independent fusion intermediate features The spatial features are obtained by stitching together the feature data of each channel and all the fusion results. ; The formula is expressed as: ； In the formula: , express The Middle Feature data of each channel; symbols Indicates a splicing operation; S20417: Human posture sequence Each frame of data is processed through steps S20411 to S20416 to obtain the spatial features of each frame of data; the sequence composed of all spatial features is used as the spatial feature sequence output by the spatial graph convolution module.

5. The intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks as described in claim 1, characterized in that: In step S2042, the recombination operation layer is used to divide the input feature sequence into finer scales along the time dimension through recombination operations; The formula is expressed as: ； In the formula: Indicates a restructuring operation; Indicates the first A time scale; The number of frames representing the feature sequence; This indicates the number of key points in each frame of data; This indicates the number of channels in each frame of data.

6. The intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks as described in claim 1, characterized in that: In step S2043, the time-map convolution module includes four branches; The first, second, and third branches each consist of cascaded 1×1 convolutional layers and ReLU activation functions; the fourth branch consists of a 1×1 convolutional layer. The output of the first branch is connected to a max pooling layer; the outputs of the second and third branches are each connected to a dynamic temporal graph convolutional layer. After element-wise summing of the outputs of the max pooling layer, the two dynamic temporal graph convolutional layers, and the fourth branch, a 1×1 convolutional layer is used for feature fusion to obtain the temporal dynamic feature sequence output by the temporal graph convolutional module.

7. The intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks as described in claim 6, characterized in that: In step S2043, the processing steps of the dynamic temporal graph convolutional layer in the temporal graph convolution module include: S20431: The feature sequence As input to the dynamic temporal graph convolutional layer; where Indicates time Corresponding features Indicates the number of frames; S20432: Generating timing based on sampling strategy of dynamic temporal graph convolutional layer sampling location set ; The sampling strategy of dynamic temporal graph convolutional layers is expressed as follows: ； In the formula: Indicates the first The first frame centered on the convolution operation Each frame position; Indicates the feeling field; Indicates feeling wild The first One trainable offset parameter; S20433: Time-based one-dimensional convolution sampling location set For feature sequences At any moment Perform feature aggregation along the time dimension to obtain time-aggregated features. ; The formula is expressed as: ； In the formula: Indicates the kernel size; Indicates the first convolution kernel One weight parameter; This represents a sampling function implemented through linear interpolation; Indicates time Corresponding features; S20434: For characteristic sequences At each time step, S20432 to S20433 are executed to obtain the corresponding temporal aggregated features; the sequence composed of the temporal aggregated features of all time steps is used as the dynamic temporal graph convolutional sequence output by the dynamic temporal graph convolutional layer. .

8. The intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks as described in claim 1, characterized in that: In step S2044, the recovery operation layer is used to restore the input feature sequence to its original scale through the recovery operation; The formula is expressed as: ； In the formula: Indicates a recovery operation; Indicates the first A time scale; The number of frames representing the feature sequence; This indicates the number of key points in each frame of data; This indicates the number of channels in each frame of data.

9. The intelligent rehabilitation training assessment method based on multi-scale spatiotemporal graph convolutional networks as described in claim 1, characterized in that: In step S207, the loss function is calculated using the following formula. : ； ； ； ； In the formula: , These represent the loss functions corresponding to the two fully connected layers; , They respectively represent the inclusion Sample small batch dataset The first and second prediction results output by two fully connected layers; Indicates the number of categories; Indicates to The probability vector obtained after softmax normalization is the first... One element; This indicates the label after label smoothing. One component; Indicates sample Unique hot-coded real labels; Indicates an indicator function, when The value is 1 if the condition is met, and 0 otherwise. This is the smoothing coefficient.

Images