Rehabilitation behavior analysis method, device and equipment based on dynamic visual cues generation

Abstract

The application discloses a rehabilitation behavior analysis method, device and equipment based on dynamic visual prompt generation, comprising: collecting action video data, extracting two-dimensional skeleton point sequence data from the action video data through a pose estimator; extracting visual features and prompt features from video modal data and skeleton point modal data through a visual classification model and a skeleton classification model respectively; pre-training the visual classification model using labeled video modal data, and then performing contrast learning on the visual classification model and the skeleton classification model using unlabeled video modal data and corresponding unlabeled skeleton modal data respectively; collecting rehabilitation training video data of a patient and extracting corresponding skeleton modal data; and inputting the video data and the corresponding skeleton modal data into the trained visual classification model and the skeleton classification model respectively for classification. The application can realize more complete action representation, reduce data dependence, and improve the accuracy of rehabilitation behavior analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of auxiliary medical technology, and in particular to a rehabilitation behavior analysis method, device and equipment based on dynamic visual prompt generation. BACKGROUND

[0002] Postoperative rehabilitation training is a key link in orthopedic treatment. Through systematic rehabilitation exercises such as joint movement exercises and low-intensity strength training, bone healing is promoted, joint stiffness is prevented, and the function of the injured part and the ability of daily activities of the patient are restored. In the field of medical devices, the application of motion capture devices, wearable sensors and intelligent rehabilitation systems provides technical support for precise rehabilitation. However, rehabilitation training requires patients to strictly follow standard movements, and incorrect movements can lead to poor healing, limited joint function, and even secondary injury.

[0003] Currently, most hospitals rely on orthopedic doctors or rehabilitation therapists to guide patients in rehabilitation training on site and supervise the standardization and frequency of movements. However, this traditional method has significant limitations: on-site supervision requires a large amount of time and resources from medical staff, especially in areas with a large number of patients or limited medical resources, increasing the operational burden of hospitals; since medical staff cannot monitor the training process of each patient in real time all day long, some patients may not be able to correctly complete rehabilitation movements due to the lack of continuous guidance, affecting the recovery effect; the evaluation of medical staff is usually based on subjective observation, lacking precise quantitative standards to judge the standardization and consistency of movements; traditional methods lack systematic data collection, making it difficult to comprehensively record and evaluate the rehabilitation progress of patients, limiting the optimization and individualized adjustment of treatment plans.

[0004] In recent years, multi-modal rehabilitation training supervision systems based on prompts have made some progress combining artificial intelligence and medical device technology. The main prompts include constructing single-frame text descriptions for videos. Text description is a static description of video, and the content expressed is very limited. For the changes before and after the action between frames, text is difficult to accurately express quantitative motion attributes such as joint changes and acceleration. High-quality frame-level text description requires a large amount of manual annotation, which is costly and subjective; text description may only reflect explicit content such as objects, scenes, and other external visible information, making it difficult to capture internal implicit information such as motion trajectories and acceleration, and the content between consecutive frames is highly similar, resulting in low information density of generated text description, wasting computing resources; text description analyzes each frame as an independent unit, which ignores the inherent temporal correlation of video data, making it impossible to accurately model the continuous evolution of the action, severely weakening the coherence and integrity of the description result.

[0005] Some technologies use image-text pre-training large models for encoding processing of text information, however, due to the difference between the task target of the large model itself and the rehabilitation behavior recognition task, the incompatibility thereof cannot accurately obtain the detailed information features of the video action, and thus it is difficult to be truly applied in actual tasks.

[0006] At present, there is no method capable of obtaining detailed information features of rehabilitation actions from video data and sufficiently integrating these features. SUMMARY

[0007] The present application provides a rehabilitation behavior analysis method, system and device based on dynamic visual prompt generation, which can realize more complete action representation, reduce data dependence and improve the accuracy of rehabilitation behavior analysis.

[0008] The technical solution of the present application is as follows:

[0009] A rehabilitation behavior analysis method based on dynamic visual prompt generation comprises the following steps:

[0010] (1) Action video data is collected as a video modal dataset; two-dimensional skeleton point sequence data is extracted from the action video data by a pose estimator as a skeleton modal data;

[0011] (2) Visual features are extracted from the video modal data by a visual classification model; prompt features are extracted from the skeleton point modal data by a skeleton classification model;

[0012] (3) The visual classification module is pre-trained using labeled video modal data, and the visual classification model and the skeleton classification model are simultaneously trained using unlabeled video modal data and corresponding unlabeled skeleton modal data, respectively, and comparative learning is performed at the positions of feature output and classification output;

[0013] (4) Rehabilitation training video data of a patient is collected, and corresponding skeleton modal data is extracted by a pose estimator; the video data and the corresponding skeleton modal data are input into the trained visual classification model and the skeleton classification model, respectively, for classification.

[0014] The present application adopts a semi-supervised dual-channel structure framework, analyzes the feature differences between the visual channel and the prompt channel through comparative learning, and enhances the action feature representation through mutual complementation. This architecture significantly improves the learning efficiency of the model under a small amount of labeled data, realizes more complete action representation, reduces data dependence, and improves the accuracy of rehabilitation behavior analysis.

[0015] In step (1), a camera is used to collect action video data, which is processed and video-level labeled by a professional rehabilitation behavior analyst.

[0016] The action video data uses a two-dimensional pose estimator to extract its corresponding 25 skeleton point coordinates, which are stored in the format of N, T, C, H, W, as skeleton modal data, where N, T, C, H, and W represent the number of samples, the number of channels, the time step, the video height, and the width, respectively.

[0017] Preferably, in step (2), the visual features are extracted from the video modal data by a visual classification model, including: video segmentation of the video modal data; using a 3D ResNet-50 to extract a feature vector containing spatial and temporal characteristics of the segmented video block; using an encoder to encode the feature vector, and converting it into a visual feature embedding through a projection layer, and finally generating a visual feature representation.

[0018] The encoder is a Transformer encoder, including 6 layers of hidden layers, each layer having 8 attention heads, and the hidden layer dimension being 512.

[0019] The projection layer embedding process uses ReLU as an activation function to introduce nonlinearity, and then applies layer normalization LayerNorm to stabilize and normalize the feature distribution.

[0020] Preferably, in step (2), the prompt features are extracted from the skeleton point modal data by a skeleton classification model, including:

[0021] (i) dividing six part regions by human topological structure, calculating the spatial relationship between joints and the mass center of each region, and obtaining single-frame features;

[0022] (ii) extracting low-speed indicators and high-speed indicators on the angle change time series by calculating the joint angle change, and calculating the average distance between the mass centers of different parts to obtain inter-frame features;

[0023] (iii) fusing the single-frame features and the inter-frame features to generate video-level prompt features.

[0024] In the skeleton point modal processing process, frame-level action features are extracted from the skeleton point sequence, including the coordination, aggregation of actions in a single frame, and the morphological change, rhythm, and part spatial divergence between frames. This analysis mechanism not only significantly improves the objectivity and information density of feature extraction, but also more completely retains the implicit kinematic features of actions, while establishing precise timing association information, thereby overcoming the inherent limitations of existing text labeling methods.

[0025] Step (i) includes:

[0026] defining a set of skeleton points wherein J represents the index of a skeletal joint; represents the index of a skeletal joint;i The two-dimensional coordinates (x, y) of a skeletal joint at time t;

[0027] Define a set of human body parts , where N represents the index of the human body part; represents the set of coordinates of the joints within the k th part at time t;

[0028] The set of centroids of the parts is calculated as follows:

[0029] ;

[0030] where, represents the number of joints contained within part , and the centroid represents the spatial center position of the k th part at time t;

[0031] Calculate the average proximity between all joints and the centroids of all parts , and the formula is as follows:

[0032] ;

[0033] where X and Y represent the set of joints A and the set of centroids Z, respectively, where ∈ X, ∈ Y; d( , ) is the and Euclidean distance between; I(⋅) is the indicator function, which is 1 when d( , ) < r and 0 otherwise; and​​​​​​​​​​​​​​​​​​​​​​​​​​​​​

[0036] ;

[0037] ;

[0038] where, is the two-dimensional vector from the centroid to the joint , is the two-dimensional vector from the centroid to the joint ; the output angle is in degrees and ranges from [0, 180];

[0039] Based on the inter-joint angle calculation, the inter-joint angle time series matrix of M centroid reference points in each frame in the time window W is obtained :

[0040] ;

[0041] where, represents the included angle formed by the Mth part centroid and the other two joint nodes in the Wth frame;

[0042] The is flattened by row into a one-dimensional feature vector with length W x M .

[0043] Subsequently, the Haar wavelet transform is applied to calculate the low-speed index and the high-speed index , the formula is as follows:

[0044] ;

[0045] ;

[0046] The average distance between the centroids of different parts The calculation formula is as follows:

[0047] ;

[0048] where, and represent the indices of the two centroids, i ≠ j ;( , ) and ( , ) are the coordinates of the two centroids at time t.

[0049] Single-frame analysis captures the changes of motion in a single frame by calculating the spatial relationship of joint and part centroid, including joint-to-joint, joint-to-part, part-to-joint, and part-to-part. Inter-frame analysis quantifies the dynamic changes between frames by joint angle, Haar wavelet transform to generate low-speed indicators and high-speed indicators, and indicator distance. The inherent implicit information is obtained directly from the video data and the temporal relationship is established, which overcomes the shortcomings of the prior art, such as high artificial labeling cost and subjectivity, inability to effectively convey the inherent implicit information of motion, and ignoring the inherent temporal correlation of video data.

[0050] Step (iii) comprises:

[0051] The single-frame features and the inter-frame features are spliced along the feature dimension to generate high-dimensional joint features .

[0052] Temporal uniform sampling is performed on the sequence by a sliding window to obtain a sequence .

[0053] .

[0054] wherein, w is the size of the sliding window; n is the number of sampled frames; t is the starting time of sampling; t is the sampling time of the i-th frame. T K A projection function is used to reduce noise, a Transformer encoder is used to model attribute interaction, and a Transformer decoder is used to generate prompt features : k

[0055] wherein, the parameters , , of the projection function, the encoder, and the decoder, respectively, are parameters of the projection function, the encoder, and the decoder; and

[0056] represents a learnable prompt vector at time t.

[0057] t

[0058] ​​​​​​​​The application adopts prompt generation based on the Transformer, integrates single-frame and inter-frame features into unified video-level prompt features through an encoder-decoder structure and a dynamic weighting mechanism, combines vectorization, projection and multi-head self-attention mechanism, realizes coordinated expression of action features at the frame level, significantly improves data adaptability and semantic representation ability, requires smaller training data scale, and can effectively improve the recognition accuracy of complex rehabilitation actions.

[0059] In step (3), the visual classification model and the skeleton classification model are simultaneously trained using unlabeled video modal data and corresponding unlabeled skeleton modal data, and contrastive learning is performed at the positions of feature output and classification output, including:

[0060] The visual features and the prompt features are vectorized for consistency comparison;

[0061] The probability distributions output by the visual classification model and the skeleton classification model are compared for consistency:

[0062] ;

[0063] Wherein, is the predicted probability distribution of the skeleton classification model for the unlabeled sample ; is the predicted probability distribution of the visual classification model for the same sample ; is the KL divergence, which measures the difference between two probability distributions, and C is the number of action categories; is the number of unlabeled samples.

[0064] The application adopts a semi-supervised dual-channel structure framework, analyzes the feature differences between the visual channel and the prompt channel through contrastive learning, and enhances the action feature expression through mutual complementation, which significantly improves the learning efficiency of the model under a small amount of labeled data, realizes more complete action representation, reduces data dependence, and improves the accuracy of rehabilitation behavior analysis.

[0065] In step (4), the two classification models are fully aligned after consistency constraint learning in the inference stage, i.e. both classification models can be used to classify rehabilitation behaviors.

[0066] A rehabilitation behavior analysis device based on dynamic visual prompt generation, comprising the following units:

[0067] A data set construction unit acquires action video data as a video modal data set; a pose estimator extracts two-dimensional skeleton point sequence data from the action video data as skeleton modal data;

[0068] The feature extraction unit extracts visual features from the video modal data through a visual classification model and extracts prompt features from the skeleton point modal data through a skeleton classification model;

[0069] The training unit: pre-trains the visual classification module using labeled video modal data, and simultaneously trains the visual classification model and the skeleton classification model using unlabeled video modal data and corresponding unlabeled skeleton modal data, respectively, and performs contrastive learning at the positions of the feature output and the classification output;

[0070] The application unit collects rehabilitation training video data of a patient, extracts corresponding skeleton modal data through a pose estimator, and inputs the video data and the corresponding skeleton modal data into the trained visual classification model and the skeleton classification model, respectively, for classification.

[0071] The application also provides a rehabilitation behavior analysis device based on dynamic visual prompt generation, which comprises a video acquisition device, a memory, a processor, and a computer program stored in the memory and executable on the processor, and the memory also stores a visual classification model and a skeleton classification model constructed and trained by the rehabilitation behavior analysis method based on dynamic visual prompt generation;

[0072] The processor implements the following steps when executing the computer program:

[0073] The video acquisition device collects rehabilitation training video data of a patient, and a pose estimator extracts corresponding skeleton modal data;

[0074] The video data and the corresponding skeleton modal data are input into the trained visual classification model and the skeleton classification model, respectively, for classification and output of classification results.

[0075] Compared with the prior art, the application has the following advantages:

[0076] In the prior art, the text prompt has the disadvantages of high manual annotation cost, subjectivity, inability to effectively convey the inherent implicit information of actions, and neglect of the inherent temporal correlation of video data, etc. The application directly obtains the inherent implicit information from the video data and establishes the temporal relationship by single-frame analysis and inter-frame analysis, overcoming the shortcomings of the prior art.

[0077] In the prior art, due to the incompatibility of the image-text pre-training large model with video data, it cannot accurately extract the action detail information in the text and establish a good long sequence video understanding. In contrast, the prompt generation proposed in the application significantly improves the data adaptability and semantic representation ability, requires a smaller training data scale, and also ensures high accuracy.

[0078] The application adopts a semi-supervised double-channel structure framework, analyzes the feature differences of the visual channel and the prompt channel through contrast learning, and enhances the motion feature expression through mutual complementation, thereby significantly improving the learning efficiency of the model under a small amount of labeled data, realizing more complete motion representation, reducing data dependence, and improving the accuracy of rehabilitation behavior analysis. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 Fig. 1 is a structural schematic diagram of a multi-modal rehabilitation motion recognition system of the application;

[0080] Figure 2 Fig. 4 is a human topological structure segmentation diagram. DETAILED DESCRIPTION

[0081] The application will be further described in detail below in combination with the drawings and examples, and it should be pointed out that the following examples are intended to facilitate the understanding of the application and do not have any limiting effect on the application.

[0082] A multi-modal rehabilitation motion recognition system comprises the following modules: a data collection and processing module, a visual feature channel, a prompt feature channel, a single-frame analysis module, an inter-frame analysis module, and a prompt generation module, and the specific module connection is as shown in Figure 1 .

[0083] The data collection and processing module: this module uses a monocular camera to collect motion video data, and pre-processes through an existing general pose estimator to form video modal data and skeleton point modal data respectively, which are used for the visual feature channel and the prompt feature channel.

[0084] The monocular camera is an RGB camera, the capture rate of the RGB camera is 30 frames / second, and the field of view range is sufficient to cover the main area of human joint activity. The processing and video-level labeling are performed by professional behavior analysts. A small amount of labeled data is used for model learning, and a large amount of unlabeled data is used for contrast learning. All video data uses a two-dimensional pose estimator HRNET to extract 25 skeleton point coordinates corresponding thereto. In order to ensure the accuracy of the data, the resolution threshold of the pose estimator is 0.6, that is, only the skeleton point data with a confidence higher than 0.6 is saved. The data is stored in a binary file with a.pt format in the format of N, T, C, H, and W, wherein N, T, C, H, and W represent the number of samples, the number of channels, the time step, the video height, and the width, respectively.

[0085] The visual feature channel: this channel is used to extract visual features from the video. The features of this channel are mainly concentrated on the task of classification, and the video motion change features are extracted through a deep learning model, which is the same as the prior art.

[0086] The visual pathway processing first proceeds with video partitioning, which splits the input video sequence into 16-pixel x 16-pixel spatio-temporal blocks with a time step of 8 frames to comprehensively capture dynamic motion patterns. To ensure the continuity and overlap of spatial information, adjacent blocks are configured with an overlap rate of 50%, thereby enhancing the capture of inter-frame local spatial dependencies. The partitioned blocks are processed in the feature extraction stage using a 3D ResNet-50, which is a convolutional neural network architecture optimized for spatio-temporal data processing. The 3D ResNet-50 input is adjusted to an input block with a resolution of 224 pixels x 224 pixels and three color channels, generating a 2048-dimensional feature vector that contains the spatial and temporal characteristics of the video block. The extracted features are subsequently encoded using a Transformer model, which is a neural network architecture that utilizes self-attention mechanisms to model complex dependencies in the feature set. The Transformer encoder includes 6 layers, each with 8 attention heads and a hidden layer dimension of 512, facilitating effective encoding of sequence and contextual feature interactions. The encoded features are converted into compact visual feature embeddings through a projection layer, ultimately generating a 256-dimensional representation. The embedding process employs ReLU as an activation function to introduce non-linearity, followed by the application of LayerNorm to stabilize and normalize the feature distribution, ensuring the robustness and consistency of the final feature vector.

[0087] Prompt feature pathway: This pathway is used to extract frame-level prompt features from the skeleton sequence, which internally includes a single-frame analysis module, an inter-frame analysis module, and a prompt generation module. The skeleton point sequence is analyzed by the single-frame analysis module and the inter-frame analysis module to obtain frame-level features of motion, which are then integrated by the prompt generation module to generate video-level prompt features.

[0088] The input of this channel is a sequence of 2D skeleton points, where each frame contains the 2D coordinates of 25 joints on the image plane. First, two types of key features are calculated by the single-frame and inter-frame analysis module: one is the single-frame feature, which divides the body into six regions according to the human body topology, calculates the spatial relationship between the joints and the centroid of each region, and measures the coordination and aggregation of the skeleton at different time points; the other is the inter-frame feature, which extracts the change of action form by calculating the angle change of the joints relative to the centroid of the region, and applies Haar wavelet transform on the angle time sequence to extract low-speed and high-speed indicators to depict the rhythm change of the action, and calculates the average distance between the centroids of different parts as an indicator distance to reflect the spatial divergence of the trunk. These two types of key features are spliced into a unified vector sequence, which is input into the prompt generation module to generate the final video-level prompt feature. This module first uses a linear mapping layer with an input dimension of attribute vector dimension and an output dimension of 512 to embed the features, and adds a sinusoidal position encoding to preserve the timing information, then projects and denoises the features through two fully connected networks with ReLU activation function. The encoder part is built with six standard Transformer structures, each consisting of a multi-head self-attention mechanism and a set of feedforward neural networks, where each attention layer contains eight attention heads, and each feedforward network has a hidden dimension of 2048, with residual connections and layer normalization operations after each sub-layer to enhance training stability. The decoder part also consists of six layers, which have the same structure as the encoder, and introduces a cross-attention mechanism to interact and fuse the learned prompt vector with the encoder output to generate the action guidance feature. The final output prompt sequence has a dimension of time frame number multiplied by 512, which is the feature expression of the prompt feature channel, and is used together with the visual channel output to participate in the consistency regularization and contrastive learning process.

[0089] Single-frame analysis module: This module is used to quantify the coordination and aggregation of the skeleton sequence in the action at this moment, and analyzes the static spatial relationship. Its attribute value is directly related to the aggregation and coordination of the body, so it can directly represent the action change in the movement process, which includes the analysis of four types of relationships: joint-joint, joint-part, part-joint, and part-part.

[0090] Definition of the set of skeleton points wherein J represents a set of 25 skeleton joints, each represents the 2D coordinates (x, y) of the i th skeleton joint at time t. At the same time, define the set of body parts wherein N represents 6 body parts (such as head, trunk, left arm, right arm, left leg, right leg), which are divided according to the human body topology as shown in Figure 2 . Each represents the coordinate set of the joint in the i-th part at time t. To make the divided parts suitable for spatial analysis formula, the centroid of each part needs to be calculated. k

[0091] Centroid of a part The calculation formula is as follows:

[0092] ;

[0093] wherein, represents the number of joints contained in the part , and the centroid represents the spatial center position of the i-th part at time t, which is used for subsequent spatial relationship analysis. k By analyzing the spatial relationship between the joints and the centroid of the part, the spatial distribution characteristics of the action are extracted, and a unified analysis function is used for calculation to measure the number of point pairs within the distance threshold r, and the formula is defined as follows:

[0094]

[0095] ;

[0096] In the spatial distribution analysis of rehabilitation actions, X and Y represent the joint set A (joints) and the centroid set Z (part centroids) respectively, wherein ∈X, ∈Y; d( , ) is the Euclidean distance between and ; I(⋅) is an indicator function, which is 1 when d( , )<r, otherwise 0; , are the sizes of set A or Z (A=25, Z=6) respectively; the normalization factor , wherein V is the volume of the spatial domain, when X=Y (excluding self-pairing) =1, otherwise =0; represents the average proximity between all joints and all part centroids.

[0097] Inter-frame analysis module: This module is used to obtain the morphological changes, rhythm changes and part spatial divergence characteristics between frames of the bone sequence, and analyzes the dynamic time sequence evolution, including angle, low-speed and high-speed indicators, and indication distance calculation. Based on the angle of each frame, the calculated low-speed and high-speed indicators can directly reflect the local changes, and the indication distance adds the credibility of the low-speed and high-speed indicators from the distance.

[0098] ​​Its joint angle captures the shape change of human action in two-dimensional plane, this method quantifies the geometric relationship of action by calculating the angle of skeletal joint relative to the centroid of the part. Given the coordinates of the centroid ,the coordinates of two joints and are and , respectively, the angle between two joints is calculated by two-dimensional vector cosine rule, as follows:

[0099] ;

[0100] ;

[0101] where is the two-dimensional vector from the centroid r to joint , and is the two-dimensional vector from the centroid r to joint . The output angle is in degree, ranging from [0, 180], representing the planar geometric relationship between two joints.

[0102] Its low-speed and high-speed index enhances the time feature representation of action, this method uses Haar wavelet transform to decompose the joint angle time series, respectively capturing the smooth trend and rapid dynamic change of action, so as to improve the distinguishing ability of feature channel. Based on the calculation of joint angle, M centroid angles of each frame are obtained. For the time window size W = 16, the data is divided into overlapping time windows, generating the angle time series matrix , defined as:

[0103] ;

[0104] where represents the included angle formed by the Mth part centroid and the other two joint nodes in the Wth frame.

[0105] Flatten by row into a one-dimensional feature vector with length W x M.

[0106] Subsequently, Haar wavelet transform is applied to calculate the low-speed index and high-speed index , as follows:

[0107]

[0108]

[0109] where, Capture low-frequency trends, reflecting stable patterns of action; Capture high-frequency details, highlighting rapid changes.

[0110] It indicates the distance quantifies the distance change between body parts in action. This method proposes a novel distance metric to calculate the average distance of each centroid to other centroids within a single frame to reflect the body's part divergence. The distance indicator is defined as follows:

[0111] ;

[0112] where, i represents the index of a particular centroid, and j≠i represents other keypoints except this centroid, , ) and ( , ) are the coordinates of two centroids at time t.

[0113] Prompt generation module: The core function of this module is to deeply integrate the feature information output by the single-frame analysis module and the inter-frame analysis module to generate video-level prompt features that can accurately express frame-level action details. Its design fully utilizes the advantages of the Transformer architecture in data adaptability and specific semantic representation generation.

[0114]

[0115] The single-frame feature vector output by the single-frame analysis module is concatenated with the inter-frame feature vector output by the inter-frame analysis module along the feature dimension to generate a high-dimensional joint feature , where and are both 4, representing the number of features output by the module.

[0116] Temporal uniform sampling is performed on the sliding window :

[0117] ;

[0118] where, T is the size of the sliding window; K is the number of sampled frames; is the starting time of sampling, is the k th sampling time;

[0119] The above formula indicates that from consecutive video frames , a fixed interval is selected frames, constituting a sequence . Assuming the video frame rate is 30 fps (30 frames per second), set the sliding window seconds, sample frame number . Then: seconds; from the starting frame , take a frame every 0.033 seconds, and finally get a sequence of 15 frames .

[0120] Use the projection function to reduce noise, use a two-layer fully connected network (FCN) to implement, which filters high-frequency noise through nonlinear transformation ReLU, and layer normalization LayerNorm stabilizes the feature distribution, where is 1024, is 512, representing the number of neurons.

[0121] ;

[0122] where, V is the joint feature vector obtained by splicing single-frame features and inter-frame features; are the weight matrices of the two-layer fully connected network, respectively, are the corresponding bias terms; represent the dimensions of single-frame features and inter-frame features, respectively, is the hidden layer dimension, is the feature embedding dimension.

[0123] Transformer encoder modeling attribute interaction, Transformer decoder generates prompt sequence. The final output prompt representation is , where the parameters , , are optimized to highlight action-specific patterns.

[0124] The training process of a multi-modal rehabilitation action recognition system is divided into two steps: the first step uses labeled data to initialize the visual channel, so that it has a certain degree of learning ability. The second step uses unlabeled data to train the visual and prompt feature channels simultaneously, and performs location contrast learning on the feature output and classification output. Through such a dual-channel structure for training, the feature information difference between the two channels can be made up, so that it can learn the most effective features, thereby improving the learning ability of the model.

[0125] Step 1: Send the visual features of the visual feature channel into the classifier to get the classification probability distribution. For C action categories, the true label of the sample is , the model-predicted probability distribution is where represents the probability of predicting class c, and the cross-entropy loss function is defined as where is the one-hot encoding of the true label, and if the sample belongs to class c, then = 1, otherwise = 0.

[0126] Second step: The visual feature channel and the prompt feature channel output are vectorized to perform consistency comparison, aiming to ensure the alignment of the features of the two channels, reduce prediction bias, and improve the stability of action classification. At the same time, the probability distributions output by the classifiers of the two channels are compared for consistency, aiming to measure the difference in prediction distribution of the model under different channels on unlabeled data, enhance the generalization ability and robustness of the model, and improve the adaptability of the model to data-scarce scenarios using unlabeled data. Specifically: where is the prediction probability distribution of the prompt channel for the unlabeled sample , and is the prediction probability distribution of the dynamic prompt channel for the same sample, is the KL divergence, which measures the difference between the two probability distributions, is the number of unlabeled samples, and C is the number of action categories.

[0127] In practical applications, the specific process is as follows:

[0128] (1) Data acquisition and preprocessing:

[0129] Use a monocular RGB camera to collect real-time rehabilitation training videos of patients; extract the corresponding two-dimensional skeleton points through a pose estimator;

[0130] (2) Double-channel inference and fusion determination:

[0131] The video frame sequence and the skeleton sequence are input into the visual channel and the prompt feature channel for processing.

[0132] The visual feature channel extracts the spatial-temporal features of the video; the prompt feature channel calculates the single-frame and inter-frame motion prompt features based on the skeleton point sequence; the features of the two channels have been fully aligned after consistency constraint learning in the inference stage, i.e., both channels can be used for classification.

[0133] The visual channel can be used alone for real-time classification, or the features of the visual channel and the prompt channel can be used jointly for prediction to improve accuracy.

[0134] For a more complete analysis, the values calculated by the cue channels can be visualized, as these are the values that have a true physical meaning and can be used to represent the state of motion of the human body. This can assist the physician in performing remote assessments.

[0135] The above-described embodiments detail the technical solutions and beneficial effects of the present application. It should be understood that the above-described embodiments are only specific embodiments of the present application and are not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.

Claims

1. A method for analyzing rehabilitation behaviors based on dynamic visual cues, characterized in that, Includes the following steps: (1) Collect motion video data as video modal dataset; extract two-dimensional skeleton point sequence data from motion video data using a pose estimator as skeletal modal data; (2) Visual features are extracted from video modal data through a visual classification model, including: video segmentation of video modal data; using 3D ResNet-50 to extract feature vectors containing the spatial and temporal characteristics of the video blocks from the segmented video blocks; encoding the feature vectors with an encoder, and then converting them into visual feature embeddings through a projection layer to finally generate visual feature representations; Cue features are extracted from skeletal point modal data using a skeletal classification model, including: (i) Divide the human body into six regions based on its topological structure, calculate the spatial relationship between the joints and the centroids of each region, and obtain single-frame features; (ii) By calculating the joint angle change, low-speed and high-speed indicators are extracted from the angle change time series, and the average distance between the centroids of different parts is calculated to obtain inter-frame features. (iii) Fuse single-frame features with inter-frame features to generate video-level cue features; (3) The visual classification module is pre-trained using labeled video modal data, and then the visual classification model and the skeletal classification model are trained simultaneously using unlabeled video modal data and corresponding unlabeled skeletal modal data, respectively. The learning is compared between the feature output and the classification output, including: vectorizing the visual features and the cue features for consistency comparison; and comparing the probability distributions output by the visual classification model and the skeletal classification model for consistency. (4) Collect rehabilitation training video data of patients and extract corresponding skeletal modal data through a pose estimator; input the video data and corresponding skeletal modal data into the trained visual classification model and skeletal classification model respectively for classification.

2. The rehabilitation behavior analysis method based on dynamic visual cues as described in claim 1, characterized in that, The encoder is a Transformer encoder, which includes 6 hidden layers, each with 8 attention heads, and the hidden layer dimension is 512.

3. The rehabilitation behavior analysis method based on dynamic visual cues as described in claim 1, characterized in that, Step (i) includes: Define the set of skeletal points ,in J Index representing skeletal joints; Indicates the first i Two-dimensional coordinates (x, y) of a skeletal joint at time t; Define a set of human body parts ,in N Indexes representing body parts; Indicates the first k The set of coordinates of joints within a given location at time t; set of centroids of parts The calculation formula is as follows: ； in, Indicates location Number of joints contained within, center of mass Indicates the first k The spatial center position of each part at time t; Calculate the average proximity between the centers of mass of all joints and all body parts. The formula is as follows: ； where \(X\) and \(Y\) represent the joint set \(A\) and the centroid set \(Z\) respectively, where \(\in X\), \(\in Y\); \(d( , ) is and the Euclidean distance between; \(I(\cdot)\) is the indicator function, which is 1 when \(d( , ) < r and 0 otherwise; and are the sizes of the joint set \(A\) and the centroid set \(Z\) respectively; the normalization factor , where \(V\) is the volume of the spatial domain, and = 1 when \(X = Y\), otherwise = 0.

4. The rehabilitation behavior analysis method based on dynamic visual cues as described in claim 1, characterized in that, Step (ii) includes: Given the centroid The coordinates are Two joints and The coordinates are respectively and Inter-joint angle The calculation is performed using the two-dimensional vector cosine rule, and the formula is as follows: ； ； in, From the center of mass To the joint a two-dimensional vector, From the center of mass To the joint A two-dimensional vector; output angle The unit is degrees, and the range is [0, 180]. Based on the calculation of inter-joint angles, the time series matrix of inter-joint angles of M centroid reference points in each frame within the time window W is obtained. : ； in, This represents the angle formed between the centroid of the Mth part and the other two joint points in the Wth frame, with the centroid of the Mth part as the reference point. Will Flattened by rows, it becomes a one-dimensional feature vector of length W×M. ; Subsequently, Haar wavelet transform was applied to calculate the low-speed index. and high-speed indicators The formula is as follows: ； ； Average distance between the centroids of different parts The calculation formula is as follows: ； in, and Indices representing the two centroids. i ≠ j ;( , )and( , Let be the coordinates of the two centroids at time t.

5. The rehabilitation behavior analysis method based on dynamic visual cues as described in claim 1, characterized in that, Step (iii) includes: Single-frame features and inter-frame features Concatenate along the feature dimensions to generate high-dimensional joint features ; The sequence is obtained by time-series uniform sampling through a sliding window. : ； in, T To adjust the sliding window size; K This represents the number of sampling frames. The sampling start time, For the first k Sampling time; Using projection function To reduce noise, a Transformer encoder is used. Modeling attribute interaction, using Transformer decoder Generate prompt features : ； Among them, parameters , , These are the parameters for the projection function, encoder, and decoder, respectively. Indicates in t Learnable cue vectors at each time step.

6. The rehabilitation behavior analysis method based on dynamic visual cues as described in claim 1, characterized in that, In step (3), the probability distributions output by the visual classification model and the skeleton classification model are compared for consistency: ； in, For skeletal classification models of unlabeled samples The predicted probability distribution; For visual classification models of the same sample The predicted probability distribution; KL divergence measures the difference between two probability distributions, where C is the number of action categories; The number of unlabeled samples.

7. A rehabilitation behavior analysis device based on dynamic visual cues, characterized in that, A rehabilitation behavior analysis method based on dynamic visual cues as described in any one of claims 1-6, comprising the following units: The dataset construction unit collects motion video data as a video modal dataset; a pose estimator extracts two-dimensional skeleton point sequence data from the motion video data as skeletal modal data. The feature extraction unit extracts visual features from video modal data using a visual classification model; Extract cue features from skeletal point modal data using a skeletal classification model; Training Unit: The visual classification module is pre-trained using labeled video modal data, and then the visual classification model and the skeletal classification model are trained simultaneously using unlabeled video modal data and corresponding unlabeled skeletal modal data, respectively, and the learning is compared between the feature output and the classification output. The application unit collects rehabilitation training video data from patients and extracts corresponding skeletal modal data through a pose estimator. The video data and the corresponding skeletal modal data are then input into the trained visual classification model and skeletal classification model, respectively, for classification.

8. A rehabilitation behavior analysis device based on dynamic visual cues, characterized in that, It includes a video acquisition device, a memory, a processor, and a computer program stored in the memory and executable on the processor. The memory also stores a visual classification model and a skeletal classification model constructed and trained by the rehabilitation behavior analysis method based on dynamic visual cues as described in any one of claims 1-6. When the processor executes the computer program, it performs the following steps: The rehabilitation training video data of the patient is collected by a video acquisition device, and the corresponding skeletal modal data is extracted by a posture estimator. The video data and the corresponding skeletal modality data are input into the trained visual classification model and skeletal classification model, respectively, for classification, and the classification results are output.

Images