Neural rehabilitation action detection method based on domain generalization neural network

Abstract

The invention provides a neural rehabilitation action detection method based on a domain generalization neural network, and the method comprises the steps: obtaining the neural rehabilitation action video frame data and descriptive text data of a patient in real time, inputting the preprocessed standard input data into a neural rehabilitation action detection model obtained through training, and outputting a motion quality evaluation result score; the newly designed neural rehabilitation action detection model comprises a feature extraction network and a multi-mode neural rehabilitation action detection network. The feature extraction network is used for extracting spatio-temporal features in a video and combining semantic information in text description to form efficient multi-modal feature representation, and the multi-modal neural rehabilitation action detection network comprehensively integrates key information of the video and the text and outputs a motion quality evaluation result score; meanwhile, a domain generalization generative adversarial network training strategy is designed for model training; the problem of distribution difference between the source domain data and the target domain data is effectively solved, and the domain generalization ability of the model under different training scenes and equipment conditions is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of neural rehabilitation intelligent detection based on multi-modal data fusion, and particularly relates to a neural rehabilitation action detection method based on domain generalization neural network. BACKGROUND

[0002] Patients with movement disorders caused by central nervous system injury usually undergo rehabilitation training to promote neural functional reorganization, thereby gradually recovering fine motor skills. Current rehabilitation guidance relies on the observation and assistance of clinicians, but medical resources are limited, especially in the context where intelligent medical devices have not yet been widely popularized, and more than 90% of rehabilitation training needs to be completed by patients at home. Due to the lack of professional rehabilitation assessment and real-time feedback, the training effect and compliance of many patients are significantly affected. Studies have shown that timely and intuitive assessment and feedback can effectively improve the training enthusiasm of patients and improve the rehabilitation effect. Therefore, developing an intelligent rehabilitation action assessment method has important clinical significance and practical application value.

[0003] Currently, traditional rehabilitation action assessment methods have undergone a process of development from manual observation to intelligentization, but existing methods still face many challenges:

[0004] Sensor signal-based recognition methods: such methods usually acquire multi-dimensional data of human actions by wearing accelerometers, gyroscopes and other sensors, and then use traditional machine learning methods (such as support vector machines, decision trees, etc.) for action classification. The advantage of these methods is that they can accurately capture the motion characteristics of the human body without the need for a large amount of environmental dependence. However, due to the need to wear multiple devices, and the possibility of interference with human activity by the sensors, there are problems of inconvenience and poor comfort in actual application.

[0005] Computer vision-based recognition methods: with the advancement of deep learning technology, video analysis-based action recognition has become an important research direction. Through deep learning models such as convolutional neural networks (CNN) and recurrent neural networks (RNN), action features can be automatically extracted from RGB videos or depth images, achieving high-precision action recognition. The advantage of such methods is that they do not rely on wearable devices and can more naturally capture human actions, making them particularly suitable for remote monitoring in home or medical environments. However, video data often faces challenges such as background interference, posture occlusion and lighting changes, increasing the complexity of action recognition.

[0006] Intelligent recognition method based on deep learning: In recent years, with the wide application of deep learning technology, the action recognition method based on neural network has made significant progress. By introducing advanced network structures such as convolutional long short-term memory network (LSTM), three-dimensional convolutional neural network (3D CNN), etc., temporal and spatial features can be automatically extracted and accurately classified. This kind of method can improve the accuracy of action recognition, and can process complex time sequence information, significantly improving the robustness of the model. However, these deep learning models often have a large number of parameters and high computational complexity, and in the case of insufficient training data or domain difference, they show poor generalization ability.

[0007] Although the existing neural rehabilitation action detection methods perform well in specific scenarios, these methods usually lack multi-modal information and cannot effectively integrate multi-modal data such as text description of movement and neural rehabilitation movement video. In addition, these detection methods usually lack domain generalization ability, that is, the model trained in a certain scene will significantly decrease in detection effect when applied in different training environments or device conditions. SUMMARY

[0008] To solve the above problems, the present application effectively integrates the spatio-temporal features in the neural rehabilitation movement video and the semantic information in the text description of movement by introducing a multi-modal fusion strategy, fully utilizes the complementarity of multi-modal data, and improves the accuracy and robustness of action detection. In order to further solve the problem of data distribution difference, the present application combines generative adversarial network (GAN) for domain adversarial training, thereby reducing the difference between source domain and target domain feature distribution, so that the model can maintain stable performance under different training conditions and device environments.

[0009] The present application provides a neural rehabilitation action detection method based on domain generalization neural network, comprising the following steps:

[0010] S1, real-time acquisition of patient's neural rehabilitation action video frame data and descriptive text data, the descriptive text data is generated by the guiding text in the rehabilitation movement manual corresponding to the different segment neural rehabilitation movement video of the patient;

[0011] S2, pre-processing of the two kinds of data to obtain model standard input data;

[0012] S3, inputting the standard input data into the trained neural rehabilitation action detection model to output the movement quality evaluation result score;

[0013] The neural rehabilitation action detection model comprises a feature extraction network and a multi-modal neural rehabilitation action detection network;

[0014] The feature extraction network comprises a frame-level skeleton key point extraction module, a human body joint action coordinate graph construction module, a space-time graph convolution network coding module STGCN, a multi-scale time convolution module TCN and an action description text coding module; for video modal data, the graph structure data of the joint node is extracted through the frame-level skeleton key point extraction module and the human body joint action coordinate graph construction module, then the STGCN and the TCN are used as the video feature encoder, the converted skeleton data is processed through the STGCN, the high-order topological structure between the body joints is obtained, the spatial features related to the motion are extracted, then three TCNs are used to extract time features at different levels, and the high-level and low-level space-time features at different time spans are concatenated; for text modal data, the action information in the video is converted into a feature vector associated with the text description through the action description text coding module; the video modal feature outputs of each step are concatenated to perform multi-modal fusion with the text modal feature, to form the input of the multi-modal neural rehabilitation action detection network, and to predict the output continuous evaluation score.

[0015] Preferably, the feature extraction network is trained through the constructed domain generalization generative adversarial network training strategy; the feature extraction network is used as a generator module of the generative adversarial network, and a feature mapping layer is combined to realize effective conversion of data; the feature mapping layer uses a linear transformation to splice different modalities of the feature vector of the source domain obtained by the feature extraction network, and then maps the feature to a target space to maintain the similarity of the source domain and the target domain data; the generator module shares parameters for the source domain and the target domain, and the target of the generator module is to minimize the distance between the generated features of the source domain and the target domain, and the optimization objective function is formula:

[0016] minL=||S-T|| 2

[0017] Wherein, S and T respectively represent the generated features of the source domain and the target domain, and L is the distance between the generated features of the source domain and the target domain.

[0018] The discriminator module comprises a video discriminator and a text discriminator, which respectively receive the video feature data and the text feature data output from the generator, process the feature data through two fully connected layers, the output dimension of the first layer is 512, and the output dimension of the second layer is 256, both of which adopt a ReLU activation function; then, the processed feature data is transmitted to an output layer, the output layer comprises a single neuron, and a Sigmoid activation function is used for binary classification, and the output value represents the similarity between the features of the source domain and the target domain, which is mathematically expressed as formula:

[0019] D(X)=σ(W out ·X+b out ),X∈{S,T}

[0020] where D(X) is the output of the discriminator, i.e., the input of the generator G, σ is the Sigmoid function, W and b are the weight and bias of the output layer, respectively, and the objective function of the discriminator is to maximize the difference between the source domain features and the target domain features, as shown in the following formula: out and b out are the weight and bias of the output layer, respectively, and the objective function of the discriminator is to maximize the difference between the source domain features and the target domain features, as shown in the following formula:

[0021] maxL = E[log D(S)] + E[log(1-D(T))]

[0022] where E represents expectation, and log represents a logarithmic function.

[0023] Preferably, the preprocessing includes denoising, supplementing, and normalizing the video data.

[0024] For video data, wavelet denoising and time series smoothing processing are used, the depth image of the i-th frame is respectively decomposed into discrete wavelets, the low-frequency component is retained, and the high-frequency component is processed by soft threshold to remove noise; then the depth image sequence after wavelet denoising is respectively processed by time series smoothing to eliminate inter-frame jitter and time discontinuity; finally, the processed video data is normalized, as shown in the following formula:

[0025]

[0026] wherein represents the i-th frame of video data after normalization, is the i-th frame of video data after wavelet denoising and time series smoothing processing, and μ and σ represent the mean and standard deviation of all frame data, respectively.

[0027] Preferably, the frame-level skeleton key point extraction module first processes each frame of video of the preprocessed video data V using an OpenPose model. For the i-th frame of video, the i-th frame of video is input into the OpenPose model to obtain a set of skeleton key points wherein represents the two-dimensional coordinates of the k-th skeleton key point in the i-th frame, and K is the total number of skeleton key points. After inputting all videos into the OpenPose model, a set of skeleton key point sequences is obtained, wherein N represents the total number of frames of the video; secondly, in order to ensure the consistency across frames, a nearest neighbor matching algorithm is used to match consecutive frames in the set of skeleton key point sequences P V to generate a skeleton action sequence with a time sequence relationship, and a skeleton action sequence is obtained. The skeleton information extracted from each frame serves as a time step, recording the motion trajectory of each joint of the target human body, and providing time sequence skeleton data for subsequent topology graph construction and feature extraction.

[0028] Preferably, the human joint motion coordinate graph construction module converts the skeleton motion sequence into a skeleton key point topology graph g=(A, V, θ), wherein V represents joints as nodes of the graph, θ represents connection relationships between the joints, and A is an edge weight matrix; the node set of each topology graph represents all key points, and the edge set represents connection relationships between the joints, ensuring that the spatial structure between the joints is retained.

[0029] Preferably, the spatio-temporal graph convolution network coding module is configured to extract spatio-temporal features from the skeleton key point topology graph g, including two time-domain convolution blocks and one space-domain convolution block, wherein the time-domain convolution blocks first extract spatial features of the skeleton key point topology graph g by graph convolution operation, and the calculation is as follows:

[0030]

[0031] wherein, and represent time convolution and connection operation, respectively, Γ μ is a time convolution kernel, W k is a learnable matrix, and σ is a sigmoid activation function, X j is the jth frame of video data, h S is the spatio-temporal feature of the current video frame; wherein,

[0032]

[0033] A k is the kth sub-matrix in A, D is a diagonal matrix, D k is the kth sub-matrix in D, and I is an identity matrix; then the spatio-temporal feature h and the skeleton key point topology graph g are input into a GLU activation function layer to obtain a feature GLU(h), the GLU activation function layer controls the information flow by element-wise weighted summation and activation of the input spatio-temporal feature h and the skeleton key point topology graph g, so that the network can capture more complex feature representation; the GLU(h) is input into a space-domain convolution to obtain a feature The space-domain convolution helps to extract the spatial dependency between the nodes of GLU(h), and provides more rich spatial features for subsequent spatio-temporal modeling.

[0034] Preferably, the multi-scale time convolution module includes a multi-scale time convolution layer and a feature concatenation layer; first, the time feature is input into the multi-scale time convolution layer to perform convolution operation on the time feature at three time scales of 5 frames, 10 frames and 20 frames, respectively, to obtain features h hop5 , h hop10 and h hop20, extract local and global dependency information related to the current time step, effectively capturing short-term and long-term motion change patterns; then, the output h hop5 , hop10 , and h hop20 of the multi-scale time convolution layer are input into the feature splicing layer to obtain the spliced feature representation as shown in the formula:

[0035] h TCN = [h hop5 , h hop10 , h hop20 ]

[0036] Capture multi-level temporal dependencies at different time scales and integrate these multi-scale information to obtain more rich and comprehensive feature representation.

[0037] Preferably, the action description text encoding module uses the OpenAI open source pre-trained CLIP model to realize action description text feature extraction and feature alignment; the action description text data T and the output h TCN of the multi-scale time convolution module are input into the CLIP model to obtain a high-dimensional feature vector l of the action description text, and the dimension of l is the same as that of h TCN ; l contains specific action vocabulary semantic feature information and corresponding joint part semantic feature information when performing neural rehabilitation action, including action elements, time information and spatial relationship in the action text description.

[0038] Preferably, the multi-modal neural rehabilitation action detection network comprises a convolutional long short-term memory network pooling module ConvLSTM, a multi-modal fusion module and an action detection output module.

[0039] Firstly, the convolutional long short-term memory network pooling module is responsible for modeling the time dependence of the time sequence video features, extracting long and short-term time features in the action data, enhancing the understanding of the model to the time dynamics and realizing the dimension reduction effect of the pooling layer; then, the multi-modal fusion module is used to mine the inter-modal shared features that are beneficial to the rehabilitation exercise prediction accuracy through the cross co-attention of the spatio-temporal features of the neural rehabilitation exercise video and the semantic information of the text description; finally, the action detection output module is used to perform exercise quality assessment according to the fused features and output the result score.

[0040] Preferably, the input of the convolutional long short-term memory network pooling module is the spliced feature h TCN = [h hop5 , h hop10 , h hop20 ] output by the multi-scale time convolution module; containing the motion trajectory features of the patient at different time spans.

[0041] Firstly, the feature vectors hhop5 ,h hop10 ,h hop20 The feature Z is obtained through splicing, then time series modeling is performed on Z, a gating mechanism is used to dynamically process the information in the time dimension, and complete spatio-temporal features are extracted under different lengths of input;

[0042] The ConvLSTM is set to a single layer, the number of hidden units is 256, and for the input sequence Z, the calculation formula is as follows:

[0043] M = ConvLSTM (Z)

[0044] Where M is the feature output by the ConvLSTM unit, and the feature sequence M output by the ConvLSTM is converted into a learnable attention feature map g' through the following formula:

[0045] g' = sigma (phi (A o M) ZW)

[0046] Where o represents Hadamard product, phi represents a standardization function, W is a weight vector obtained through 1-dimensional convolution, and A is the edge weight matrix of the skeleton key point topology g. Finally, the attention feature map g' is output through the splicing of the features of the time series convolution, and the feature h of the video is obtained. f .

[0047] Compared with the prior art, the present application has the following beneficial effects:

[0048] (1) Improve the intelligence and automation level of action detection: the present application constructs an end-to-end neural rehabilitation action detection model based on STGCN and LSTM, which significantly improves the intelligence and automation level of neural rehabilitation action detection. The designed feature extraction model can efficiently extract complex nonlinear features in action data, and through the end-to-end architecture, the whole detection process is simplified, and the overall performance of the model is improved. Compared with the traditional method which requires a large amount of labeled data, the present application can realize high-precision prediction with only a small amount of data, significantly reducing the cost and difficulty of data acquisition.

[0049] (2) Improve the robustness and cross-domain generalization ability of the model: the generative adversarial network model constructed by the present application effectively solves the problem of distribution difference of neural rehabilitation data between different domains through the interactive training of the generator and the discriminator, and realizes the efficient generalization of the source domain and the target domain data. Combined with the long short-term memory network, the present application can capture the long-range dependence in the time series and optimize the feature extraction process, and improve the adaptability of the model in the variable rehabilitation environment.

[0050] (3) Enhancing the model's multi-modal understanding ability: The invention significantly improves the model's understanding of neurorehabilitation movements by introducing a multi-modal fusion strategy. By combining the spatiotemporal features of neurorehabilitation movement videos with the semantic information of textual descriptions, the complementary information in different modal data is fully exploited. Combined with the ConvLSTM layer, the model can analyze the role of different body joints in the movement evaluation process. The model can integrate the key clues contained in multi-modal data to generate more semantically deep feature representations. This fusion strategy not only enhances the model's understanding of movement data, but also provides a more comprehensive basis for the evaluation of different rehabilitation movements.

[0051] (4) Improving real-time monitoring and decision support ability: The invention successfully applies to actual neurorehabilitation training through the optimized cross-domain model, which can monitor the patient's movement performance in real time and provide accurate evaluation results, helping clinical staff to identify potential problems and adjust rehabilitation programs in a timely manner. Compared with the existing technology which requires a lot of manual intervention, the invention realizes the intelligentization and automation of the whole process, effectively improving the efficiency and quality of the neurorehabilitation process, ensuring the scientificity and efficiency of patient rehabilitation. At the same time, by reducing the error of rehabilitation training, the training effect is significantly improved, the patient's rehabilitation speed is improved, and the patient's continuous progress in the rehabilitation process is ensured. This real-time detection mechanism not only improves the effect of neurorehabilitation treatment, but also guarantees the patient's rehabilitation process, improves the treatment efficiency, and ensures the efficient use of medical resources. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The overall technical route flowchart of the invention.

[0053] Figure 2 The topological graph construction schematic diagram of the invention.

[0054] Figure 3 The space-time graph convolution network coding module schematic diagram of the invention.

[0055] Figure 4 The generative adversarial network schematic diagram of the invention.

[0056] Figure 5 The multi-modal neurorehabilitation movement detection network schematic diagram of the invention.

[0057] Figure 6 The convolutional long short-term memory network pooling module schematic diagram of the invention.

[0058] Figure 7 The experimental result comparison diagram of the invention. DETAILED DESCRIPTION

[0059] The application effectively integrates the spatial and temporal features in the neural rehabilitation movement video and the semantic information in the movement text description by introducing a multi-modal fusion strategy, fully utilizes the complementarity of multi-modal data, and improves the accuracy and robustness of action detection. In order to further solve the problem of data distribution difference, the application combines a generative adversarial network (GAN) for domain adversarial training, thereby reducing the difference between the feature distributions of the source domain and the target domain, and enabling the model to maintain stable performance under different training conditions and device environments. The overall technical route is as shown in Figure 1

[0060] S1, initial data set formation: the neural rehabilitation action detection method in the application relies on two types of multi-modal data: source domain data and target domain data. The source domain data is obtained by recording labeled neural rehabilitation action videos and corresponding action description texts using an RGBD camera in an experimental environment, and the target domain data is un-labeled movement videos and auxiliary description texts recorded by the same device in a patient's home environment. The two types of data are subjected to consistent data preprocessing, including noise removal, standardization, coordinate conversion processing of video data, and word segmentation, semantic standardization and embedding representation of text data. The video modality is converted into a graph structure representation by extracting the coordinate information of each body joint, and the text modality generates a semantic feature vector through a text encoding module for subsequent multi-modal fusion processing.

[0061] ​S2, feature extraction network construction: the feature extraction network is composed of a frame-level skeleton key point extraction module, a human joint action coordinate graph construction module, a space-time graph convolution network (STGCN) encoding module, a multi-scale time convolution module (TCN), and an action description text encoding module. The time feature, space feature and text semantic feature of the neural rehabilitation action data are effectively captured. First, for video modal data, a video feature encoder is used for feature extraction. First, the graph structure data of the joint nodes is extracted by the frame-level skeleton key point extraction module and the human joint action coordinate graph construction module, and then the combination of STGCN and TCN is used as the video feature encoder. The converted skeleton data is processed by STGCN to explore the high-order topological structure between the joints of the body and extract the spatial features related to the movement, reflecting the movement pattern of each part of the body. Then, three TCNs are used to extract different levels of time features. The high and low level time-space features are connected in different time spans to express the dynamic video information of the video. For text data, each type of rehabilitation exercise requires the patient to learn according to the specified text description in the rehabilitation manual. The text of the learned neural rehabilitation exercise is input into the action description text encoding module, and a pre-trained CLIP model is used as a text feature encoder. Through the image-text contrast learning mechanism of the CLIP model, the action information in the video is converted into a feature vector associated with the text description. Finally, the video modal feature outputs of each step are concatenated and multi-modal fused with the text modal feature to predict the continuous evaluation score.

[0062] S3, constructing domain generalization generative adversarial network training strategy: in order to improve the adaptability of the model in different rehabilitation environments, the application proposes a domain generalization training strategy based on generative adversarial network (GAN). First, the generator module takes the different modal features extracted by the video feature encoder and the text feature encoder in the feature extraction network of the source domain and the target domain as input, and then inputs the different modal features into the feature mapping module to map to the target space. In this process, the model can extract the shared features and unique features of the data of the two domains. Then, the features are input into the discriminator module, and the discriminator judges the source of the input sample by comparing the feature distribution difference between the input sample and the real target domain sample. If the feature distribution difference between the sample generated by the generator and the real target domain sample is large, the discriminator will give a lower discrimination score. Through continuous adversarial training, the generator will continuously improve to generate more realistic samples, so that the discriminator is difficult to distinguish between generated samples and real samples. The core idea of the strategy is to force the generator to learn a feature representation that can be applied to both the source domain and the target domain through adversarial training, so as to promote more accurate neural rehabilitation movement prediction of the model in different scenarios. The proposed generative adversarial network training strategy is only used in the training stage to enhance the adaptability of the encoder between different domains. In the test and deployment stage, the training strategy is not used, but the feature encoding network of different modalities contained in the generator is directly used.

[0063] S4, constructing a multi-modal weighted neural rehabilitation action prediction network: in order to improve the accuracy of neural rehabilitation action detection, the application constructs a composite prediction model including a convolutional long short-term memory (ConvLSTM) encoding module, a multi-modal fusion module and an action detection output module. First, the ConvLSTM module is responsible for modeling the time dependence of the time series video features, extracting long and short term time features in the action data, enhancing the understanding of the model to the time dynamics and realizing the dimension reduction effect of the pooling layer. Then, the multi-modal fusion module is used to mine the inter-modal shared features that are useful for rehabilitation movement prediction from the spatio-temporal features of the neural rehabilitation video and the semantic information of the text description through cross co-attention. Key features are mined from multiple dimensions to highlight the influence of the core joints in the action, thereby making up for the problem of insufficient single modal features. Finally, the action detection output module is used to predict according to the fused features. The model not only can provide accurate prediction in complex environment, but also can maintain good performance in cross-domain situation, adapt to diversified patient data.

[0064] S5, model deployment and use: the neural rehabilitation action detection model designed by the present application can obtain the motion data of the patient in real time by combining with the RGBD camera. After data preprocessing and feature extraction, the model can evaluate the patient's motion in real time, including real-time video stream and learned rehabilitation motion guidance text, which is standardized and used as input model data. The feature extraction network converts the video stream input and descriptive text into video features and text features, where the video feature encoder and text feature encoder come from the generator module optimized for domain generalization performance during the training process using the generative adversarial network training strategy. The features of the two modalities will be input to the multi-modal weighted neural rehabilitation action prediction network, and the rehabilitation motion quality score will be predicted in time to feedback the rehabilitation progress or potential problems. In practical application, the model can effectively evaluate the patient's motion performance in the home environment and compare it with the motion in the laboratory environment, helping the rehabilitation doctor to develop more accurate individualized rehabilitation plans.

[0065] The application will be further described below in conjunction with specific embodiments.

[0066] I. Dataset construction

[0067] 1. Original neural rehabilitation motion data collection: as a neural rehabilitation data-driven technology, the present application needs to collect RGBD camera data and action text description data of different patients during the neural rehabilitation process. These data include the motion trajectory, joint angle and motion timing of different patients during the neural rehabilitation motion, which are of different lengths and can be classified into source domain patient data V S and target domain patient data V T . At the same time, in order to make up for the deficiency of visual data, the present application also introduces the text description data of the motion. These data come from the manual for guiding the patient to perform the rehabilitation motion, and each video data corresponds to the action text description in the learned manual, which covers the action type, target part and execution requirement of the rehabilitation motion, and provides key semantic information. They can be classified into source domain patient data T S and target domain patient data T T .

[0068] 2. Video data preprocessing and conversion: considering that the motion data collected by the RGBD camera is often disturbed by noise, light and other factors, the present application performs denoising, supplementing and standardization on the data. Specifically, for video data from the source domain V S and the target domain V T , wavelet denoising and time series smoothing are used to process the depth image and Discrete wavelet decomposition is performed, the low frequency component is reserved, and the high frequency component is processed by soft threshold to remove noise, as follows:

[0069]

[0070] wherein, and are high frequency and low frequency components of wavelet decomposition of the depth image respectively.

[0071] TH(H) represents a soft threshold function of the high frequency component, and is defined as:

[0072] TH(H) = sign(H) * max(|H| - lambda, 0)

[0073] wherein lambda is a threshold value, and IW(.) represents an inverse wavelet transform. Further, the depth image sequence after wavelet denoising is respectively subjected to time sequence smoothing processing, so as to eliminate inter-frame jitter and time discontinuity.

[0074] The depth sequence of the source domain is denoted as V The formula is adopted as follows:

[0075]

[0076] The depth sequence of the target domain is denoted as V The formula is adopted as follows:

[0077]

[0078] wherein omega j is a Gaussian weighting function, k is a sliding window size, and controls a smoothing range. Thus, the optimized video input data V and V are obtained, so as to ensure consistency and accuracy of the data.

[0079] By collecting RGBD camera data under different scenes, and through wavelet denoising and time sequence smoothing processing, the data denoising and optimization processing are completed. In order to further improve the consistency of the data and the model generalization performance, for the source domain data V S1 and the target domain data V T1 , the application performs standardization operation on the processed video data. As shown in the formula:

[0080]

[0081] wherein, represents the i-th frame of video data after the standardization operation, is the i-th frame of video data after wavelet denoising and time sequence smoothing processing, mu and sigma respectively represent the mean and standard deviation of all frame data, and the calculation is as follows:

[0082]

[0083]

[0084] V S1 i frame-level data in and V T1 i frame-level data in Perform the above standardization processing to generate a more uniformly distributed source domain video data V S2 and target domain video data V T2 , providing more stable input data for subsequent feature extraction and domain alignment.

[0085] 2. Construction of Neurorehabilitation Movement Detection Model

[0086] The neurorehabilitation action detection model includes a feature extraction network and a multimodal neurorehabilitation action detection network; the feature extraction network includes a frame-level skeleton key point extraction module, a human joint action coordinate graph construction module, a spatiotemporal graph convolutional network encoding module (STGCN), a multi-scale temporal convolution module (TCN), and an action description text encoding module; for video modality data, the frame-level skeleton key point extraction module and the human joint action coordinate graph construction module are used to extract the graph structure data of the joint points, and then STGCN and TCN are used as video feature encoders. The converted skeleton data is processed by STGCN to obtain the high-order topological structure between body joints and extract motion-related spatial features. Then, three TCNs are used to extract temporal features at different levels, and the high- and low-level spatiotemporal features are concatenated at different time spans; for text modality data, the action information in the video is converted into a feature vector associated with the text description through the action description text encoding module; the video modality feature output of each stride is concatenated and multimodally fused with the text modality features to form the input of the multimodal neurorehabilitation action detection network, and a continuous evaluation score is predicted and output.

[0087] 1. Feature Extraction Network

[0088] The feature extraction model constructed by the present invention includes a frame-level skeleton key point extraction module, a human joint motion coordinate graph construction module, a spatiotemporal graph convolutional network (STGCN) encoding module, a multi-scale temporal convolutional module (TCN), and an action description text encoding module; it can simultaneously capture the temporal and spatial features of neurorehabilitation action data. First, the frame-level skeleton key point extraction module is used to extract the preprocessed source domain V S2 and the target domain V T2The frame-level skeleton key point information is extracted from the video data to form preliminary feature data representing the motion state of each joint of the body. Then, the key point data is converted into a topological graph form by using a human joint action coordinate graph construction module to reflect the connection relationship and topological structure between joints, laying a foundation for subsequent feature extraction. Subsequently, a spatio-temporal graph convolution network (STGCN) is used to encode the constructed human joint action coordinate graph to explore the high-order topological relationship and dynamic interaction between joints. This stage can extract spatial features related to the motion pattern, fully showing the motion state of each part of the body. Then, a multi-scale temporal convolution module (TCN) is used to model features of different time scales, including three temporal convolution layers, to extract different levels of temporal features and integrate high and low level features to further enhance the representation ability of time series changes. Finally, the feature outputs of each time step are concatenated to provide high-quality input features for the generator and discriminator modules in the subsequent generative adversarial network and multi-modal neural rehabilitation action detection model. In complex neural rehabilitation scenarios, this feature extraction model can efficiently capture key features in the spatial and temporal dimensions, even in cases of large data dynamic changes or small data volumes, while maintaining high-precision feature expression capabilities, providing reliable support for real-time motion assessment and rehabilitation training optimization.

[0089] The frame-level skeleton key point extraction module: first, the input video features are processed to extract skeleton key point information from the preprocessed video data in the source domain and the target domain. The OpenPose model is used to process each frame of video in the source domain multi-modal data S={V S2 ,T S} and the target domain data T={V T2 ,T T}. First, the OpenPose model is used to process each frame of the preprocessed video data V. For the i-th frame, the i-th frame of video is input into the OpenPose model to obtain the skeleton key point set where represents the two-dimensional coordinates of the k-th skeleton key point in the i-th frame, and K is the total number of skeleton key points. After inputting all the video into the OpenPose model, the skeleton key point sequence set P is obtained, where N represents the total number of frames of the video. Second, in order to ensure the consistency across frames, a nearest neighbor matching algorithm is used to match consecutive frames in the skeleton key point sequence set P V to generate a skeleton action sequence with a time sequence relationship, obtaining the skeleton action sequence For the source domain data, the skeleton action sequence is obtained. For the target domain data, the skeleton action sequence is obtained.Wherein N represents the total number of frames of the video. The skeleton information extracted from each frame is taken as a time step, which records the motion trajectory of each joint of the source domain and the target domain target human body, and provides time sequence skeleton data for subsequent topology graph construction and feature extraction.

[0090] The human joint action topology graph construction module: this module is used to convert the skeleton key point information in the source domain video data V S3 and the target domain video data V T3 into the topology graph form of human joint action. The purpose of this process is to establish the spatial relationship between the joints, and to provide a basis for subsequent feature extraction and spatio-temporal relationship modeling. Specifically, for the source domain video data V S3 , the skeleton key point information will be converted into the topology graph G S =(V S ,θ S ), and for the target domain video data V T3 , the skeleton key point information will be converted into the topology graph G T =(V T ,θ T ). Wherein V S and V T represent the joints as the nodes of the graph, and θ S and θ T represent the connection relationship between the joints, and the weight matrix A S and A T of the edge are constructed. The node set of each topology graph represents all the key points, and the edge set represents the connection relationship between the joints. Each frame in the source domain and the target domain generates a corresponding joint topology graph, ensuring that the spatial structure between the joints is preserved. In this way, the human action data in the source domain and the target domain is converted into graph structure data, providing input for the subsequent spatio-temporal graph convolution network (STGCN) encoding module. These topology graphs capture the dynamic interaction relationship between each joint of the human body, and further provide rich spatial information for feature learning and action recognition. The specific process is shown in Figure 2 .

[0091] The spatio-temporal graph convolution network (STGCN) encoding module: in the present application, the spatio-temporal graph convolution network (STGCN) encoding module is used to extract spatio-temporal features from the skeleton key point data of the source domain and the target domain, and serves as an important component of the video feature encoder. First, the input data of the source domain and the target domain are the source domain skeleton key point topology graph G S and the target domain skeleton key point topology graph T S, the nodes of each graph represent human joints, and the edges represent the connection between joints. Each STGCN includes two temporal convolution blocks and one spatial convolution block, where the temporal convolution block first extracts spatial features by graph convolution operation, and the spatial features of the source domain and the target domain are calculated as follows:

[0092]

[0093] where h S i and h T i are the new features of the source domain and the target domain node i, h S j and h T j are the features of the neighbor node j, N(i) is the neighborhood set of node i, W S and W T are the learning weight matrices of the source domain and the target domain respectively, b S and b T are the bias terms. Then the GLU activation is used to obtain the source domain feature GLU(h S i ) and the target domain feature GLU(h T i ), and the activation function controls the information flow by element-wise weighted sum and activation of the input two parts, so that the network can capture more complex feature representation. Further spatial convolution is performed, which helps to extract the spatial dependency between the nodes of the source domain and the target domain, and provides more rich spatial features for subsequent spatio-temporal modeling. Through this series of operations, the spatio-temporal graph convolution network can learn spatial and temporal features in the source domain and the target domain respectively, and the source domain feature and the target domain feature extracted by STGCN can be obtained, so as to effectively extract the spatio-temporal features of human joint action and provide high-quality feature representation for subsequent action detection task. The specific process is as follows Figure 3 .

[0094] Multi-scale temporal convolution module TCN: in the present application, a multi-scale temporal convolution module is used to extract different levels of time features from the spatio-temporal features of the source domain and the target domain, and constitutes a video feature encoder together with the STGCN. Specifically, through the multi-scale temporal convolution network (TCN) module, different time scales of features are modeled, and high-level and low-level features are concatenated to further enhance the model's representation ability for time series changes. First, the temporal convolution layer extracts features related to time changes by performing convolution operations on the data at each time step. For the source domain S3 and the target domain T3, multiple STGCN modules are used for feature extraction at 3 different time steps, including high-level and low-level temporal convolutions. Specifically, frame-level data with a stride of 5, 10, and 20 are selected for feature encoding, and these different time span data are used to capture short-term and long-term dependencies in the time series.

[0095] At each time scale, the time series features of the source domain and the target domain are subjected to independent spatio-temporal graph convolution operations to further extract local and global dependency information related to the current time step. These features will be processed by convolution kernels to effectively capture short-term (e.g., 5 frames) and long-term (e.g., 20 frames) motion change patterns. For different time scales, the model uses convolution operations to capture data features within different time lengths, while adjusting the size of the convolution kernel to adapt to different time steps, thereby enhancing the model's representation ability. Then, the source domain S3 and the target domain T3 features after different scale convolutions are concatenated respectively to form more rich feature representations as follows:

[0096] h S TCN =[h S hop5 ,h S hop10 ,h S hop20 ]

[0097] h T TCN =[h T hop5 ,h T hop10 ,h T hop20 ]

[0098] wherein h S hop5 and h T hop5 are features encoded by STGCN with a stride of 5 frames; h S hop10 and h Thop10 is the case of 10 frames as a stride; h S hop20 and h T hop20 is the case of 20 frames as a stride. In this way, the model can capture multi-level temporal dependencies at different time scales and integrate these multi-scale information to obtain more rich and comprehensive feature representation. These multi-level features will serve as input to further promote the generation and detection tasks of subsequent models, thereby improving the performance of the source domain and the target domain in different tasks, especially in the accurate capture of time features in tasks such as neural rehabilitation action detection.

[0099] Action description text encoding module: The design of this module aims to enhance the multi-modal understanding ability of the neural rehabilitation action detection model by extracting the semantic features of the action description text. It contains an encoder for text features. Specifically, for the input text data T S and T T and the extracted video features h S TCN and h T TCN , a pre-trained CLIP model is used for feature extraction to obtain the source domain feature l S and the target domain feature l T of the text. The action description text is input into the CLIP model as a supplementary information, and the CLIP model jointly encodes the text and the video to generate a high-dimensional feature vector of the action description text. Through the pre-trained multi-modal contrastive learning mechanism of the CLIP model, the semantic information of the text is preferentially aligned with the visual features of the video. The text input is processed by the text encoder of the CLIP, and the generated text features can capture the action elements, temporal information and spatial relationships in the description, which are crucial for the recognition of neural rehabilitation actions. The pre-trained CLIP used not only understands specific action words in the text (such as "raise arm" and "squat down"), but also captures the execution manner and related body parts according to the context. For example, for the description of the squatting action, the CLIP model can extract action information related to joint parts such as "ankle", "knee" and "spine", and convert them into numerical feature vectors, which are integrated into the subsequent model processing.

[0100] 2. Domain generalization generative adversarial network training strategy

[0101] Generator module (coding module): the generator module G constructed by the application is a feature extraction network, and is combined with a feature mapping module to realize effective conversion of data. The feature mapping layer uses a linear transformation to splice different modalities of the feature vectors of the source domain obtained by the video feature encoder and the text feature encoder, and then maps the features to a target space to maintain the similarity of the source domain and the target domain data. It is expressed as formula:

[0102] S=G(f map (l S ,h S TCN )))

[0103] T=G(f map (l T ,h T TCN ))

[0104] Where the generator G shares parameters for the source domain and the target domain. In the generator, the neural rehabilitation actions of the source domain and the target domain are input respectively, and are processed by the shared feature mapping module, so that the generated features of the two are as close as possible. This design not only enhances the comparability of the generated features, but also provides greater identification challenge for the subsequent discriminator. The goal of the generator is to minimize the distance between the generated features of the source domain and the target domain, and the objective function is optimized as formula:

[0105] minL=||S-T|| 2

[0106] Where S and T represent the generated features of the source domain and the target domain respectively, and L is the distance between the generated features of the source domain and the target domain. This process ensures the robustness and adaptability of the model when processing different domain data.

[0107] Discriminator module (distinguishing module): the discriminator module constructed by the application is used to effectively distinguish the neural rehabilitation movements of the source domain and the target domain. The discriminator module D includes a video discriminator and a text discriminator, which respectively receive the video feature data and the text feature data output by the generator, and process the feature data through two fully connected layers. The output dimension of the first layer is 512, and the output dimension of the second layer is 256, both of which use ReLU activation function to enhance the nonlinear representation ability of the features. Subsequently, the processed feature data is transmitted to the output layer, which includes a single neuron using Sigmoid activation function for binary classification. The output value represents the similarity between the features of the source domain and the target domain, which is mathematically expressed as formula:

[0108] D(X)=σ(W out ·X+b out ),X∈{S,T}

[0109] where D(X) is the output of the discriminator, i.e., the input of the generator G, σ is the Sigmoid function, W out and b out are the weights and bias of the output layer, respectively. The goal of the discriminator is to improve the discriminative ability of the model by maximizing the difference between the source domain features and the target domain features, and its optimization objective function is formula:

[0110] maxL = E[log D(S)] + E[log(1-D(T))

[0111] where E represents expectation, and log represents the logarithmic function. In this process, the discriminator module ensures that the source domain and target domain data can be effectively distinguished, and the robustness of the model is improved. Through such design, the discriminator module can not only distinguish the source domain and target domain data, but also strengthen the model's ability to judge the similarity of the source domain and target domain features.

[0112] Domain generalization adversarial training strategy: The domain generalization adversarial training strategy of the present application aims to improve the adaptability of the model between different domains through the adversarial learning between the generator module and the discriminator module. In the training process, the generator and the discriminator are alternately optimized to realize the effective fusion of the source domain and target domain features. The goal of the generator is to generate target domain features similar to the source domain features, so as to "deceive" the discriminator, while the goal of the discriminator is to accurately distinguish the source domain features and the target domain features. Specifically, the loss function of the generator is defined as formula:

[0113]

[0114] where, and represent expectation, respectively. Through such adversarial training strategy, the video feature encoder (STGCN and TCN) and the text feature encoder CLIP in the generator will be optimized in the continuous game with the video discriminator and the text discriminator, so that the generated target domain features are close to the source domain features in distribution, thereby enhancing the generalization ability of the model and improving the prediction accuracy under different neural rehabilitation action detection conditions. The specific structure is as shown in Figure 4 .

[0115] 3. Multi-modal neural rehabilitation action detection network

[0116] The video features and text features output by the feature extraction network are used as data samples, and the patient rehabilitation score is used as a label to train a multi-modal neural rehabilitation action detection network; the multi-modal neural rehabilitation action detection network proposed in the application integrates a convolutional long short-term memory (ConvLSTM) encoding module, aiming to improve the accuracy of neural rehabilitation action detection. The ConvLSTM encoding module replaces the traditional pooling layer, so that the model can input variable-length video data and better focus on modeling the time dependence of action data, extracting long and short-term time features. More importantly, the role of each joint in each movement is different, and capturing this joint role is crucial for determining the quality of movement. However, ordinary STGCN treats all body joints equally. When evaluating rehabilitation movements, this joint role should change according to the changes in the temporal and spatial context. This motivates the implementation of a multi-modal fusion strategy, in which the co-attention mechanism is used to fuse the skeletal data in the video modality with the movement description text. Through the co-attention mechanism, the model can simultaneously focus on the key information in the video modality and the text modality, dynamically adjusting the importance of each joint in the movement. The overall structure of the multi-modal neural rehabilitation action detection network is shown in Figure 5 .

[0117] Convolutional long short-term memory network pooling module: the convolutional long short-term memory network (ConvLSTM) pooling module designed in the application aims to solve the problem of dependence on fixed input length of traditional global pooling or attention pooling, and realizes flexible processing of variable-length video input. Through this module, the model can effectively capture the spatio-temporal features of neural rehabilitation movement data and adapt to input sequences of different lengths, providing support for neural rehabilitation action detection in variable scenarios.

[0118] The input of this module is the feature-extracted source domain video feature vector h S TCN = [h S hop5 , h S hop10 , h S hop20 ] and the target domain video feature vector h T TCN = [h T hop5 , h T hop10 , h T hop20 ], which respectively contain the movement trajectory features of the patient at different time spans. First, the feature vectors of different steps are spliced and combined to generate a feature representation Z S and Z TThen, the extracted spatial features are input into the ConvLSTM unit for time series modeling. ConvLSTM combines the advantages of convolution operation and long short-term memory network, and overcomes the limitation of traditional pooling methods on the length of input sequence. It uses the gating mechanism (input gate, forget gate and output gate) to dynamically process the information in the time dimension, so that complete spatio-temporal features can be extracted under different lengths of input.

[0119] In a specific implementation, the ConvLSTM unit is set to a single layer, and the number of hidden units is 256, so as to enhance the feature expression ability while maintaining the calculation efficiency. For the input sequence Z S and Z T , the formula can be calculated as:

[0120] M S =ConvLSTM(Z S )

[0121] M T =ConvLSTM(Z T )

[0122] where M S and M T are the features output by the source domain and the target domain respectively. Through the design of variable-length input, the ConvLSTM pooling module replaces the traditional global pooling or attention pooling method, which not only improves the modeling ability of time series features, but also enhances the adaptability of the model in processing different lengths of input data. This provides more flexible and efficient feature representation for the subsequent attention weighting mechanism and action classification module. The feature sequence M S and M T output by the convolution long short-term memory network are converted into learnable attention feature maps through the formula:

[0123] G S =σ(φ(A S ⊙M S )Z S W S )

[0124] G T =σ(φ(A T ⊙M T )Z T W T )

[0125] where ⊙ represents Hadamard product, φ represents standardization function, Z S and Z T are the merged features, and W S and W Tis the weight vector obtained by 1-dimensional convolution. Finally, the features are spliced and output through the time convolution to obtain the video features h S f and h T f The specific process is shown in Figure 6

[0126] Multimodal fusion module: The multimodal fusion module in the application adopts a co-attention mechanism to effectively fuse information from different modalities, so as to improve the accuracy and robustness of neural rehabilitation action detection. In the process of neural rehabilitation action evaluation, video data and action description text serve as two important information sources, which provide temporal and spatial features of movement and semantic understanding, respectively. In order to fully utilize the advantages of the two modalities, a co-attention mechanism is adopted to realize dynamic information fusion of video and text modalities.

[0127] Firstly, the video data is processed by a spatio-temporal graph convolution network (STGCN) and a convolutional long short-term memory network (ConvLSTM) encoding module to extract the spatial features and temporal dependencies of the skeleton data. Specifically, for the video features h S f and h T f and the text features l S and l T The co-attention mechanism dynamically allocates the weight of each modality in the movement recognition task by calculating the similarity between the video and text modalities. This mechanism can automatically adjust the contribution of different modalities during model training, ensuring that the model can rely on the spatial features of the video data at key moments and rely more on the semantic information of the text description at other moments.

[0128] Finally, through the fused multimodal features, the model can more accurately capture the temporal information of the movement and the spatial dynamics of the joints, and also understand the semantic level of the movement. This strategy effectively improves the accuracy of neural rehabilitation action detection and enhances the generalization ability of the model in cross-domain environments, providing strong support for real-time evaluation and personalized feedback of the patient's rehabilitation state in practical applications.

[0129] Action detection output module: The action detection output module in the application is responsible for converting the fused multimodal features into a quality score of the neural rehabilitation action, providing an intuitive prediction result for rehabilitation action evaluation. The core design of the module includes a multilayer perceptron and a fully connected layer, which utilizes nonlinear mapping to perform deep-level processing on the input features and extract key information highly related to the quality of the action. Subsequently, by receiving the multimodal feature fusion features from the multimodal fusion module, the fully connected layer maps the processed features into a prediction score y s ​, quantifying the completion quality of the neural rehabilitation action. The design of this module supports the rapid generation of action quality scores, enabling the system to monitor the patient's rehabilitation progress in real-time and provide medical personnel with accurate and efficient action evaluation criteria.

[0130] III. Model Deployment and Application

[0131] Neural Rehabilitation Data Collection Device Deployment: High-precision RGBD cameras and motion capture devices are deployed in the patient's rehabilitation training environment to collect real-time video data of neural rehabilitation actions in the source and target domains. Descriptive text data is generated from the instructional text in the rehabilitation exercise manual corresponding to different segments of the patient's neural rehabilitation exercise video. Through high-quality data collection, accurate and complete training data are ensured to support subsequent model training and prediction.

[0132] Neural Rehabilitation Action Prediction Model Deployment: The optimized neural rehabilitation action prediction model is deployed in the rehabilitation monitoring system to collect and preprocess RGBD video data from different patients and corresponding rehabilitation exercise descriptive text in real-time. The preprocessed data is input into the trained model to generate multi-modal features in the actual scenario, and then real-time action detection and evaluation are performed using the neural rehabilitation action model to output the exercise quality evaluation result score, helping doctors understand the patient's rehabilitation status in a timely manner.

[0133] Model Verification and Prediction Effect Evaluation: After the model is deployed, actual patient rehabilitation data is collected regularly to verify and evaluate the prediction effect of the model. By comparing with the actual rehabilitation effect, the accuracy and stability of the model are ensured. If necessary, the model is adjusted and optimized to improve the prediction accuracy and adaptability of the model. To verify the effectiveness of the method of the present application, a test set consisting of 20 neural rehabilitation patients and their actual rehabilitation effects is used to verify the method proposed in the present application. As shown in Figure 7 , sample numbers 1-20 represent 20 neural rehabilitation patients, the blue histogram represents the true value of the patient's rehabilitation effect, and the red histogram represents the predicted value of the patient's rehabilitation effect. As shown in Figure 7 , the majority of the rehabilitation exercise quality prediction values of the present application method are less than 5% of the expert evaluation standard error in the 20 test samples, indicating that the present application method has high prediction accuracy and stability, and can provide reliable basis for patient rehabilitation evaluation.

[0134] Intelligent Feedback and Decision Support: Based on the prediction results of the model, combined with the patient's rehabilitation data and actual situation, the system will automatically generate personalized rehabilitation training feedback to help clinical staff develop scientific training programs. According to the predicted exercise trend, the system can early warn the patient's rehabilitation progress problem, timely adjust the training intensity and content, and ensure the safety, effectiveness and medical requirements of the patient's rehabilitation process.

[0135] The above merely provides preferred embodiments of the present application but is not intended to limit the present application. Based on the above teachings, one skilled in the art will be able to devise various modifications and variations without departing from the spirit and scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the present application.

[0136] The above merely provides preferred embodiments of the present application but is not intended to limit the present application. Based on the above teachings, one skilled in the art will be able to devise various modifications and variations without departing from the spirit and scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the present application.

Claims

1. A method for detecting neurorehabilitation movements based on a domain generalization neural network, characterized in that: The following steps are involved: S1, acquiring in real time the patient's neurorehabilitation movement video frame data and descriptive text data, wherein the descriptive text data is generated by the guiding text in the rehabilitation exercise manual corresponding to different segments of the patient's neurorehabilitation movement video; S2, preprocess the two types of data to obtain the model standard input data; S3, inputs the standard input data into the trained neurorehabilitation movement detection model and outputs the movement quality assessment result score; The neurorehabilitation action detection model includes a feature extraction network and a multimodal neurorehabilitation action detection network; The feature extraction network includes a frame-level skeleton key point extraction module, a human joint motion coordinate graph construction module, a spatiotemporal graph convolutional network encoding module STGCN, a multi-scale temporal convolution module TCN and an action description text encoding module; for video modality data, the graph structure data of the joint points are extracted by the frame-level skeleton key point extraction module and the human joint motion coordinate graph construction module, and then STGCN and TCN are used as video feature encoders. The converted skeleton data is processed by STGCN to obtain the high-order topological structure between the body joints and extract the spatial features related to the motion. Then, three TCNs are used to extract temporal features at different levels, and the high- and low-level spatiotemporal features are connected in series at different time spans; for text modality data, the action information in the video is converted into a feature vector associated with the text description by the action description text encoding module; the video modality feature output of each stride is connected in series and multimodally fused with the text modality feature to form the input of the multimodal neurorehabilitation action detection network, and a continuous evaluation score is predicted and output.

2. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 1, wherein: The feature extraction network is trained by constructing a domain generalization generative adversarial network training strategy; The feature extraction network is used as the generator module of the generative adversarial network, and is combined with the feature mapping layer to achieve effective data conversion. The feature mapping layer uses a linear transformation to splice the different modes of the feature vectors in the source domain obtained by the feature extraction network, and then uses feature mapping to transfer them to the target space to maintain the similarity between the source and target domain data. The generator module shares the parameters of the source domain and the target domain. The goal of the generator module is to minimize the distance between the generated features of the source domain and the target domain, and optimize the objective function as follows: min L=||S-T|| 2 Among them, S and T represent the generated features of the source domain and the target domain respectively, and L is the distance between the generated features of the source domain and the target domain; The discriminator module includes a video discriminator and a text discriminator, which receive the video feature data and text feature data output by the generator respectively. The feature data is processed by two fully connected layers. The output dimension of the first layer is 512, and the output dimension of the second layer is 256. Both layers use the ReLU activation function. Subsequently, the processed feature data is passed to the output layer, which contains a single neuron and uses the Sigmoid activation function for binary classification. The output value represents the similarity between the source domain and target domain features, which is mathematically expressed as the formula: D(X)=σ(W out ·X+b out ),X∈{S,T} Among them, D(X) is the output of the discriminator, that is, the input of the generator G, σ is the Sigmoid function, W out and b out are the weights and biases of the output layer respectively. The goal of the discriminator is to maximize the difference between the source domain features and the target domain features. The optimization objective function is: max L=E[log D(S)]+E[log(1-D(T))] Here, E stands for expectation and log stands for logarithmic function.

3. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 1, wherein: The preprocessing includes denoising, supplementing and standardizing the video data. For the video data, wavelet denoising and time series smoothing are used. The depth image of the i-th frame is subjected to discrete wavelet decomposition, the low-frequency components are retained, and the high-frequency components are subjected to soft threshold processing to remove noise. Then, the depth image sequence after wavelet denoising is subjected to time series smoothing to eliminate inter-frame jitter and time discontinuity. Finally, the processed video data is standardized as shown in the following formula: in, Represents the video data of the i-th frame after normalization operation, is the video data of the i-th frame after wavelet denoising and time series smoothing, μ and σ represent the mean and standard deviation of all frame data respectively.

4. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 1, wherein: The frame-level skeleton key point extraction module first uses the OpenPose model to process each frame of the preprocessed video data V. For the i-th frame video, the i-th frame video is input into the OpenPose model to obtain the skeleton key point set in Represents the two-dimensional coordinates of the kth skeleton key point in the i-th frame, K is the total number of skeleton key points, and the skeleton key point sequence set is obtained after all videos are input into the OpenPose model Where N represents the total number of frames in the video; secondly, in order to ensure consistency across frames, the nearest neighbor matching algorithm is used to match the skeleton key point sequence set P V Match the continuous frames in the image to generate a skeleton action sequence with a temporal relationship, and obtain the skeleton action sequence The skeleton information extracted from each frame is used as a time step to record the motion trajectory of each joint of the target human body, providing time-series skeleton data for subsequent topological map construction and feature extraction.

5. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 4, wherein: The human joint motion coordinate graph construction module converts the skeleton motion sequence into a skeleton key point topology graph g = (A, V, θ), where V represents the joints as nodes of the graph, θ represents the connection relationship between joints, and A is the edge weight matrix; the node set of each topology graph represents all key points, while the edge set represents the connection relationship between each joint, ensuring that the spatial structure between joints is preserved.

6. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 1, wherein: The spatiotemporal graph convolutional network coding module is used to extract spatiotemporal features from the skeleton key point topology graph g, which includes two temporal convolution blocks and one spatial convolution block. The temporal convolution block first extracts the spatial features of the skeleton key point topology graph g by graph convolution operation, and the calculation formula is as follows: in, and Represent the temporal convolution and connection operations, Γ μ is the temporal convolution kernel, W k is a learnable matrix, σ is the sigmoid activation function, X j is the j-th frame video data, h S is the spatiotemporal feature of the current video frame; A k is the kth submatrix in A, D is a diagonal matrix, D k is the kth submatrix in D, and I is the unit matrix; then the spatiotemporal feature h and the skeleton key point topology map g are input into the GLU activation function layer to obtain the feature GLU(h). The GLU activation function layer controls the information flow by performing element-by-element weighted summation and activation on the input spatiotemporal feature h and the skeleton key point topology map g, so that the network can capture more complex feature representations; GLU(h) is input into the spatial domain convolution to obtain the feature Spatial convolution helps extract the spatial dependencies between GLU(h) nodes, providing richer spatial features for subsequent spatiotemporal modeling.

7. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 1, wherein: The multi-scale time convolution module includes a multi-scale time convolution layer and a feature splicing layer. First, the time feature is input into the multi-scale time convolution layer, and the time feature is convolved at three time scales of 5 frames, 10 frames, and 20 frames to obtain the feature h of the three time scales of 5, 10, and 20. hop5 、h hop10 and h hop20 , extracting local and global dependency information related to the current time step, effectively capturing the short-term and long-term motion change patterns; then, the output h of the multi-scale temporal convolution layer is converted to hop5 、h hop10 and h hop20 Input the feature concatenation layer and obtain the concatenated feature representation as follows: h TCN =[h hop5 ,h hop10 ,h hop20 ] Multi-level temporal dependencies are captured at different time scales, and these multi-scale information are integrated to obtain a richer and more comprehensive feature representation.

8. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 1, wherein: The action description text encoding module uses the OpenAI open source pre-trained CLIP model to realize action description text feature extraction and feature alignment; the action description text data T and the h output by the multi-scale time convolution module are combined. TCN At the same time, the CLIP model is input to obtain the high-dimensional feature vector l of the action description text, and the dimension of l is the same as h TCN The same; l contains the specific action vocabulary semantic feature information and the corresponding joint semantic feature information when performing neurorehabilitation actions, including action elements, time information and spatial relationships in the action text description.

9. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 1, wherein: The multimodal neural rehabilitation action detection network includes a convolutional long short-term memory network pooling module ConvLSTM, a multimodal fusion module and an action detection output module; First, the convolutional long short-term memory network pooling module is responsible for modeling the temporal dependency of temporal video features, extracting long-term and short-term temporal features from motion data, enhancing the model's understanding of temporal dynamics and achieving the dimensionality reduction effect of the pooling layer; then, a multimodal fusion module is used to combine the spatiotemporal features of neurorehabilitation exercise videos with the semantic information of text descriptions through cross-attention to extract inter-modal shared features that are conducive to accurate and useful prediction of rehabilitation exercises; finally, the motion detection output module is used to evaluate the motion quality based on the fused features and output the result score.

10. The method for detecting neurorehabilitation movements based on a domain generalization neural network according to claim 9, wherein: The input of the convolutional long short-term memory network pooling module is the splicing feature h output by the multi-scale time convolution module TCN =[h hop5 ,h hop10 ,h hop20 ]; contains the patient's motion trajectory characteristics at different time spans; First, the feature vectors h of different strides are hop5 ,h hop10 ,h hop20 After splicing, feature Z is obtained; then, Z is used for time series modeling; The gating mechanism is used to dynamically process the information in the time dimension and extract complete spatiotemporal features under inputs of different lengths; The ConvLSTM is set to a single layer with 256 hidden units. For the input sequence Z, the calculation formula is as follows: M=ConvLSTM(Z) Where M is the feature output by the ConvLSTM unit; after the feature sequence M output by ConvLSTM, the connection matrix is ​​converted into a learnable attention feature map g' through the following formula: g'=σ(φ(A⊙M)ZW) Where ⊙ represents the Hadamard product, φ represents the normalization function, W is the weight vector obtained by 1D convolution, A is the edge weight matrix of the skeleton key point topology map g; finally, the attention feature map g' is spliced ​​and output by temporal convolution to obtain the video feature h f .

Images