Method for evaluating limb nerve rehabilitation status based on multi-modal sensor data fusion

Abstract

The present invention provides a method for evaluating the limb nerve rehabilitation status based on multi-modal sensor data fusion, which can obtain the motion inertial data, electromyogram signal data of a patient during rehabilitation movements in real time, and the text data of the corresponding limb nerve rehabilitation action manual; input the data into the designed and trained multi-modal data feature extraction network to obtain motion inertial data features, electromyogram signal data features and text semantic features; input the motion inertial data features, electromyogram signal data features and text semantic features into the designed and trained multi-modal data feature fusion network to obtain multi-modal fusion features; input the multi-modal fusion features into the trained prediction output network to obtain the prediction value of the patient's limb nerve rehabilitation status. The present invention can more accurately evaluate the limb nerve rehabilitation status of patients through multi-modal data fusion and time-dependent feature extraction, providing a more reliable diagnosis basis for doctors. At the same time, it reduces the labor cost and improves the evaluation efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent detection for neurorehabilitation of multimodal data fusion, and particularly relates to a method for evaluating the limb nerve rehabilitation state based on multimodal sensor data fusion. Background Technique

[0002] In recent years, with the rapid progress of artificial intelligence and Internet of Things technologies, methods for evaluating the limb nerve rehabilitation state based on artificial intelligence technologies have been proposed and applied in clinical diagnosis to assist doctors in judging the limb nerve rehabilitation state of patients, solving the labor cost and improving the diagnosis efficiency.

[0003] For example, the Chinese invention patent with the application number CN202310635568.X proposes a control method and system for the interaction behavior of a rehabilitation robot. The specific steps include: first, acquiring the electromyogram signals of the monitored patient in a predetermined time period, the respiration frequency values and heart rate values at multiple predetermined time points within the predetermined time period, and the text description of the rehabilitation process of the monitored patient; then, fully expressing the correlation feature distribution information between the temporal collaborative correlation features of the electromyogram signals, respiration frequency values and heart rate values of the patient and the semantic understanding features of the text description of the rehabilitation process of the patient through artificial intelligence and deep learning technologies, so as to accurately detect and evaluate the rehabilitation state of the patient.

[0004] The Chinese invention patent with the application number CN202210436298.5 proposes a method for modeling the motor imagery electroencephalogram of unilateral limb patients based on neuron optimization, which includes the following steps: Step 1, professionals help the patient wear and use an electroencephalogram acquisition device; Step 2, the patient performs motor imagery according to the prompt, and the electroencephalogram acquisition device transmits the motor imagery electroencephalogram signals to a computer for personalized training and modeling through wired or wireless transmission; Step 3, after the computer system receives the electroencephalogram data, it first preprocesses the data, including filtering, baseline removal, and data slicing and integration; Step 4, the processed data is sent to a deep learning model for feature extraction and output of classification results.

[0005] However, the above solutions and existing technical attempts have the following defects:

[0006] 1. The prior art fails to fully consider the effective fusion of different modal sensor data, only making simple data feature matching, unable to fully fuse the data information of multiple modalities, with low matching accuracy between different modal data, resulting in poor accuracy of limb nerve rehabilitation state evaluation;

[0007] 2. The prior art fails to fully consider the time dependence of limb data obtained by sensors. The limb rehabilitation actions performed by patients are continuous in time. Therefore, fully capturing the inherent time dependence of limb data can improve the accuracy of limb nerve rehabilitation state assessment. Summary of the Invention

[0008] To address the above problems, the present invention introduces multi-modal data fusion and time-dependent feature extraction to more accurately evaluate the limb nerve rehabilitation state of patients and provide more reliable diagnostic basis for doctors.

[0009] The present invention provides a method for assessing the limb nerve rehabilitation state based on multi-modal sensor data fusion, including the following steps:

[0010] S1. Real-time obtain the motion inertial data, electromyogram signal data of the patient during the rehabilitation action, and the text data of the corresponding limb nerve rehabilitation action manual.

[0011] S2. Input the data in S1 into the trained multi-modal data feature extraction network, which includes a time-dependent feature extraction module and a text semantic feature extraction module. Among them, the motion inertial data and electromyogram signal data are input into the time-dependent feature extraction module to obtain motion inertial data features and electromyogram signal data features, and the text data of the limb nerve rehabilitation action manual is input into the text semantic feature extraction module to obtain text semantic features.

[0012] S3. Input the motion inertial data features, electromyogram signal data features, and text semantic features into the trained multi-modal data feature fusion network to obtain multi-modal fusion features.

[0013] S4. Input the multi-modal fusion features into the trained prediction output network to obtain the predicted value of the limb nerve rehabilitation state of the patient, and the doctor makes subsequent diagnoses for the patient based on the predicted value of the limb nerve rehabilitation state.

[0014] Further, the motion inertial data specifically is:

[0015] When the patient performs the rehabilitation action according to the text description in the limb rehabilitation manual, record the X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data generated by the inertial sensor when the patient performs the th limb rehabilitation action.

[0016] Further, the time-dependent feature extraction module includes a first recurrent neural network layer, a second recurrent neural network layer, a first convolutional layer, a second convolutional layer, a first multi-head attention mechanism layer, a second multi-head attention mechanism layer, a first LSTM layer, a second LSTM layer, a third convolutional layer, a fourth convolutional layer, and a feature fusion layer;

[0017] Motion inertia data features The specific acquisition process is as follows:

[0018] S1. Flip the motion inertia data of the patient to obtain a reverse motion inertia data sequence ;

[0019] S2. Input and into the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract the spatial dependence relationships between the limb neurorehabilitation actions in each dimension data of the X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data respectively, and obtain forward spatial dependence features and reverse spatial dependence features ;

[0020] S3. Input the forward spatial dependence features and the reverse spatial dependence features into the first multi-head attention mechanism and the second multi-head attention mechanism respectively, use the multi-head attention mechanism to calculate the correlation weights between the 9 dimensions of the X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data, and perform weighted sum calculations on the features between the 9-dimensional data in the forward spatial dependence features and the reverse spatial dependence features to obtain forward fusion features and reverse fusion features ;

[0021] S4. Input the forward fusion features and the reverse fusion features into the first convolutional layer and the second convolutional layer respectively. The convolutional kernel scales of the first convolutional layer and the second convolutional layer are both , aiming to increase the forward fusion features and the reverse fusion features The number of channels is used to convert low-dimensional features into high-dimensional features to obtain positive high-dimensional features and reverse high-dimensional features ;

[0022] S5. Input the positive high-dimensional features and reverse high-dimensional features into the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the positive time-dependent relationship in the sequence; the second LSTM layer captures the reverse time-dependent relationship in the sequence. The specific calculation formula is as follows:

[0023] ;

[0024] ;

[0025] Among them, and are the weight matrices of the first LSTM layer and the second LSTM layer respectively; and are the biases of the first LSTM layer and the second LSTM layer respectively; and are the activation function sets of the first LSTM layer and the second LSTM layer respectively; and are the positive time long-term dependence features and the reverse time long-term dependence features respectively;

[0026] S6. Input the positive time long-term dependence features and reverse time long-term dependence features into the third convolutional layer and the fourth convolutional layer respectively. The convolutional kernel scales of the third convolutional layer and the fourth convolutional layer are both . Convert the high-dimensional features into low-dimensional features to obtain positive low-dimensional features and reverse low-dimensional features ;

[0027] S7. Input the positive low-dimensional features and reverse low-dimensional features into the feature fusion layer to perform weighted fusion of the positive features and the reverse features to obtain the motion inertia data features .

[0028] Furthermore, the time-dependent feature extraction module includes a first recurrent neural network layer, a second recurrent neural network layer, a first convolutional layer, a second convolutional layer, a first multi-head attention mechanism layer, a second multi-head attention mechanism layer, a first LSTM layer, a second LSTM layer, a third convolutional layer, a fourth convolutional layer, and a feature fusion layer;

[0029] The electromyogram signal data features The specific acquisition process is as follows:

[0030] S1. Reverse the sequence of the myoelectric signal data of the patient to obtain the reverse myoelectric signal data sequence ;

[0031] S2. Input and into the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract the spatial dependence relationships between the limb nerve rehabilitation actions in and respectively, and obtain the forward spatial dependence feature and the reverse spatial dependence feature ;

[0032] S3. Input the forward spatial dependence feature and the reverse spatial dependence feature into the first multi-head attention mechanism and the second multi-head attention mechanism respectively, and use the multi-head attention mechanism to calculate the correlation weights between the limb nerve rehabilitation actions in and respectively, and perform weighted sum calculation on the features between the -dimensional limb nerve rehabilitation actions in the forward spatial dependence feature and the reverse spatial dependence feature to obtain the forward fusion feature and the reverse fusion feature ;

[0033] S4. Input the forward fusion feature and the reverse fusion feature into the first convolutional layer and the second convolutional layer respectively. The convolutional kernel scales of the first convolutional layer and the second convolutional layer are both , convert the low-dimensional features into high-dimensional features, and obtain the forward high-dimensional feature and the reverse high-dimensional feature ;

[0034] S5. Input the forward high-dimensional feature and the reverse high-dimensional feature into the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the forward time dependence relationship in the sequence; the second LSTM layer captures the reverse time dependence relationship in the sequence. The specific calculation formula is as follows:

[0035] ; The specific calculation formula is as follows:

[0036] ;

[0037] ;

[0038] Among them, and are the weight matrices of the first LSTM layer and the second LSTM layer respectively; and are the biases of the first LSTM layer and the second LSTM layer respectively; and are the activation function sets of the first LSTM layer and the second LSTM layer respectively; and are the forward time long-term dependence feature and the backward time long-term dependence feature respectively;

[0039] S6. Input the forward time long-term dependence feature and the backward time long-term dependence feature into the third convolutional layer and the fourth convolutional layer respectively. The convolutional kernel scales of the third convolutional layer and the fourth convolutional layer are both , convert the high-dimensional features into low-dimensional features, and obtain the forward low-dimensional feature and the backward low-dimensional feature ;

[0040] S7. Input the forward low-dimensional feature and the backward low-dimensional feature into the feature fusion layer to perform weighted fusion of the forward feature and the backward feature to obtain the myoelectric signal data feature .

[0041] Furthermore, the semantic text feature extraction module includes a text convolutional layer, a first gated unit layer, a second gated unit layer, a first max pooling layer, a second max pooling layer and a feature splicing layer; the specific processing process is as follows:

[0042] S1. Input the limb rehabilitation movement text description data into the text convolutional layer to obtain local text features. The specific calculation formula is as follows:

[0043] ;

[0044] Among them, is the local feature obtained through the text convolutional layer; is the text convolutional kernel, is the width of the text convolutional kernel, is the number of input or output channels of the text convolutional kernel; is sequence at position of dimensional word vector window; is the bias term; is the activation function of the text convolutional layer;

[0045] S2, input into the first gating unit layer and the second gating unit layer in sequence to obtain global semantic features , capture the temporal dependence relationship between sequence data, and store continuous limb rehabilitation action text state information in the hidden layer;

[0046] S3, input and into the first max pooling layer and the second max pooling layer respectively to obtain low-dimensional local semantic features and low-dimensional global semantic features , and the max pooling layer downsamples and , retains the most significant local features, and reduces the number of feature parameters at the same time;

[0047] S4, input the low-dimensional local semantic features and the low-dimensional global semantic features into the feature concatenation to obtain text semantic features representing the local semantic and global semantic dependence relationships , and the specific calculation formula is as follows:

[0048] = [CLS]+ +[SEP] + +[SEP];

[0049] where [CLS] and [SEP] are the sequence start and separator tokens respectively.

[0050] Furthermore, the multimodal data feature fusion network includes a first feature segmentation layer, a second feature segmentation layer, a third feature segmentation layer, a rehabilitation action feature attention layer, and a dropout layer; the specific processing process is as follows:

[0051] S1, input the motion inertia data features , the electromyogram signal data features and the text semantic features into the first feature segmentation layer, the second feature segmentation layer, and the third feature segmentation layer respectively, segment the features according to the limb nerve rehabilitation action categories, and obtain the motion inertia data feature segmentation set , the electromyogram signal data feature segmentation set and the text semantic feature segmentation set ;

[0052] S2, the motion inertia data feature segmentation set , the electromyogram signal data feature segmentation set and the text semantic feature segmentation set The input rehabilitation action feature attention layer calculates attention at different rehabilitation action levels, captures the complex interaction relationships between them, and performs weighted feature fusion. The specific calculation formula is as follows:

[0053] ;

[0054] ;

[0055] Among them, is the weight of the th limb rehabilitation action, is the multi-modal feature matrix , is the learnable weight matrix, is the multi-modal fusion feature, represents kinds of limb rehabilitation actions.

[0056] Furthermore, the prediction output network consists of three fully connected layers and a Softmax activation function layer connected in sequence. The multi-modal fusion feature output by the multi-modal data feature fusion network is input into the prediction output network to obtain the evaluation result of the patient's limb nerve rehabilitation status. The specific calculation formula is as follows:

[0057] ;

[0058] ;

[0059] ;

[0060] ;

[0061] Among them, 、 、 and are the weight matrices of the first fully connected layer, the second fully connected layer, the third fully connected layer, and the Softmax activation function layer respectively, 、 、 and are the biases of the first fully connected layer, the second fully connected layer, the third fully connected layer, and the Softmax activation function layer respectively.

[0062] Further, the obtained training set and test set are used to conduct overall training and testing on the multi-modal data feature extraction network, multi-modal data feature fusion network, and prediction output network. The loss function adopts the cross-entropy loss function, and the specific calculation formula is as follows:

[0063] ;

[0064] where, is the real limb nerve rehabilitation status of the patient, is the limb nerve rehabilitation status of the patient obtained by the prediction output network, is the scale of the training set.

[0065] Compared with the prior art, the present invention has the following beneficial effects:

[0066] Through multi-modal data fusion and time-dependent feature extraction, the present invention can more accurately evaluate the limb nerve rehabilitation state of patients, providing a more reliable diagnosis basis for doctors. At the same time, it reduces the labor cost. The traditional evaluation of limb nerve rehabilitation status requires a large amount of human resources and time cost, while the method proposed by the present invention realizes automatic evaluation by using artificial intelligence technology, significantly reducing the labor cost and improving the evaluation efficiency.

[0067] 1. Multi-modal data fusion: The present invention proposes a method for evaluating the limb nerve rehabilitation state based on multi-modal sensor data fusion. Different from the prior art that only simply matches the data features of different modalities, the present invention realizes the effective fusion of three modalities of data, namely motion inertial data, electromyogram signal data, and text data, by constructing a new data processing flow and network structure. This fusion not only improves the utilization rate of data but also significantly improves the accuracy of evaluating the limb nerve rehabilitation state.

[0068] 2. Time-dependent feature extraction: Considering that the limb rehabilitation actions performed by patients are continuous in time, the present invention introduces a time-dependent feature extraction module in the data feature extraction stage. This module uses technologies such as recurrent neural network (RNN), long short-term memory network (LSTM), and multi-head attention mechanism to effectively capture the time dependence in motion inertial data and electromyogram signal data, thereby further improving the accuracy of evaluation.

[0069] 3. Text semantic feature extraction: For the text descriptions in the limb rehabilitation manual, the present invention designs a text semantic feature extraction module. This module fully excavates the local semantic and global semantic dependence relationships of text data through structures such as text convolutional layers, gated unit layers, and max pooling layers, realizing high-quality feature learning. This processing method enables text data to play a greater role in the evaluation of limb nerve rehabilitation state.

[0070] 4. Multimodal Data Feature Fusion Network: In the feature fusion stage, the present invention constructs a multimodal data feature fusion network. Through structures such as a feature segmentation layer, a rehabilitation action feature attention layer, and a dropout layer, this network realizes the correlation calculation among the motion inertial data features, electromyogram signal data features, and text semantic features, and performs sufficient feature fusion at the rehabilitation action level. This fusion method not only improves the accuracy of evaluation but also enhances the interpretability of the model.

[0071] Meanwhile, the method proposed by the present invention is not only applicable to the evaluation of limb nerve rehabilitation status but also can be extended to other fields that require multimodal data fusion, such as motion analysis, health monitoring, etc. This expandability makes the method proposed by the present invention have a broader application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is the overall logic flowchart of the present invention.

[0073] Figure 2 is the structural schematic diagram of the time-dependent feature extraction module of the present invention.

[0074] Figure 3 is the structural schematic diagram of the semantic text feature extraction module of the present invention.

[0075] Figure 4 is the structural schematic diagram of the multimodal data feature fusion network of the present invention.

[0076] Figure 5 is the comparison diagram of the experimental results of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0077] The overall technical route of the present invention is as Figure 1 shown: First, construct a data set, perform data annotation and divide it into a training set and a test set; then construct a multimodal data feature extraction network, a multimodal data feature fusion network, and a prediction output network, and deploy them in a rehabilitation monitoring system for real-time detection and evaluation after training the three networks through the data set and training strategy: collect the motion inertial data, electromyogram signal data, and limb nerve rehabilitation action manual text data of the patient; input the collected motion inertial data and electromyogram signal data into the trained time-dependent feature extraction module to obtain motion inertial data features and electromyogram signal data features; input the limb nerve rehabilitation action manual text data into the trained text semantic feature extraction module to obtain text semantic features; input the motion inertial data features, electromyogram signal data features, and text semantic features into the trained multimodal data feature fusion network to obtain multimodal fusion features; input the multimodal fusion features into the trained prediction output network to obtain the predicted value of the patient's limb nerve rehabilitation status, and the doctor makes a subsequent diagnosis of the patient based on the predicted value of the limb nerve rehabilitation status and his own experience.

[0078] The invention will be further described below in conjunction with specific embodiments.

[0079] I. Dataset production

[0080] In the present invention, a high-quality dataset is the key foundation to ensure the good performance of the model. To this end, the present application proposal adopts a rigorous dataset production process, aiming to provide sufficient high-quality training data for the model and improve the recommendation accuracy.

[0081] 1. Data collection:

[0082] The present application proposal needs to collect three modalities of data to comprehensively evaluate the limb nerve rehabilitation status of patients. The first modality of data is motion inertial data. Triaxial acceleration time series data, triaxial angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data generated by the upper limb movement of the patient during rehabilitation actions are collected using inertial sensors. The second modality of data is electromyogram signal data. Electromyogram plays an important role in rehabilitation engineering. Muscle electricity not only carries information such as nerve state and muscle state, but also contains a large amount of rehabilitation-related information such as the disease recovery status and disease progression of the patient. The information contained in electromyogram data has great value in aspects such as rehabilitation effect evaluation, muscle fatigue judgment and prediction. The time series data of electromyogram signals in the upper limb muscles are collected using patch-type electromyogram sensors. The third modality of data is text data, specifically the text description of each limb rehabilitation action in the limb rehabilitation manual.

[0083] The specific steps for producing the motion inertial dataset are as follows: After the st patient wears an inertial sensor on the upper limb, the action is corrected according to the text description in the limb rehabilitation manual. After the patient standardly completes the rd limb rehabilitation action, record the X-axis acceleration time series data generated by the patient performing the th limb rehabilitation action output by the inertial sensor and the Y-axis acceleration time series data and the Z-axis acceleration time series data and the X-axis angular velocity time series data and the Y-axis angular velocity time series data and the Z-axis angular velocity time series data and the roll angle time series data and the pitch angle time series data and the yaw angle time series data ; After each of the patients has completed all the limb rehabilitation actions, a motion inertial dataset is obtained ;

[0084] The specific steps for making the EMG signal dataset are as follows: First, place the patch-type EMG sensor on the upper limb muscles of the th patient. After the patient completes the th limb rehabilitation movement in a standard manner, based on the EMG signal time series data generated by the patient during the th limb rehabilitation movement output by the patch-type EMG sensor ; After each of the kinds of limb rehabilitation movements of each patient in the patients are completed, an EMG signal dataset is obtained

[0085] The specific steps for making the text dataset are as follows: First, perform data cleaning on the text description of the th limb rehabilitation movement in the limb rehabilitation manual, removing non-text information, conjunctions, and auxiliary words, including: Regular expression replacement: Replace "!", "?", "~", "\", "@", "#", "$", and "%" in the text with spaces; Remove invalid data: Delete invalid spaces in the information; Text tokenization: In this application, NLPIR is used to tokenize the text description of limb rehabilitation movements. This is an efficient and accurate tokenization tool that can identify and divide the words in the text, improving the structured degree of the corpus. By performing tokenization through NLPIR, the key steps, important joint parts, and movement amplitudes in the text description of limb rehabilitation movements can be more accurately extracted. The text data structure after tokenization is clearer, which helps with subsequent feature extraction and semantic analysis; The text data structure after tokenization is clearer, which helps with subsequent feature extraction and semantic analysis; After all the text descriptions of the kinds of limb rehabilitation movements are cleaned, a text description dataset of limb rehabilitation movements is obtained ;

[0086] 2. Data annotation

[0087] Doctors divide the patients into three levels: excellent limb nerve rehabilitation, good limb nerve rehabilitation, and poor limb nerve rehabilitation according to the patient's limb nerve rehabilitation situation, and use one-hot encoding to encode the three rehabilitation levels. Among them, the excellent limb nerve rehabilitation is encoded as 0, the good limb nerve rehabilitation is encoded as 1, and the poor limb nerve rehabilitation is encoded as 2. Finally, a set of limb nerve rehabilitation status labels for the patients is obtained ;

[0088] 3. Training set and test set production

[0089] Combine the motion inertia dataset and the EMG signal dataset , Limb rehabilitation movement text description dataset And the corresponding set of limb nerve rehabilitation status labels They are respectively divided into two independent datasets, the training set and the test set, according to the ratio of 8:2. The training set is used for the iterative learning of the model's parameters, and the test set is used for the evaluation of the model's accuracy. Among them, the test set is randomly sampled and constructed by a dataset partitioning tool and is independent of the training set data.

[0090] II. Multimodal data feature extraction network

[0091] The multimodal data feature extraction network includes a time-dependent feature extraction module and a text semantic feature extraction module. Among them, the time consistency feature extraction module is used to extract the features of motion inertia data and electromyogram signal data and capture time dependence; the text semantic feature extraction module is used to extract the semantic features of limb rehabilitation movement text description data

[0092] 1. Time-dependent feature extraction module

[0093] The time-dependent feature extraction module is as Figure 2 shown, and includes a first recurrent neural network layer, a second recurrent neural network layer, a first convolutional layer, a second convolutional layer, a first multi-head attention mechanism layer, a second multi-head attention mechanism layer, a first LSTM layer, a second LSTM layer, a third convolutional layer, a fourth convolutional layer, and a feature fusion layer; the motion inertia data of the

[0094] th patient and the electromyogram signal data of the th patient are respectively input into the time-dependent feature extraction module to obtain the motion inertia data features and the electromyogram signal data features respectively. The specific calculation steps are as follows:

[0095] (1) Obtaining motion inertia data features:

[0096] In the first step, is sequence-flipped to obtain the reverse motion inertia data sequence ;

[0097] In the second step, and Input the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract the spatial dependence relationships between the limb neurorehabilitation actions in each dimension of the X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data respectively, and obtain the forward spatial dependence features and the reverse spatial dependence features ;

[0098] In the third step, input the forward spatial dependence features and the reverse spatial dependence features into the first multi-head attention mechanism and the second multi-head attention mechanism respectively. Use the multi-head attention mechanism to calculate the correlation weights between the 9 dimensions of the X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data, and perform weighted summation calculation on the features between the 9 dimensions of the forward spatial dependence features and the reverse spatial dependence features to obtain the forward fusion features and the reverse fusion features ;

[0099] In the fourth step, input the forward fusion features and the reverse fusion features into the first convolutional layer and the second convolutional layer respectively. The convolutional kernel scales of the first convolutional layer and the second convolutional layer are both , aiming to increase the number of channels of the forward fusion features and the reverse fusion features , convert the low-dimensional features into high-dimensional features, improve the expression ability of the features, and obtain the forward high-dimensional features and the reverse high-dimensional features ;

[0100] In the fifth step, input the forward high-dimensional features and the reverse high-dimensional features into the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the forward time dependence relationship in the sequence; the second LSTM layer captures the reverse time dependence relationship in the sequence. This bidirectional processing can retain the key information in the sequence and transmit it to the subsequent layers for more complex feature extraction and prediction. The specific calculation formula is as follows:

[0101] ;

[0102] ;

[0103] Among them, and are the weight matrices of the first LSTM layer and the second LSTM layer respectively; and are the biases of the first LSTM layer and the second LSTM layer respectively; and are the activation function sets of the first LSTM layer and the second LSTM layer respectively; and are the forward time long-term dependence features and the backward time long-term dependence features respectively;

[0104] Step 6: Input the forward time long-term dependence feature and the backward time long-term dependence feature into the third convolutional layer and the fourth convolutional layer respectively. The convolutional kernel scales of the third convolutional layer and the fourth convolutional layer are both , aiming to reduce the number of channels of the forward time long-term dependence feature and the backward time long-term dependence feature , convert the high-dimensional features into low-dimensional features, reduce the computational burden of the model and the risk of overfitting, and obtain the forward low-dimensional feature and the backward low-dimensional feature ;

[0105] Step 7: Input the forward low-dimensional feature and the backward low-dimensional feature into the feature fusion layer to perform weighted fusion of the forward feature and the backward feature to obtain the motion inertia data feature , and the specific calculation formula is as follows:

[0106] ;

[0107] (2) Acquisition of electromyogram signal data features:

[0108] Step 1: Perform sequence flipping on to obtain the reverse electromyogram signal data sequence ;

[0109] Step 2: Input and into the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract the and spatial dependence relationships between the limb neurorehabilitation actions respectively, and obtain the forward spatial dependence feature and the backward spatial dependence feature ;

[0110] In the third step, the forward spatial dependence features and the reverse spatial dependence features are respectively input into the first multi-head attention mechanism and the second multi-head attention mechanism, and the multi-head attention mechanism is used to calculate respectively and the correlation weights between the limb neurorehabilitation actions in and the reverse spatial dependence features in dimensional limb neurorehabilitation actions, and the features between the and the reverse fusion features are weighted and summed to obtain the forward fusion feature

[0111] In the fourth step, the forward fusion feature and the reverse fusion feature are respectively input into the first convolutional layer and the second convolutional layer. The convolutional kernel scales of the first convolutional layer and the second convolutional layer are both , aiming to increase the number of channels of the forward fusion feature and the reverse fusion feature , convert the low-dimensional features into high-dimensional features, enhance the expression ability of the features, and obtain the forward high-dimensional feature and the reverse high-dimensional feature ;

[0112] In the fifth step, the forward high-dimensional feature and the reverse high-dimensional feature are respectively input into the first LSTM layer and the second LSTM layer. The first LSTM layer captures the forward time dependence relationship in the sequence; the second LSTM layer captures the reverse time dependence relationship in the sequence. This bidirectional processing can retain the key information in the sequence and transmit it to the subsequent layers for more complex feature extraction and prediction. The specific calculation formula is as follows:

[0113] ;

[0114] ;

[0115] where and are the weight matrices of the first LSTM layer and the second LSTM layer respectively; and are the biases of the first LSTM layer and the second LSTM layer respectively; and are the activation function sets of the first LSTM layer and the second LSTM layer respectively; and They are the forward-time long-term dependence feature and the backward-time long-term dependence feature respectively;

[0116] Step 6: Input the forward-time long-term dependence feature and the backward-time long-term dependence feature into the third convolutional layer and the fourth convolutional layer respectively. The convolutional kernel scales of the third convolutional layer and the fourth convolutional layer are both . The purpose is to reduce the number of channels of the forward-time long-term dependence feature and the backward-time long-term dependence feature , convert the high-dimensional features into low-dimensional features, reduce the computational burden of the model and the risk of overfitting, and obtain the forward low-dimensional feature and the backward low-dimensional feature ;

[0117] Step 7: Input the forward low-dimensional feature and the backward low-dimensional feature into the feature fusion layer to perform weighted fusion of the forward feature and the backward feature to obtain the myoelectric signal data feature . The specific calculation formula is as follows:

[0118] ;

[0119] 2. Text semantic feature extraction module

[0120] The text semantic feature extraction module is used to extract the semantic features of the limb rehabilitation action text description data. As Figure 3 shown, it includes a text convolutional layer, a first gated unit layer, a second gated unit layer, a first max pooling layer, a second max pooling layer, and a feature splicing layer to fully mine the local semantic and global semantic dependence relationships of the limb rehabilitation action text description data and achieve high-quality feature learning;

[0121] Input the limb rehabilitation action text description data set into the text semantic feature extraction module to obtain the movement text semantic feature . The specific calculation steps are as follows:

[0122] Step 1: Input into the text convolutional layer to obtain local text features. The text convolutional neural network has strong feature learning ability in natural language processing tasks, especially good at automatically extracting local hierarchical features from raw data and capturing the underlying patterns that constitute semantics. This layer performs convolutional operations on the input limb rehabilitation action text description data set by applying a sliding convolutional kernel to generate a new sequence of feature maps. The specific calculation formula is as follows:

[0123] ;

[0124] Among them, is the local feature obtained through the text convolutional layer; is the text convolution kernel, is the width of the text convolution kernel, is the number of input / output channels of the text convolution kernel; is sequence at position of dimensional word vector window; is the bias term; is the activation function of the text convolutional layer. This convolutional structure can effectively capture the local correlation of the input data, thereby extracting rich feature representations;

[0125] Step 2: Input into the first gated unit layer and the second gated unit layer in sequence to obtain the global semantic feature . The gated unit can effectively capture the temporal dependence relationship between sequence data and store the continuous limb rehabilitation action text state information in the hidden layer. Each operator in the model is associated with corresponding weights;

[0126] Step 3: Input and into the first max-pooling layer and the second max-pooling layer respectively to obtain the low-dimensional local semantic feature and the low-dimensional global semantic feature . The max-pooling layer downsamples and , retains the local most significant features, and at the same time reduces the number of feature parameters. The max-pooling layer retains the most important information when extracting features, enabling the model to focus more on the most significant part of the features;

[0127] Step 4: Input the low-dimensional local semantic feature and the low-dimensional global semantic feature into the feature concatenation to obtain the text semantic feature that represents the local semantic and global semantic dependence relationship. The specific calculation formula is as follows:

[0128] = [CLS]+ +[SEP] + +[SEP];

[0129] Among them, [CLS] and [SEP] are the sequence start and separator markers respectively.

[0130] III. Multimodal Data Feature Fusion Network

[0131] The multimodal data feature fusion network is asFigure 4 As shown, it includes a first feature segmentation layer, a second feature segmentation layer, a third feature segmentation layer, a rehabilitation action feature attention layer, and a dropout layer, which are used to fully calculate the motion inertia data features , the electromyogram signal data features and the text semantic features The relevance among the three is realized to achieve sufficient feature fusion at the rehabilitation action level. The specific steps are as follows:

[0132] First step, input the motion inertia data features , the electromyogram signal data features and the text semantic features into the first feature segmentation layer, the second feature segmentation layer, and the third feature segmentation layer respectively. The features are segmented according to the limb nerve rehabilitation action categories to obtain the motion inertia data feature segmentation set , the electromyogram signal data feature segmentation set and the text semantic feature segmentation set , where , and respectively represent the motion inertia data features, electromyogram signal data features, and text data features corresponding to the th limb rehabilitation action;

[0133] Second step, input the motion inertia data feature segmentation set , the electromyogram signal data feature segmentation set and the text semantic feature segmentation set into the rehabilitation action feature attention layer to calculate the attention at different rehabilitation action levels, capture the complex interaction relationships among them, and perform weighted feature fusion. This method can significantly improve the accuracy and interpretability of multimodal fusion;

[0134] The specific calculation formula is as follows:

[0135] ;

[0136] ;

[0137] Among them, is the weight of the th limb rehabilitation action, is the multimodal feature matrix , is the learnable weight matrix, is the multimodal fusion feature.

[0138] IV. Prediction Output Network

[0139] The prediction output network consists of three fully connected layers connected in sequence and a Softmax activation function layer, and inputting into the prediction output network to obtain the evaluation result of the patient's limb nerve rehabilitation status , and the specific calculation formula is as follows:

[0140] ;

[0141] ;

[0142] ;

[0143] ;

[0144] where 、 、 and are the weight matrices of the first fully connected layer, the second fully connected layer, the third fully connected layer and the Softmax activation function layer respectively, 、 、 and are the biases of the first fully connected layer, the second fully connected layer, the third fully connected layer and the Softmax activation function layer respectively.

[0145] V. Model Training

[0146] Using the obtained training set and test set to conduct overall training and testing on the time-dependent feature extraction module, the text semantic feature extraction module, the multi-modal data feature fusion network and the prediction output network, and the loss function adopts the cross-entropy loss function, and the specific calculation formula is as follows:

[0147] ;

[0148] where is the true limb nerve rehabilitation status of the patient, is the limb nerve rehabilitation status of the patient obtained by the prediction output network, is the scale of the training set.

[0149] VI. Model Deployment and Experiment

[0150] Step 1, collect the patient's motion inertial data, electromyogram signal data and limb nerve rehabilitation action manual text data according to the acquisition method of the data training dataset;

[0151] Step 2: Input the motion inertia data and EMG signal data collected in Step 1 into the trained time-dependent feature extraction module to obtain motion inertia data features and EMG signal data features; input the text data of the limb nerve rehabilitation action manual into the trained text semantic feature extraction module to obtain text semantic features;

[0152] Step 3: Input the motion inertia data features, EMG signal data features, and text semantic features into the trained multi-modal data feature fusion network to obtain multi-modal fusion features;

[0153] Step 4: Input the multi-modal fusion features into the trained prediction output network to obtain the predicted value of the patient's limb nerve rehabilitation status. The doctor makes subsequent diagnoses for the patient based on the predicted value of the limb nerve rehabilitation status and their own experience.

[0154] To verify the effectiveness of the method of the present invention, a test set consisting of 20 patients is used to verify the method proposed by the present invention. As Figure 5 shown, the sample numbers 1 - 20 represent 20 patients. The blue circles represent the true values of the limb nerve rehabilitation status of the patients, and the orange circles represent the predicted values of the limb nerve rehabilitation status of the patients. It can be seen from the figure that the number of samples predicted incorrectly by the method of the present invention is 1, indicating that the method of the present invention has a high prediction accuracy.

[0155] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0156] Although the specific implementation manners of the present invention are described above, it is not a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications or deformations that can be made without creative labor on the basis of the technical solution of the present invention are still within the protection scope of the present invention.

Claims

1. A method for determining the predicted value of the limb nerve rehabilitation state based on multi-modal sensor data fusion, characterized in that, It includes the following steps: S1. Obtain the motion inertia data, electromyogram signal data of the patient during the rehabilitation movement, and the corresponding text data of the limb nerve rehabilitation movement manual in real time. The text data of the limb nerve rehabilitation movement manual is a text description for guiding the patient to perform types of rehabilitation movements; S2, input the data in S1 into the trained multi-modal data feature extraction network, which includes a time-dependent feature extraction module and a text semantic feature extraction module; Among them, the motion inertia data and the electromyogram signal data are input into the time-dependent feature extraction module to obtain the motion inertia data features and the electromyogram signal data features, and the limb nerve rehabilitation action manual text data is input into the text semantic feature extraction module to obtain the text semantic features; The time-dependent feature extraction module adopts a dual-branch parallel spatio-temporal feature extraction and fusion architecture, including a dual-branch parallel spatio-temporal feature extraction channel and a feature fusion layer; the first spatio-temporal feature extraction channel includes a first recurrent neural network layer, a first multi-head attention mechanism layer, a first convolutional layer, a first LSTM layer, and a third convolutional layer connected in sequence; the second spatio-temporal feature extraction channel includes a second recurrent neural network layer, a second multi-head attention mechanism layer, a second convolutional layer, a second LSTM layer, and a fourth convolutional layer connected in sequence; the third convolutional layer of the first spatio-temporal feature extraction channel and the fourth convolutional layer of the second spatio-temporal feature extraction channel are respectively connected to the feature fusion layer; The semantic text feature extraction module includes a text convolutional layer, a first gated unit layer, a second gated unit layer, a first max pooling layer, a second max pooling layer, and a feature splicing layer; the text convolutional layer is respectively connected to the first max pooling layer and the first gated unit layer; the first gated unit layer, the second gated unit layer, and the second max pooling layer are connected in sequence; the first max pooling layer and the second max pooling layer are respectively connected to the feature splicing layer; S3, input the motion inertia data features, the electromyogram signal data features, and the text semantic features into the trained multi-modal data feature fusion network to obtain multi-modal fusion features; The multi-modal data feature fusion network includes a first feature segmentation layer, a second feature segmentation layer, a third feature segmentation layer, a rehabilitation action feature attention layer, and a dropout layer; the specific processing process is as follows: Input the motion inertia data features , electromyogram signal data features and text semantic features into the first feature segmentation layer, the second feature segmentation layer, and the third feature segmentation layer respectively, and segment the features according to the limb nerve rehabilitation action categories to obtain the motion inertia data feature segmentation set , the electromyogram signal data feature segmentation set and the text semantic feature segmentation set ; Split set of motion inertia data features and split set of electromyogram signal data features and split set of text semantic features are input into the rehabilitation action feature attention layer to calculate attention at different rehabilitation action levels, capture the complex interaction relationships between them, and perform weighted feature fusion. The specific calculation formula is as follows: Among them, is the weight of the th limb rehabilitation movement, is the multimodal feature matrix , is the learnable weight matrix, is the multimodal fusion feature, represents types of limb rehabilitation movements; S4, input the multi-modal fusion features into the trained prediction output network to obtain the predicted value of the limb nerve rehabilitation state of the patient.

2. The method for determining the predicted value of the limb nerve rehabilitation state based on multi-modal sensor data fusion according to claim 1, characterized in that: The specific motion inertia data is: When the patient performs rehabilitation movements according to the text description in the limb rehabilitation manual, record the X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data generated by the patient performing the th limb rehabilitation movement output by the inertial sensor.

3. The method for determining the predicted value of the limb nerve rehabilitation state based on multi-modal sensor data fusion according to claim 1, characterized in that: Motion inertia data characteristics The specific acquisition process is as follows: S1. Reverse the order of the patient's motion inertia data to obtain a reverse motion inertia data sequence ; S2, input and into the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract the spatial dependence relationships between the limb neurorehabilitation actions in each dimension data of the X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data and yaw angle time series data respectively, and obtain the forward spatial dependence features and reverse spatial dependence features ; S3, input the forward spatial dependence feature and the reverse spatial dependence feature into the first multi-head attention mechanism and the second multi-head attention mechanism respectively. Use the multi-head attention mechanism to calculate the correlation weights between the nine dimensions of the X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data, and perform weighted summation calculations on the features between the nine-dimensional data in the forward spatial dependence feature and the reverse spatial dependence feature to obtain the forward fusion feature and the reverse fusion feature ; S4, input the forward fusion feature and the reverse fusion feature into the first convolutional layer and the second convolutional layer respectively. The convolutional kernel scales of the first convolutional layer and the second convolutional layer are both , aiming to increase the number of channels of the forward fusion feature and the reverse fusion feature , convert the low-dimensional features into high-dimensional features, and obtain the forward high-dimensional feature and the reverse high-dimensional feature ; S5, input the forward high-dimensional features and the reverse high-dimensional features into the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the forward temporal dependencies in the sequence; The second LSTM layer captures the reverse time-dependent relationship in the sequence; the specific calculation formula is as follows: Among them, and are the weight matrices of the first LSTM layer and the second LSTM layer respectively; and are the biases of the first LSTM layer and the second LSTM layer respectively; and are the activation function sets of the first LSTM layer and the second LSTM layer respectively; and are the forward time long-term dependence feature and the backward time long-term dependence feature respectively; S6. Input the forward long-term temporal dependence feature and the backward long-term temporal dependence feature into the third convolutional layer and the fourth convolutional layer respectively. The convolutional kernel scales of the third convolutional layer and the fourth convolutional layer are both . Convert the high-dimensional features into low-dimensional features to obtain the forward low-dimensional feature and the backward low-dimensional feature ; S7, input the forward low-dimensional feature and the reverse low-dimensional feature into the feature fusion layer to obtain the weighted fusion of the forward feature and the reverse feature, thereby obtaining the motion inertia data feature .

4. The method for determining the predicted value of the limb nerve rehabilitation state based on multi-modal sensor data fusion according to claim 1, characterized in that: The characteristics of the electromyogram signal data The specific process of acquisition is as follows: S1. Obtain the EMG signal data of the patient and perform sequence flipping to obtain the reverse EMG signal data sequence ; S2, input and into the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract the and spatial dependence relationships between the limb neurorehabilitation actions respectively, and obtain the forward spatial dependence feature and the reverse spatial dependence feature ; S3. Input the forward spatial dependence features and the reverse spatial dependence features into the first multi-head attention mechanism and the second multi-head attention mechanism respectively, and use the multi-head attention mechanism to calculate the and correlation weights between the limb neurorehabilitation actions in, and perform weighted summation calculation on the features between the dimensional limb neurorehabilitation actions in the forward spatial dependence features and the reverse spatial dependence features to obtain the forward fusion feature and the reverse fusion feature ; S4, input the forward fusion feature and the reverse fusion feature into the first convolutional layer and the second convolutional layer respectively. The convolutional kernel scales of the first convolutional layer and the second convolutional layer are both , convert the low-dimensional features into high-dimensional features, and obtain the forward high-dimensional feature and the reverse high-dimensional feature ; S5, input the forward high-dimensional feature and the reverse high-dimensional feature into the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the forward time-dependent relationship in the sequence; The second LSTM layer captures the reverse time-dependent relationship in the sequence; the specific calculation formula is as follows: Among them, and are the weight matrices of the first LSTM layer and the second LSTM layer respectively; and are the biases of the first LSTM layer and the second LSTM layer respectively; and are the activation function sets of the first LSTM layer and the second LSTM layer respectively; and are the forward-time long-term dependence features and the backward-time long-term dependence features respectively; S6. Input the forward long-term temporal dependence feature and the backward long-term temporal dependence feature into the third convolutional layer and the fourth convolutional layer respectively. The convolutional kernel scales of the third convolutional layer and the fourth convolutional layer are both , convert the high-dimensional features into low-dimensional features, and obtain the forward low-dimensional feature and the backward low-dimensional feature ; S7, input the forward low-dimensional feature and the reverse low-dimensional feature into the feature fusion layer to obtain the weighted fusion of the forward and reverse features and get the myoelectric signal data feature .

5. The method for determining the predicted value of the limb nerve rehabilitation state based on multi-modal sensor data fusion according to claim 1, wherein: The specific processing process of the semantic text feature extraction module is: S1, input the limb rehabilitation movement text description data into the text convolution layer to obtain local text features. The specific calculation formula is as follows: Among them, is the local feature obtained through the text convolutional layer; is the text convolution kernel, is the width of the text convolution kernel, is the number of input or output channels of the text convolution kernel; is the sequence at position of the d-dimensional word vector window; is the bias term; is the activation function of the text convolutional layer; S2, input into the first gated unit layer and the second gated unit layer in sequence to obtain global semantic features , capture the temporal dependency between sequence data, and store consecutive limb rehabilitation action text state information in the hidden layer; S3. Input and into the first max pooling layer and the second max pooling layer respectively to obtain low-dimensional local semantic features and low-dimensional global semantic features . The max pooling layer downsamples and , retains the most significant local features, and reduces the number of feature parameters at the same time;​​​​​​​​​​​​ S4, concatenate the low-dimensional local semantic features and the low-dimensional global semantic features to obtain text semantic features representing the local semantic and global semantic dependency relationships , and the specific calculation formula is as follows: = [CLS]+ +[SEP] + +[SEP] Among them, [CLS] and [SEP] are the sequence start and separator markers respectively.

6. The method for determining the predicted value of the limb nerve rehabilitation state based on multi-modal sensor data fusion according to claim 1, characterized in that: The prediction output network consists of three fully-connected layers connected in sequence and a Softmax activation function layer, and the multi-modal fusion features output by the multi-modal data feature fusion network are input into the prediction output network to obtain the evaluation result of the patient's limb nerve rehabilitation status , and the specific calculation formula is as follows: Among them, 、 、 and are the weight matrices of the first fully connected layer, the second fully connected layer, the third fully connected layer, and the Softmax activation function layer respectively, 、 、 and are the biases of the first fully connected layer, the second fully connected layer, the third fully connected layer, and the Softmax activation function layer respectively.

7. A method for determining the predicted value of the limb nerve rehabilitation state based on multi-modal sensor data fusion according to claim 1, characterized in that: The obtained training set and test set are used to perform overall training and testing on the multi-modal data feature extraction network, the multi-modal data feature fusion network, and the prediction output network. The loss function uses the cross-entropy loss function, and the specific calculation formula is as follows: Among them, is the real nerve rehabilitation status of the patient's limb, is the nerve rehabilitation status of the patient's limb obtained by the prediction output network, is the scale of the training set.

Images