Limb nerve rehabilitation state evaluation method based on multi-modal sensor data fusion

Abstract

The invention provides a limb nerve rehabilitation state evaluation method based on multi-modal sensor data fusion. Motion inertia data, electromyographic signal data and text data of a corresponding limb nerve rehabilitation action manual of a patient during rehabilitation actions are obtained in real time; inputting the data into a designed and trained multi-modal data feature extraction network to obtain motion inertial data features, electromyographic signal data features and text semantic features; inputting the motion inertia data features, the electromyographic signal data features and the text semantic features into a designed and trained multi-modal data feature fusion network to obtain multi-modal fusion features; and inputting the multi-modal fusion features into a trained prediction output network to obtain a limb nerve rehabilitation state prediction value of the patient. Through multi-modal data fusion and time-dependent feature extraction, the limb nerve rehabilitation state of the patient is evaluated more accurately, and a more reliable diagnosis basis is provided for doctors. Meanwhile, the labor cost is reduced and the evaluation efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal data fusion neural rehabilitation intelligent detection, and in particular relates to a limb neural rehabilitation state assessment method based on multimodal sensor data fusion. Background Art

[0002] In recent years, with the rapid progress of artificial intelligence and Internet of Things technologies, a limb nerve rehabilitation status assessment method based on artificial intelligence technology has been proposed and applied in clinical diagnosis, assisting doctors in judging the limb nerve rehabilitation status of patients, solving labor costs and improving diagnostic efficiency.

[0003] For example, the Chinese invention patent with application number CN202310635568.X proposes a control method and system for the interactive behavior of a rehabilitation robot. The specific steps include: first, obtaining the electromyographic signals of the monitored patient in a predetermined time period, the respiratory rate values ​​and heart rate values ​​at multiple predetermined time points in the predetermined time period, and a text description of the rehabilitation process of the monitored patient; then, through artificial intelligence and deep learning technology, the correlation feature distribution information between the temporal collaborative correlation features of the patient's electromyographic signals, respiratory rate values ​​and heart rate values ​​and the semantic understanding features of the text description of the patient's rehabilitation process is fully expressed, so as to accurately detect and evaluate the patient's rehabilitation status.

[0004] The Chinese invention patent with application number CN202210436298.5 proposes a method for EEG modeling of motor imagery for patients with unilateral limbs based on neuron optimization, which includes the following steps: Step 1, professionals help patients wear and use EEG acquisition equipment; Step 2, patients perform motor imagery according to prompts, and the EEG acquisition equipment transmits the motor imagery EEG signals to the computer through wired or wireless transmission for personalized training modeling; Step 3, after receiving the EEG data, the computer system first pre-processes the data, including filtering, removing baselines, and data slicing integration; Step 4, the processed data is sent to the deep learning model for feature extraction and output classification results.

[0005] However, the above solutions and prior art attempts have the following defects: 1. The existing technology fails to fully consider the effective fusion of sensor data of different modalities. It only performs simple data feature matching and cannot fully integrate data information of multiple modalities. The matching accuracy between data of different modalities is low, resulting in poor accuracy in limb nerve rehabilitation status assessment; 2. The existing technology fails to fully consider the time dependency of the limb data obtained by the sensor. The limb rehabilitation movements performed by the patient are time-continuous. Therefore, fully capturing the intrinsic time dependency of the limb data can improve the accuracy of the assessment of the limb nerve rehabilitation status. Summary of the invention

[0006] In view of the above problems, the present invention introduces multimodal data fusion and time-dependent feature extraction to more accurately evaluate the patient's limb nerve rehabilitation status and provide doctors with a more reliable diagnostic basis.

[0007] The present invention provides a method for assessing the state of limb nerve rehabilitation based on multimodal sensor data fusion, comprising the following steps: S1, real-time acquisition of the patient's motion inertia data, electromyographic signal data, and text data of the corresponding limb nerve rehabilitation action manual when the patient is doing rehabilitation movements; S2, inputting the data in S1 into the trained multimodal data feature extraction network, wherein the multimodal data feature extraction network includes a time-dependent feature extraction module and a text semantic feature extraction module; wherein the motion inertia data and the electromyographic signal data are input into the time-dependent feature extraction module to obtain motion inertia data features and electromyographic signal data features, and the limb nerve rehabilitation movement manual text data is input into the text semantic feature extraction module to obtain text semantic features; S3, inputting the motion inertia data features, electromyographic signal data features and text semantic features into the trained multimodal data feature fusion network to obtain multimodal fusion features; S4, input the multimodal fusion features into the trained prediction output network to obtain the patient's limb nerve rehabilitation status prediction value, and the doctor makes a follow-up diagnosis on the patient based on the limb nerve rehabilitation status prediction value.

[0008] Furthermore, the motion inertia data is specifically: While the patient performs the rehabilitation movements as described in the text in the limb rehabilitation manual, the patient performs the first The X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data and yaw angle time series data generated by each limb rehabilitation movement.

[0009] Furthermore, the time-dependent feature extraction module includes a first recurrent neural network layer, a second recurrent neural network layer, a first convolutional layer, a second convolutional layer, a first multi-head attention mechanism layer, a second multi-head attention mechanism layer, a first LSTM layer, a second LSTM layer, a third convolutional layer, a fourth convolutional layer and a feature fusion layer; Motion inertia data characteristics The specific acquisition process is as follows: S1, the patient's motion inertia data Perform sequence flip to obtain reverse motion inertial data sequence ; S2, will and The first recurrent neural network layer and the second recurrent neural network layer are respectively input, and the spatial dependency relationship between each limb neurorehabilitation action in each dimension of the X-axis acceleration time series data, the Y-axis acceleration time series data, the Z-axis acceleration time series data, the X-axis angular velocity time series data, the Y-axis angular velocity time series data, the Z-axis angular velocity time series data, the roll angle time series data, the pitch angle time series data and the yaw angle time series data is extracted using the recurrent neural network to obtain the positive spatial dependency features. and inverse spatial dependence features ; S3, the positive spatial dependency feature and inverse spatial dependence features The first multi-head attention mechanism and the second multi-head attention mechanism are input respectively, and the multi-head attention mechanism is used to calculate the association weights between the 9 dimensions of X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data and yaw angle time series data, and the positive spatial dependency feature is calculated. and inverse spatial dependence features The features between the 9-dimensional data are weighted and summed to obtain the forward fusion features and reverse fusion features ; S4, forward fusion features and reverse fusion features Input the first convolutional layer and the second convolutional layer respectively, the convolution kernel scales of the first convolutional layer and the second convolutional layer are both , the purpose is to increase the forward fusion features and reverse fusion features The number of channels is used to transform low-dimensional features into high-dimensional features and obtain positive high-dimensional features and reverse high-dimensional features ; S5, forward high-dimensional features and reverse high-dimensional features Input the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the positive time dependency in the sequence; the second LSTM layer captures the reverse time dependency in the sequence. The specific calculation formula is as follows: ; ; in, and These are the weight matrices for the first LSTM layer and the second LSTM layer respectively; and They are the biases of the first LSTM layer and the second LSTM layer respectively; and They are the activation function sets for the first LSTM layer and the second LSTM layer respectively; and They are respectively the positive time long-term dependence feature and the reverse time long-term dependence feature; S6, the positive time long-term dependence feature and reverse time-dependent characteristics The third convolutional layer and the fourth convolutional layer are input respectively, and the convolution kernel scales of the third convolutional layer and the fourth convolutional layer are , convert high-dimensional features into low-dimensional features to obtain positive low-dimensional features and inverse low-dimensional features ; S7, forward low-dimensional features and inverse low-dimensional features The input feature fusion layer performs weighted fusion of forward and reverse features to obtain motion inertial data features. .

[0010] Furthermore, the time-dependent feature extraction module includes a first recurrent neural network layer, a second recurrent neural network layer, a first convolutional layer, a second convolutional layer, a first multi-head attention mechanism layer, a second multi-head attention mechanism layer, a first LSTM layer, a second LSTM layer, a third convolutional layer, a fourth convolutional layer and a feature fusion layer; The electromyographic signal data features The specific process of obtaining is: S1, the patient's electromyographic signal data Perform sequence flip to obtain reverse electromyographic signal data sequence ; S2, and Input the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract and The spatial dependency relationship between the limb nerve rehabilitation actions in the limb nerve rehabilitation process is used to obtain the positive spatial dependency features. and inverse spatial dependence features ; S3, the positive spatial dependency feature and inverse spatial dependence features Input the first multi-head attention mechanism and the second multi-head attention mechanism respectively, and use the multi-head attention mechanism to calculate and The association weights between the limb neurorehabilitation actions in the and inverse spatial dependence features middle The weighted summation of the features between the limb neurorehabilitation actions is performed to obtain the forward fusion features. and reverse fusion features ; S4, forward fusion features and reverse fusion features Input the first convolutional layer and the second convolutional layer respectively, and the convolution kernel scales of the first convolutional layer and the second convolutional layer are both , transform low-dimensional features into high-dimensional features to obtain positive high-dimensional features and reverse high-dimensional features ; S5, forward high-dimensional features and reverse high-dimensional features Input the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the positive time dependency in the sequence; the second LSTM layer captures the reverse time dependency in the sequence. The specific calculation formula is as follows: ; The specific calculation formula is as follows: ; ; in, and These are the weight matrices for the first LSTM layer and the second LSTM layer respectively; and They are the biases of the first LSTM layer and the second LSTM layer respectively; and They are the activation function sets for the first LSTM layer and the second LSTM layer respectively; and They are respectively the positive time long-term dependence feature and the reverse time long-term dependence feature; S6, the positive time long-term dependence feature and reverse time-dependent characteristics The third convolutional layer and the fourth convolutional layer are input respectively, and the convolution kernel scales of the third convolutional layer and the fourth convolutional layer are , convert high-dimensional features into low-dimensional features to obtain positive low-dimensional features and inverse low-dimensional features ; S7, forward low-dimensional features and inverse low-dimensional features The input feature fusion layer performs weighted fusion of forward and reverse features to obtain the EMG data features. .

[0011] Furthermore, the semantic text feature extraction module includes a text convolution layer, a first gated unit layer, a second gated unit layer, a first maximum pooling layer, a second maximum pooling layer and a feature splicing layer; the specific processing process is: S1, limb rehabilitation action text description data Input the text convolution layer to obtain local text features. The specific calculation formula is as follows: ; in, is the local feature obtained through the text convolution layer; is the text convolution kernel, is the text convolution kernel width, The number of input or output channels of the text convolution kernel; for Sequence at position of Dimension word vector window; is the bias term; Activation function for the text convolution layer; S2, Sequentially input the first gated unit layer and the second gated unit layer to obtain global semantic features , capturing the temporal dependencies between sequence data and storing continuous limb rehabilitation action text state information in the hidden layer; S3, and Input the first maximum pooling layer and the second maximum pooling layer respectively to obtain low-dimensional local semantic features and low-dimensional global semantic features , the maximum pooling layer pair and Downsampling is performed to retain the most significant local features while reducing the number of feature parameters; S4, low-dimensional local semantic features and low-dimensional global semantic features The input features are concatenated to obtain text semantic features that represent local semantics and global semantic dependencies. , the specific calculation formula is as follows: = [CLS]+ +[SEP] + +[SEP]; Among them, [CLS] and [SEP] are sequence start and separation markers respectively.

[0012] Furthermore, the multimodal data feature fusion network includes a first feature segmentation layer, a second feature segmentation layer, a third feature segmentation layer, a rehabilitation action feature attention layer and a random inactivation layer; the specific processing process is: S1, the motion inertia data features , EMG signal data characteristics and text semantic features Input the first feature segmentation layer, the second feature segmentation layer, and the third feature segmentation layer respectively, segment the features according to the limb nerve rehabilitation action category, and obtain the motion inertia data feature segmentation set , EMG signal data feature segmentation set and text semantic feature segmentation set ; S2, segment the motion inertia data features into a set , EMG signal data feature segmentation set and text semantic feature segmentation set Input the rehabilitation action feature attention layer, calculate the attention at different rehabilitation action levels, capture the complex interactive relationship between them, and perform weighted feature fusion. The specific calculation formula is as follows: ; ; in, For the The weight of each limb rehabilitation action, is the multimodal feature matrix , is the learnable weight matrix, is the multimodal fusion feature. represent Limb rehabilitation exercises.

[0013] Furthermore, the prediction output network is composed of three fully connected layers and a Softmax activation function layer connected in sequence, which combines the multimodal fusion features output by the multimodal data feature fusion network Input prediction output network obtains patient limb nerve rehabilitation status assessment results , the specific calculation formula is as follows: ; ; ; ; in, 、 、 and are the weight matrices of the first fully connected layer, the second fully connected layer, the third fully connected layer, and the Softmax activation function layer, respectively. 、 、 and They are the biases of the first fully connected layer, the second fully connected layer, the third fully connected layer and the Softmax activation function layer respectively.

[0014] Furthermore, the obtained training set and test set are used to perform overall training and testing on the multimodal data feature extraction network, the multimodal data feature fusion network and the prediction output network. The loss function adopts the cross entropy loss function, and the specific calculation formula is as follows: ; in, This is the real condition of the patient's limb nerve rehabilitation. It is the patient's limb nerve rehabilitation status obtained by the prediction output network. is the training set size.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses multimodal data fusion and time-dependent feature extraction to more accurately assess the patient's limb nerve rehabilitation status, providing doctors with a more reliable basis for diagnosis. At the same time, it reduces labor costs. Traditional limb nerve rehabilitation status assessment requires a lot of human resources and time costs, while the method proposed in the present invention uses artificial intelligence technology to achieve automated assessment, significantly reducing labor costs and improving assessment efficiency.

[0016] 1. Multimodal data fusion: This invention proposes a method for assessing the state of limb nerve rehabilitation based on multimodal sensor data fusion. Different from the prior art that simply matches the data features of different modalities, this invention achieves the effective fusion of three modal data, namely, motion inertial data, electromyographic signal data and text data, by constructing a new data processing flow and network structure. This fusion not only improves the utilization rate of data, but also significantly improves the accuracy of limb nerve rehabilitation state assessment.

[0017] 2. Time-dependent feature extraction: Considering that the limb rehabilitation movements performed by patients are time-continuous, the present invention introduces a time-dependent feature extraction module in the data feature extraction stage. This module uses technologies such as recurrent neural network (RNN), long short-term memory network (LSTM) and multi-head attention mechanism to effectively capture the time dependency in motion inertia data and electromyographic signal data, thereby further improving the accuracy of the evaluation.

[0018] 3. Text semantic feature extraction: For the text description in the limb rehabilitation manual, the present invention designs a text semantic feature extraction module. This module fully mines the local semantics and global semantic dependencies of text data through structures such as text convolution layer, gated unit layer, and maximum pooling layer, and realizes high-quality feature learning. This processing method enables text data to play a greater role in the assessment of limb nerve rehabilitation status.

[0019] 4. Multimodal data feature fusion network: In the feature fusion stage, the present invention constructs a multimodal data feature fusion network. The network realizes the correlation calculation between motion inertia data features, electromyographic signal data features and text semantic features through structures such as feature segmentation layer, rehabilitation action feature attention layer and random inactivation layer, and performs sufficient feature fusion at the rehabilitation action level. This fusion method not only improves the accuracy of the evaluation, but also enhances the interpretability of the model.

[0020] At the same time, the method proposed in the present invention is not only applicable to the assessment of limb nerve rehabilitation status, but can also be extended to other fields that require multimodal data fusion, such as motion analysis, health monitoring, etc. This extensibility makes the method proposed in the present invention have a wider application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is the overall logic flow chart of the present invention.

[0022] Figure 2 It is a schematic diagram of the structure of the time-dependent feature extraction module of the present invention.

[0023] Figure 3 It is a schematic diagram of the structure of the semantic text feature extraction module of the present invention.

[0024] Figure 4 Schematic diagram of the multimodal data feature fusion network structure of the present invention.

[0025] Figure 5 It is a comparison chart of the experimental results of the present invention. DETAILED DESCRIPTION

[0026] The overall technical route of the present invention is as follows Figure 1As shown in the figure: first, a data set is constructed, and data annotation is performed and divided into training set and test set; then a multimodal data feature extraction network, a multimodal data feature fusion network and a prediction output network are constructed. After the three networks are trained through the data set and training strategy, they are deployed in the rehabilitation monitoring system for real-time detection and evaluation: the patient's motion inertia data and electromyographic signal data and the text data of the limb nerve rehabilitation action manual are collected; the collected motion inertia data and electromyographic signal data are input into the trained time-dependent feature extraction module to obtain motion inertia data features and electromyographic signal data features; the text data of the limb nerve rehabilitation action manual is input into the trained text semantic feature extraction module to obtain text semantic features; the motion inertia data features, electromyographic signal data features and text semantic features are input into the trained multimodal data feature fusion network to obtain multimodal fusion features; the multimodal fusion features are input into the trained prediction output network to obtain the patient's limb nerve rehabilitation status prediction value, and the doctor makes a subsequent diagnosis of the patient based on the limb nerve rehabilitation status prediction value and his own experience.

[0027] The invention will be further described below in conjunction with specific embodiments.

[0028] 1. Dataset Preparation In this invention, high-quality datasets are the key basis for ensuring good performance of the model. To this end, this application proposal adopts a rigorous dataset production process, aiming to provide sufficient high-quality training data for the model and improve the recommendation accuracy. 1. Data collection: This application proposal needs to collect three types of data to comprehensively evaluate the patient's limb nerve rehabilitation status. The first type of modal data is motion inertial data. Inertial sensors are used to collect the three-axis acceleration time series data, three-axis angular velocity time series data, roll angle time series data, pitch angle time series data, and yaw angle time series data generated by the patient's upper limb movement when performing rehabilitation movements; the second type of modal data is electromyographic signal data, which plays an important role in rehabilitation engineering. Myoelectricity not only carries information such as nerve status and muscle status, but also contains a large amount of rehabilitation-related information such as the patient's disease recovery status and disease progression. The information contained in electromyographic data is of great value in rehabilitation effect evaluation, muscle fatigue judgment and prediction, etc., and patch-type electromyographic sensors are used to collect electromyographic signal time series data in upper limb muscles; the third type of modal data is text data, specifically the text description of each limb rehabilitation movement in the limb rehabilitation manual; The specific steps of making motion inertial data set are as follows: After wearing the inertial sensor on the upper limb, the patient performed the movement correction according to the text description in the limb rehabilitation manual. After the first limb rehabilitation movement, the patient who recorded the inertial sensor output performed the second X-axis acceleration time series data generated by limb rehabilitation movements , Y-axis acceleration time series data , Z-axis acceleration time series data , X-axis angular velocity time series data , Y-axis angular velocity time series data , Z-axis angular velocity time series data , Roll angle time series data , pitch angle time series data and yaw angle time series data ; Each of the patients will After all the limb rehabilitation movements are completed, the motion inertia data set is obtained. ; The specific steps of preparing the EMG signal dataset are as follows: The patient's upper limb muscles are located in the patient's standard After the first limb rehabilitation movement, the patient performed the second Time series data of myoelectric signals generated by limb rehabilitation movements ; Each of the patients will After all the limb rehabilitation movements are completed, the electromyographic signal data set is obtained ; The specific steps of making the text dataset are as follows: First, The text description of each limb rehabilitation action is cleaned to remove non-text information, conjunctions and auxiliary words, including: Regular expression replacement: replace "!", "?", "~", "\", "@", "#", "$" and "%" in the text with spaces; Invalid data deletion: delete invalid spaces in the information; Text segmentation: This application uses NLPIR to perform word segmentation on the text description of limb rehabilitation actions. This is an efficient and accurate word segmentation tool that can identify and divide words in the text and improve the degree of structuring of the corpus. Through word segmentation by NLPIR, the key steps, important joint parts and movement ranges in the text description of limb rehabilitation actions can be more accurately extracted. The text data structure after word segmentation processing is clearer, which is conducive to subsequent feature extraction and semantic analysis; The text data structure after word segmentation processing is clearer, which is conducive to subsequent feature extraction and semantic analysis; After all the limb rehabilitation action text descriptions are cleaned, the limb rehabilitation action text description dataset is obtained. ; 2. Data annotation Doctors will The patients were divided into three levels: excellent limb nerve rehabilitation, good limb nerve rehabilitation and poor limb nerve rehabilitation. The three rehabilitation levels were coded using one-hot coding, where excellent limb nerve rehabilitation was coded as 0, good limb nerve rehabilitation was coded as 1, and poor limb nerve rehabilitation was coded as 2. A collection of limb neurorehabilitation status labels for each patient ; 3. Preparation of training and test sets The motion inertia dataset , EMG signal dataset , Limb Rehabilitation Action Text Description Dataset And the corresponding limb nerve rehabilitation status label set The data are divided into two independent data sets, a training set and a test set, in a ratio of 8:2. The training set is used for iterative learning of model parameters, and the test set is used to evaluate model accuracy. The test set is constructed by random sampling using the data set partitioning tool and is independent of the training set data.

[0029] 2. Multimodal Data Feature Extraction Network The multimodal data feature extraction network includes a time-dependent feature extraction module and a text semantic feature extraction module. The time consistency feature extraction module is used to extract the features of motion inertia data and electromyographic signal data to capture time dependency; the text semantic feature extraction module is used to extract the semantic features of the text description data of limb rehabilitation movements. 1. Time-dependent feature extraction module Time-dependent feature extraction module such as Figure 2 As shown, it includes the first recurrent neural network layer, the second recurrent neural network layer, the first convolutional layer, the second convolutional layer, the first multi-head attention mechanism layer, the second multi-head attention mechanism layer, the first LSTM layer, the second LSTM layer, the third convolutional layer, the fourth convolutional layer and the feature fusion layer; the first Motion inertial data of each patient and EMG data of patients Input the time-dependent feature extraction module to obtain motion inertia data features respectively and EMG data features , the specific calculation steps are as follows: (1) Acquisition of motion inertia data features: The first step is to Perform sequence flip to obtain reverse motion inertial data sequence ; The second step is to and The first recurrent neural network layer and the second recurrent neural network layer are respectively input, and the spatial dependency relationship between each limb neurorehabilitation action in each dimension of the X-axis acceleration time series data, the Y-axis acceleration time series data, the Z-axis acceleration time series data, the X-axis angular velocity time series data, the Y-axis angular velocity time series data, the Z-axis angular velocity time series data, the roll angle time series data, the pitch angle time series data and the yaw angle time series data is extracted using the recurrent neural network to obtain the positive spatial dependency features. and inverse spatial dependence features ; The third step is to transform the positive spatial dependency feature and inverse spatial dependence features The first multi-head attention mechanism and the second multi-head attention mechanism are input respectively, and the multi-head attention mechanism is used to calculate the association weights between the 9 dimensions of X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data and yaw angle time series data, and the positive spatial dependency feature is calculated. and inverse spatial dependence features The features between the 9-dimensional data are weighted and summed to obtain the forward fusion features and reverse fusion features ; The fourth step is to forward fusion features and reverse fusion features Input the first convolutional layer and the second convolutional layer respectively, and the convolution kernel scales of the first convolutional layer and the second convolutional layer are both , the purpose is to increase the forward fusion features and reverse fusion features The number of channels is used to transform low-dimensional features into high-dimensional features, improve the expressiveness of features, and obtain positive high-dimensional features. and reverse high-dimensional features ; The fifth step is to convert the positive high-dimensional features and reverse high-dimensional features Input the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the positive time dependency in the sequence; the second LSTM layer captures the reverse time dependency in the sequence. This bidirectional processing can retain the key information in the sequence and pass it to the subsequent layers for more complex feature extraction and prediction. The specific calculation formula is as follows: ; ; in, and These are the weight matrices for the first LSTM layer and the second LSTM layer respectively; and They are the biases of the first LSTM layer and the second LSTM layer respectively; and They are the activation function sets for the first LSTM layer and the second LSTM layer respectively; and They are respectively the positive time long-term dependence feature and the reverse time long-term dependence feature; Step 6: Long-term positive time dependency features and reverse time-dependent characteristics The third convolutional layer and the fourth convolutional layer are input respectively, and the convolution kernel scales of the third convolutional layer and the fourth convolutional layer are , the purpose is to reduce the positive time long-term dependence characteristics and reverse time-dependent characteristics The number of channels is used to convert high-dimensional features into low-dimensional features, reduce the computational burden of the model and the risk of overfitting, and obtain positive low-dimensional features. and inverse low-dimensional features ; The seventh step is to convert the positive low-dimensional features and inverse low-dimensional features The input feature fusion layer performs weighted fusion of forward and reverse features to obtain motion inertial data features. , the specific calculation formula is as follows: ; (2) Acquisition of electromyographic signal data features: The first step is to Perform sequence flip to obtain reverse electromyographic signal data sequence ; The second step is to and Input the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract and The spatial dependency relationship between the limb nerve rehabilitation actions in the limb nerve rehabilitation process is used to obtain the positive spatial dependency features. and inverse spatial dependence features ; The third step is to transform the positive spatial dependency feature and inverse spatial dependence features Input the first multi-head attention mechanism and the second multi-head attention mechanism respectively, and use the multi-head attention mechanism to calculate and The association weights between the limb neurorehabilitation actions in the and inverse spatial dependence features middle The weighted summation of the features between the limb neurorehabilitation actions is performed to obtain the forward fusion features. and reverse fusion features ; The fourth step is to forward fusion features and reverse fusion features Input the first convolutional layer and the second convolutional layer respectively, the convolution kernel scales of the first convolutional layer and the second convolutional layer are both , the purpose is to increase the forward fusion features and reverse fusion features The number of channels is used to transform low-dimensional features into high-dimensional features, improve the expressiveness of features, and obtain positive high-dimensional features. and reverse high-dimensional features ; The fifth step is to convert the positive high-dimensional features and reverse high-dimensional features Input the first LSTM layer and the second LSTM layer respectively. The first LSTM layer captures the positive time dependency in the sequence; the second LSTM layer captures the reverse time dependency in the sequence. This bidirectional processing can retain the key information in the sequence and pass it to the subsequent layers for more complex feature extraction and prediction. The specific calculation formula is as follows: ; ; in, and These are the weight matrices for the first LSTM layer and the second LSTM layer respectively; and They are the biases of the first LSTM layer and the second LSTM layer respectively; and They are the activation function sets for the first LSTM layer and the second LSTM layer respectively; and They are respectively the positive time long-term dependence feature and the reverse time long-term dependence feature; Step 6: Long-term positive time dependency features and reverse time-dependent characteristics The third convolutional layer and the fourth convolutional layer are input respectively, and the convolution kernel scales of the third convolutional layer and the fourth convolutional layer are , the purpose is to reduce the positive time long-term dependence characteristics and reverse time-dependent characteristics The number of channels is used to convert high-dimensional features into low-dimensional features, reduce the computational burden of the model and the risk of overfitting, and obtain positive low-dimensional features. and inverse low-dimensional features ; The seventh step is to convert the positive low-dimensional features and inverse low-dimensional features The input feature fusion layer performs weighted fusion of forward and reverse features to obtain the EMG data features. , the specific calculation formula is as follows: ; 2. Text semantic feature extraction module The text semantic feature extraction module is used to extract the semantic features of the text description data of limb rehabilitation actions, such as Figure 3 As shown, it includes a text convolution layer, a first gated unit layer, a second gated unit layer, a first maximum pooling layer, a second maximum pooling layer and a feature concatenation layer to fully explore the local semantics and global semantic dependencies of the text description data of limb rehabilitation movements and achieve high-quality feature learning; Limb Rehabilitation Action Text Description Dataset Input text semantic feature extraction module to obtain the semantic features of the operation text , the specific calculation steps are as follows: Step 1: The text convolution layer is used to obtain local text features. The text convolution neural network has a strong feature learning ability in natural language processing tasks. It is particularly good at automatically extracting local hierarchical features from raw data and capturing the underlying patterns that constitute semantics. This layer applies a sliding convolution kernel to the input limb rehabilitation action text description dataset. Perform convolution operation on it to generate a new feature mapping sequence. The specific calculation formula is as follows: ; in, is the local feature obtained through the text convolution layer; is the text convolution kernel, is the text convolution kernel width, The number of input and output channels of the text convolution kernel; for Sequence at position of Dimension word vector window; is the bias term; is the activation function of the text convolution layer. The convolution structure can effectively capture the local correlation of the input data, thereby extracting rich feature representations; Step 2: Sequentially input the first gated unit layer and the second gated unit layer to obtain global semantic features ,The gate unit can effectively capture the temporal dependencies between the sequence data and store the continuous limb rehabilitation action text state information in the hidden layer.,Each operator in the model is associated with a corresponding weight; Step 3: and Input the first maximum pooling layer and the second maximum pooling layer respectively to obtain low-dimensional local semantic features and low-dimensional global semantic features , the maximum pooling layer pair and Downsampling is performed to retain the most significant local features while reducing the number of feature parameters. The maximum pooling layer retains the most important information when extracting features, allowing the model to focus more on the most significant parts of the features. Step 4: low-dimensional local semantic features and low-dimensional global semantic features The input features are concatenated to obtain text semantic features that represent local semantics and global semantic dependencies. , the specific calculation formula is as follows: = [CLS]+ +[SEP] + +[SEP]; Among them, [CLS] and [SEP] are sequence start and separation markers respectively.

[0030] 3. Multimodal Data Feature Fusion Network Multimodal data feature fusion network Figure 4 As shown, it includes the first feature segmentation layer, the second feature segmentation layer, the third feature segmentation layer, the rehabilitation action feature attention layer and the random inactivation layer, which are used to fully calculate the motion inertia data features. , EMG signal data characteristics and text semantic features The correlation between the three can achieve full feature fusion at the rehabilitation action level. The specific steps are as follows: The first step is to transform the motion inertia data features , EMG signal data characteristics and text semantic features Input the first feature segmentation layer, the second feature segmentation layer, and the third feature segmentation layer respectively, segment the features according to the limb nerve rehabilitation action category, and obtain the motion inertia data feature segmentation set , EMG signal data feature segmentation set and text semantic feature segmentation set ,in , and Respectively represent The motion inertia data features, electromyographic signal data features and text data features corresponding to each limb rehabilitation action; The second step is to segment the motion inertia data features into a set , EMG signal data feature segmentation set and text semantic feature segmentation set Input the rehabilitation action feature attention layer, calculate the attention at different rehabilitation action levels, capture the complex interactive relationship between them, and perform weighted feature fusion. This method can significantly improve the accuracy and interpretability of multimodal fusion. The specific calculation formula is as follows: ; ; in, For the The weight of each limb rehabilitation action, is the multimodal feature matrix , is the learnable weight matrix, It is a multi-modal fusion feature.

[0031] 4. Prediction Output Network The prediction output network consists of three fully connected layers and a Softmax activation function layer connected in sequence. Input prediction output network obtains patient limb nerve rehabilitation status assessment results , the specific calculation formula is as follows: ; ; ; ; in, 、 、 and are the weight matrices of the first fully connected layer, the second fully connected layer, the third fully connected layer, and the Softmax activation function layer, respectively. 、 、 and They are the biases of the first fully connected layer, the second fully connected layer, the third fully connected layer and the Softmax activation function layer respectively.

[0032] 5. Model Training The obtained training set and test set are used to conduct overall training and testing on the time-dependent feature extraction module, text semantic feature extraction module, multimodal data feature fusion network and prediction output network. The loss function adopts the cross entropy loss function. The specific calculation formula is as follows: ; in, This is the real condition of the patient's limb nerve rehabilitation. It is the patient's limb nerve rehabilitation status obtained by the prediction output network. is the training set size.

[0033] 6. Model Deployment and Experimentation Step 1: Collect the patient's motion inertia data and electromyographic signal data and limb neurorehabilitation action manual text data according to the data training data set collection method; Step 2: Input the motion inertia data and electromyographic signal data collected in step 1 into the trained time-dependent feature extraction module to obtain motion inertia data features and electromyographic signal data features; input the text data of the limb nerve rehabilitation action manual into the trained text semantic feature extraction module to obtain text semantic features; Step 3: Input the motion inertia data features, electromyography signal data features and text semantic features into the trained multimodal data feature fusion network to obtain multimodal fusion features; Step 4: Input the multimodal fusion features into the trained prediction output network to obtain the patient's limb nerve rehabilitation status prediction value. The doctor makes a subsequent diagnosis of the patient based on the limb nerve rehabilitation status prediction value and his own experience.

[0034] In order to verify the effectiveness of the method of the present invention, a test set consisting of 20 patients was used to verify the method proposed by the present invention. Figure 5 As shown, sample numbers 1-20 represent 20 patients, the blue circles represent the actual values ​​of the limb nerve rehabilitation status of Parkinson's patients, and the orange circles represent the predicted values ​​of the limb nerve rehabilitation status of the patients. It can be seen from the figure that the number of samples predicted incorrectly by the method of the present invention is 1, indicating that the method of the present invention has a high prediction accuracy.

[0035] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0036] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for assessing limb nerve rehabilitation status based on multimodal sensor data fusion, characterized in that: The following steps are involved: S1, real-time acquisition of the patient's motion inertia data, electromyographic signal data, and text data of the corresponding limb nerve rehabilitation action manual when the patient is doing rehabilitation movements; S2, inputting the data in S1 into the trained multimodal data feature extraction network, wherein the multimodal data feature extraction network includes a time-dependent feature extraction module and a text semantic feature extraction module; The motion inertia data and electromyographic signal data are input into the time-dependent feature extraction module to obtain motion inertia data features and electromyographic signal data features, and the limb nerve rehabilitation action manual text data is input into the text semantic feature extraction module to obtain text semantic features; S3, inputting the motion inertia data features, electromyographic signal data features and text semantic features into the trained multimodal data feature fusion network to obtain multimodal fusion features; S4, input the multimodal fusion features into the trained prediction output network to obtain the patient's limb nerve rehabilitation status prediction value, and the doctor makes a follow-up diagnosis on the patient based on the limb nerve rehabilitation status prediction value.

2. The method for assessing limb nerve rehabilitation status based on multimodal sensor data fusion according to claim 1, characterized in that: The motion inertia data is specifically: While the patient performs the rehabilitation movements as described in the text in the limb rehabilitation manual, the patient performs the first The X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data and yaw angle time series data generated by each limb rehabilitation movement.

3. The method for assessing limb nerve rehabilitation status based on multimodal sensor data fusion according to claim 1, characterized in that: The time-dependent feature extraction module includes a first recurrent neural network layer, a second recurrent neural network layer, a first convolutional layer, a second convolutional layer, a first multi-head attention mechanism layer, a second multi-head attention mechanism layer, a first LSTM layer, a second LSTM layer, a third convolutional layer, a fourth convolutional layer and a feature fusion layer; Motion inertia data characteristics The specific acquisition process is as follows: S1, the patient's motion inertia data Perform sequence flip to obtain reverse motion inertial data sequence ; S2, and The first recurrent neural network layer and the second recurrent neural network layer are respectively input, and the spatial dependency relationship between each limb neurorehabilitation action in each dimension of the X-axis acceleration time series data, the Y-axis acceleration time series data, the Z-axis acceleration time series data, the X-axis angular velocity time series data, the Y-axis angular velocity time series data, the Z-axis angular velocity time series data, the roll angle time series data, the pitch angle time series data and the yaw angle time series data is extracted using the recurrent neural network to obtain the positive spatial dependency features. and inverse spatial dependence features ; S3, the positive spatial dependency feature Inverse spatial dependency features The first multi-head attention mechanism and the second multi-head attention mechanism are input respectively, and the multi-head attention mechanism is used to calculate the association weights between the 9 dimensions of X-axis acceleration time series data, Y-axis acceleration time series data, Z-axis acceleration time series data, X-axis angular velocity time series data, Y-axis angular velocity time series data, Z-axis angular velocity time series data, roll angle time series data, pitch angle time series data and yaw angle time series data, and the positive spatial dependency feature is calculated. and inverse spatial dependence features The features between the 9-dimensional data are weighted and summed to obtain the forward fusion features and reverse fusion features ; S4, forward fusion features and reverse fusion features Input the first convolutional layer and the second convolutional layer respectively, the convolution kernel scales of the first convolutional layer and the second convolutional layer are both , the purpose is to increase the forward fusion features and reverse fusion features The number of channels is used to transform low-dimensional features into high-dimensional features and obtain positive high-dimensional features and reverse high-dimensional features ; S5, forward high-dimensional features and reverse high-dimensional features Input into the first LSTM layer and the second LSTM layer respectively, the first LSTM layer captures the positive temporal dependency in the sequence; The second LSTM layer captures the reverse time dependency in the sequence; the specific calculation formula is as follows: ； ； in, and These are the weight matrices for the first LSTM layer and the second LSTM layer respectively; and They are the biases of the first LSTM layer and the second LSTM layer respectively; and They are the activation function sets for the first LSTM layer and the second LSTM layer respectively; and They are respectively the positive time long-term dependence feature and the reverse time long-term dependence feature; S6, the positive time long-term dependence feature and reverse time-dependent characteristics The third convolutional layer and the fourth convolutional layer are input respectively, and the convolution kernel scales of the third convolutional layer and the fourth convolutional layer are , convert high-dimensional features into low-dimensional features to obtain positive low-dimensional features and inverse low-dimensional features ; S7, forward low-dimensional features and inverse low-dimensional features The input feature fusion layer performs weighted fusion of forward and reverse features to obtain motion inertial data features. .

4. The method for assessing limb nerve rehabilitation status based on multimodal sensor data fusion according to claim 1, characterized in that: The time-dependent feature extraction module includes a first recurrent neural network layer, a second recurrent neural network layer, a first convolutional layer, a second convolutional layer, a first multi-head attention mechanism layer, a second multi-head attention mechanism layer, a first LSTM layer, a second LSTM layer, a third convolutional layer, a fourth convolutional layer and a feature fusion layer; The electromyographic signal data features The specific process of obtaining is: S1, the patient's electromyographic signal data Perform sequence flip to obtain reverse electromyographic signal data sequence ; S2, and Input the first recurrent neural network layer and the second recurrent neural network layer respectively, and use the recurrent neural network to extract and The spatial dependency relationship between the limb nerve rehabilitation actions in the limb nerve rehabilitation process is used to obtain the positive spatial dependency features. and inverse spatial dependence features ; S3, the positive spatial dependency feature and inverse spatial dependence features Input the first multi-head attention mechanism and the second multi-head attention mechanism respectively, and use the multi-head attention mechanism to calculate and The association weights between the limb neurorehabilitation actions in the and inverse spatial dependence features middle The weighted summation of the features between the limb neurorehabilitation actions is performed to obtain the forward fusion features. and reverse fusion features ; S4, forward fusion features and reverse fusion features Input the first convolutional layer and the second convolutional layer respectively, the convolution kernel scales of the first convolutional layer and the second convolutional layer are both , transform low-dimensional features into high-dimensional features to obtain positive high-dimensional features and reverse high-dimensional features ; S5, forward high-dimensional features and reverse high-dimensional features Input into the first LSTM layer and the second LSTM layer respectively, the first LSTM layer captures the positive temporal dependency in the sequence; The second LSTM layer captures the reverse time dependency in the sequence; the specific calculation formula is as follows: ； ； in, and These are the weight matrices for the first LSTM layer and the second LSTM layer respectively; and They are the biases of the first LSTM layer and the second LSTM layer respectively; and They are the activation function sets for the first LSTM layer and the second LSTM layer respectively; and They are respectively the positive time long-term dependence feature and the reverse time long-term dependence feature; S6, the positive time long-term dependence feature and reverse time-dependent characteristics The third convolutional layer and the fourth convolutional layer are input respectively, and the convolution kernel scales of the third convolutional layer and the fourth convolutional layer are , convert high-dimensional features into low-dimensional features to obtain positive low-dimensional features and inverse low-dimensional features ; S7, forward low-dimensional features and inverse low-dimensional features The input feature fusion layer performs weighted fusion of forward and reverse features to obtain the EMG data features. .

5. The method for assessing limb nerve rehabilitation status based on multimodal sensor data fusion according to claim 1, characterized in that: The semantic text feature extraction module includes a text convolution layer, a first gated unit layer, a second gated unit layer, a first maximum pooling layer, a second maximum pooling layer and a feature splicing layer; the specific processing process is: S1, limb rehabilitation action text description data Input the text convolution layer to obtain local text features. The specific calculation formula is as follows: ； in, is the local feature obtained through the text convolution layer; is the text convolution kernel, is the text convolution kernel width, The number of input or output channels of the text convolution kernel; for Sequence at position of Dimension word vector window; is the bias term; Activation function for the text convolution layer; S2, will Sequentially input the first gated unit layer and the second gated unit layer to obtain global semantic features , capturing the temporal dependencies between sequence data and storing continuous limb rehabilitation action text state information in the hidden layer; S3, and Input the first maximum pooling layer and the second maximum pooling layer respectively to obtain low-dimensional local semantic features and low-dimensional global semantic features , the maximum pooling layer pair and Downsampling is performed to retain the most significant local features while reducing the number of feature parameters; S4, low-dimensional local semantic features and low-dimensional global semantic features The input features are concatenated to obtain text semantic features that represent local semantics and global semantic dependencies. , the specific calculation formula is as follows: = [CLS]+ +[SEP] + +[SEP]; Among them, [CLS] and [SEP] are sequence start and separation markers respectively.

6. The method for assessing limb nerve rehabilitation status based on multimodal sensor data fusion according to claim 1, characterized in that: The multimodal data feature fusion network includes a first feature segmentation layer, a second feature segmentation layer, a third feature segmentation layer, a rehabilitation action feature attention layer and a random inactivation layer; the specific processing process is: S1, the motion inertia data features EMG signal data characteristics and text semantic features Input the first feature segmentation layer, the second feature segmentation layer, and the third feature segmentation layer respectively, segment the features according to the limb nerve rehabilitation action category, and obtain the motion inertia data feature segmentation set , EMG signal data feature segmentation set and text semantic feature segmentation set ; S2, segment the motion inertia data features into a set , EMG signal data feature segmentation set and text semantic feature segmentation set Input the rehabilitation action feature attention layer, calculate the attention at different rehabilitation action levels, capture the complex interactive relationship between them, and perform weighted feature fusion. The specific calculation formula is as follows: ； ； in, For the The weight of each limb rehabilitation action, is the multimodal feature matrix , is the learnable weight matrix, is the multimodal fusion feature. represent Limb rehabilitation exercises.

7. The method for assessing limb nerve rehabilitation status based on multimodal sensor data fusion according to claim 1, characterized in that: The prediction output network consists of three fully connected layers and a Softmax activation function layer connected in sequence, which combines the multimodal fusion features output by the multimodal data feature fusion network. Input prediction output network obtains patient limb nerve rehabilitation status assessment results , the specific calculation formula is as follows: ； ； ； ； in, 、 、 and are the weight matrices of the first fully connected layer, the second fully connected layer, the third fully connected layer, and the Softmax activation function layer, respectively. 、 、 and They are the biases of the first fully connected layer, the second fully connected layer, the third fully connected layer and the Softmax activation function layer respectively.

8. The method for assessing limb nerve rehabilitation status based on multimodal sensor data fusion according to claim 1, characterized in that: The obtained training set and test set are used to train and test the multimodal data feature extraction network, the multimodal data feature fusion network and the prediction output network as a whole. The loss function adopts the cross entropy loss function. The specific calculation formula is as follows: ； in, This is the real condition of the patient's limb nerve rehabilitation. It is the patient's limb nerve rehabilitation status obtained by the prediction output network. is the training set size.

Images