Evaluation method of cerebral apoplexy rehabilitation evaluation model based on multi-mode electroencephalogram and myoelectricity fusion

Abstract

The invention belongs to the technical field of rehabilitation assessment, and provides a cerebral apoplexy rehabilitation assessment method and system based on multi-mode electroencephalogram and myoelectricity fusion. According to the method, 59-channel electroencephalogram signals and 14-channel electromyographic signals are preprocessed, mutual information between channels is calculated to construct a correlation matrix, spatial-temporal features are extracted in combination with a convolutional neural network (CNN) and a Transform self-attention mechanism, and rehabilitation level classification is achieved. According to the method, coherence between electroencephalogram / myoelectricity channels is quantified by adopting mutual information, and 59 * 59 and 14 * 14 dimensional feature matrixes are constructed; a CNN-Transform hybrid model is designed to optimize feature fusion, and the classification precision is improved; gradient weighted class activation mapping (Grad-CAM) is introduced to analyze feature saliency, and correlation between an electroencephalogram channel and a focus is revealed; and developing a visual evaluation system, and displaying the multi-modal data and the historical trend in real time. The system stores patient information and evaluation results through a MySQL database, and supports doctor-patient online interaction. Compared with traditional scale evaluation, the method has the advantages that the evaluation objectivity is improved by using multi-modal objective data, the model credibility is enhanced by combining interpretability analysis, and technical support is provided for formulating a personalized rehabilitation scheme.

Description

Technical Field

[0001] The present invention relates to the fields of multi-modal signal feature extraction and fusion, and rehabilitation function evaluation, and relates to a method for establishing a stroke rehabilitation evaluation model, a feature interpretation method and a system. Background Art

[0002] Stroke is a serious neurological disease that often causes functional impairments in aspects such as movement, sensation, language, and cognition in patients. Rehabilitation treatment is an important means to improve the quality of life of stroke patients and is crucial for improving the quality of life of patients. However, the rehabilitation process is complex and there are large individual differences, and effective evaluation means are needed to monitor and guide rehabilitation treatment. Traditional rehabilitation evaluation methods mainly rely on the subjective judgment of therapists and the use of standardized clinical evaluation scales, such as the Fugl-Meyer score, the Ueda Min rehabilitation assessment scale, the Barthel index, etc. However, these methods have great limitations: strong subjectivity, limited frequency, and insufficient objective data.

[0003] Multi-modal data fusion technology combines data from different sources such as electroencephalography (EEG) and electromyography (EMG), and can provide more comprehensive and accurate patient status information. For example, EEG signals can reflect the neural activities of the brain, and EMG signals can reflect the electrical activities of muscles. By combining the two, the motor ability of patients and the recovery of the neuromuscular system can be evaluated more comprehensively, which helps to more accurately evaluate the rehabilitation effect. Mutual information can be used to estimate the mutual dependence relationship between two time series and is very suitable for exploring the coherence characteristics between EEG and EMG signal channels. The application prospect of deep learning technology in the field of brain-computer interface (BCI) is broad. Through automatic feature extraction, pattern recognition and real-time processing, the performance and user experience of the BCI system have been significantly improved. Through deep learning algorithms, effective features can be extracted from complex multi-modal data, and a prediction model can be established to realize the automatic evaluation of the rehabilitation level.

[0004] Due to the transparency of machine learning and deep learning, it is particularly important to explore the physical meaning of feature values and explain the correlation between features and brain function states and lesions in combination with biomedical knowledge, which is more likely to improve the trust and acceptance of the model. More common methods include Gradient-weighted Class Activation Mapping (Grad-CAM) and SHAP (Shapley Additive exPlanations). Since doctors and patients cannot communicate in a timely manner, it is not conducive to the rehabilitation of patients. Nowadays, the network is developed. Establishing a database and presenting it in the form of software can not only help rehabilitation physicians intuitively understand the rehabilitation situation of patients, but also enable patients to adjust the intensity of rehabilitation exercises in a timely manner. Summary of the Invention

[0005] The present invention provides a method for multimodal electroencephalogram and electromyogram fusion, a method for establishing a stroke rehabilitation evaluation model, and a system, aiming to achieve the precision, universality, and simplicity of stroke rehabilitation evaluation, and to achieve accurate evaluation of the rehabilitation status of stroke patients and more friendly interaction with patients.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] On the one hand, the present invention provides a method for establishing a stroke rehabilitation evaluation model, which includes a multimodal signal feature extraction and fusion stage and a feature classification stage;

[0008] Step S1: Downsample the collected electroencephalogram signal to 250 Hz, remove the 50 Hz power frequency interference through an adaptive filter, perform band-pass filtering of 0.1 - 80 Hz, remove interference signals such as electrocardiogram, electrooculogram, and head movement based on independent component analysis, and remove the electromyogram signal in the electroencephalogram signal based on canonical correlation analysis. Segment according to the time stamps of different task segments and label them accordingly;

[0009] Step S2: Remove the DC bias from the collected electromyogram signal, perform band-pass filtering of 10 - 450 Hz, and perform full-wave rectification and downsampling of the electromyogram signal to 250 Hz; Obtain the time stamp of the corresponding task point of the electromyogram signal according to the sampling rate and time stamp of the synchronously collected electroencephalogram signal, and calculate as follows:

[0010]

[0011] In the formula, T emg,i represents the i-th time stamp of the electromyogram signal, S emg represents the sampling rate of the electromyogram signal, S eeg represents the sampling rate of the electroencephalogram signal, T eeg,i represents the i-th time stamp of the electroencephalogram signal.

[0012] Step S3: Calculate the mutual information of the preprocessed electroencephalogram signal and electromyogram signal respectively, calculate the mutual dependence relationship of the information between channels, and the information entropy of channel X is expressed as:

[0013]

[0014] In the formula, p n represents the probability density, and M represents different regional spaces;

[0015] The joint information entropy between two channels is expressed by the formula:

[0016]

[0017] where X n , X m represent two time series n and m respectively;

[0018] Then the mutual information MI between channel X and channel Y XY is expressed by the formula as:

[0019]

[0020] where H X represents the information entropy of channel X, H Y represents the information entropy of channel Y, H XY represents the joint information entropy between channels XY;

[0021] According to the mutual information, the channel correlation features are extracted to obtain two symmetric feature matrices P and Q with dimensions of 59×59 and 14×14;

[0022]

[0023]

[0024] where m(i,j) = m(j,i), n(i,j) = n(j,i), and i, j represent the rows and columns of the matrix respectively.

[0025] Step S4: The two feature matrices P and Q obtained in step S3 are respectively input into the convolutional neural network model to learn the spatio-temporal features between them and the correlation information features with the upper limb linkage task. Among them, the network of the electroencephalogram signal consists of three convolutional layers, two pooling layers and two batch normalization layers, and the network of the electromyogram signal consists of two one-dimensional convolutional layers and two pooling layers; the Flatten layer flattens the multi-dimensional tensors of the electroencephalogram and electromyogram signals output by the convolutional layer into two one-dimensional vectors, and then the two vectors are concatenated through the concat function to achieve the fusion at the feature level;

[0026] Step S5: The fused feature vector obtained in step S4 is input into the Transformer self-attention mechanism. Through the self-attention layer and the feed-forward neural network layer in the encoder and decoder, the model can adaptively focus on the important feature regions of the input signal, further improving the accuracy of the electroencephalogram and electromyogram signal feature extraction;

[0027] Step S6: The spatio-temporal feature data obtained in step S5 is input into the fully connected layer for classification. The fully connected layer uses the Softmax activation function, and the RG classifier is trained to calculate the classification accuracy rate, and the confusion matrix is drawn to visualize the classification effect of each category.

[0028] A further improvement of the present invention lies in combining the self-attention mechanism with the convolutional neural network. In step S4, a three-layer encoder-decoder structure is adopted. The encoder consists of a self-attention layer and a feed-forward neural network layer, and the decoder consists of a self-attention layer, an encoder-decoder attention layer, and a feed-forward neural network layer. It does not need to process the input sequence in order, so it can better handle long-distance dependencies.

[0029] Research method for the interpretability of eigenvalues:

[0030] Step S6, perform weighted summation on the EEG feature map of the convolutional layer in step S3 through gradient-weighted class activation mapping to generate a saliency map of the input image, highlighting the signal channels that the model focuses on when making decisions, revealing the relationship between the brain lesions and the channel signals. The calculation process is as follows:

[0031] Calculate the gradient of class c with respect to the feature map A k :

[0032] Perform global average pooling on the gradient to obtain the weight

[0033] Multiply the feature map A k by the weight and perform weighted summation to obtain the saliency map:

[0034] In the formula, A k represents the k-th feature map of the convolutional layer; y c is the output score of class c; Z is the product of the width and height of the feature map A k ; ReLU is the rectified linear unit, which is used to filter out the negative value part.

[0035] Find the channels in the corresponding EEG signals that have the greatest influence on the model according to the obtained feature saliency map, and further reveal the relationship between different channels and the brain function state and lesions.

[0036] Build a software for the stroke rehabilitation assessment system;

[0037] Step S7, construct a MySQL database containing the EEG and EMG data of stroke patients, the Fugl-Meyer upper limb motor function assessment scale and the corresponding model evaluation results, and the saliency map of the mapping relationship between features and brain lesions.

[0038] Step S8, use mysql-connector of Python to connect to the MySQL database and insert data through INSERT in SQL statements.

[0039] Step S9, use Visual Studio software to create the front-end pages based on the.NET Framework framework and the MVVM (Model-View-ViewModel) architecture, including pages such as user registration, login, rehabilitation panel, online consultation, etc. All use WPF forms and XAML language.

[0040] Step S10, establish the software background of the rehabilitation assessment system using the C# language, connect the database created in Step S8 through the MySql.Data.MySqlClient toolkit, and feedback it to the front-end page in the form of lists and waveform charts.

[0041] The positive effects of the technical solution of the present invention are as follows:

[0042] 1. The present invention extracts and fuses the features between channels of EEG and EMG signal features, constructs a rehabilitation assessment model, learns the spatio-temporal characteristics of the features, and obtains the rehabilitation assessment results of the patient. From the perspective of multi-modal information fusion, a comprehensive and objective assessment of the rehabilitation status of stroke patients is carried out, making the obtained data more scientific and accurate.

[0043] 2. The present invention studies the physical meaning of the eigenvalues, which helps to reveal the relationship between the EEG signal channels and the brain function state and lesions.

[0044] 3. The present invention can visualize the digital signal features, intuitively help the rehabilitation physician understand the patient's situation and timely change and formulate rehabilitation plans, improving the simplicity and universality of communication between doctors and patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a method flow chart of the stroke rehabilitation assessment model and system of multi-modal brain-muscle electrical fusion of the present invention

[0046] Figure 2 It is a framework diagram of the rehabilitation level assessment model in the embodiment of the present invention DETAILED DESCRIPTION OF THE INVENTION

[0047] The present invention will be further described in detail below with reference to the accompanying drawings in the embodiments:

[0048] See Figure 1 , which shows a flow chart of the establishment method and system display of a stroke rehabilitation level assessment model in the embodiment of the present invention. The method includes stages of feature extraction and fusion, significance analysis of features, and software construction of the visualization model;

[0049] Step S1, preprocess and segment the 59-channel EEG signals and 14-channel EMG signals collected from the stroke-upper limb linkage experiment, and the specific implementation is as follows:

[0050] Step S11, preprocessing EEG data. The EEG data is downsampled to 250Hz using the mne package provided by Python, 50Hz power frequency interference is removed by an adaptive filter, 0.1-80Hz bandpass filtering is performed, bad tracks are manually removed and bad tracks are differenced, interference signals such as electrocardiogram, electrooculogram, and head movement are removed based on independent component analysis, and myoelectric signals in EEG signals are removed based on canonical correlation analysis. The EEG signals are segmented according to the timestamps of different task segments and labeled accordingly to obtain multi-channel time series data of task-state EEG signals.

[0051] Step S12, preprocessing the EMG data. The pyemgpipeline package is used to remove the DC bias of the EMG signal, perform a 10-450Hz bandpass filter, and perform full-wave rectification and 250Hz downsampling on the EMG signal. The 14-channel time series data of the EMG signal corresponding to the task point is obtained according to the sampling rate and timestamp of the synchronously collected EEG signal, and the calculation is as follows:

[0052]

[0053] Where T emg,i represents the i-th timestamp of the electromyographic signal, S emg Represents the sampling rate of the electromyographic signal, S eeg represents the sampling rate of EEG signal, T eeg,i Represents the i-th timestamp of the EEG signal.

[0054] Step S2, calculate the mutual information of the preprocessed EEG signal and EMG signal, calculate the interdependence of the information between channels, and the information entropy of channel X It is expressed as:

[0055]

[0056] In the formula, p n represents probability density, and M represents different regional spaces;

[0057] Joint information entropy between two channels The formula is:

[0058]

[0059] Where X n , X m Represent two time series n and m respectively;

[0060] Then the mutual information MI between channel X and channel Y is XY The formula is:

[0061] MI XY =H X +H Y-H XY ,

[0062] where H X represents the information entropy of channel X, and H Y represents the information entropy of channel Y, and H XY represents the joint information entropy between channels XY;

[0063] Extract the channel correlation features according to the mutual information, and obtain two feature matrices P and Q with dimensions of 59×59 and 14×14;

[0064]

[0065]

[0066] where m(i,j) = m(j,i), n(i,j) = n(j,i), and i and j represent the rows and columns of the matrix respectively.

[0067] Step S3, Send the two feature matrices P and Q obtained in step S2 into the convolutional neural network model to learn the spatio-temporal features between them. The Flatten layer flattens the multi-dimensional tensors of the EEG and EMG signals output by the convolutional layer into two one-dimensional vectors, and then the two vectors are concatenated through the concat function to achieve the fusion at the feature level.

[0068] Step S31, The convolutional neural network model of the EEG signal includes a temporal convolutional layer, a depth convolutional layer, a point convolutional layer, two batch normalization layers, and two max pooling layers. The size of the convolutional kernel is 3, the size of the pooling kernel is 2, and the ELU activation function is used. The calculation formula is as follows:

[0069]

[0070] where x represents the vector output after convolution; α is a hyperparameter with a default value of 1.

[0071] Step S32, The convolutional neural network model of the EMG signal includes two one-dimensional convolutional layers, a batch normalization layer, and two max pooling layers, and the RELU activation function is used. The calculation formula is as follows:

[0072] f(x) = max(0, x),

[0073] x is the weighted sum output of a certain layer of the neural network.

[0074] Step S33, Concatenate the one-dimensional feature vector of the EEG signal and the one-dimensional feature vector of the EMG signal flattened by the Flatten layer through the concat function to form a new fused feature vector.

[0075] Step S4: Input the fused feature vectors obtained in step S33 into the Transformer self-attention mechanism to further learn the spatio-temporal features of the fused data, and then input them into the fully connected layer for classification. The fully connected layer uses the Softmax activation function, and the calculation formula is as follows:

[0076]

[0077] where \(x\) represents the input vector, which is the output of the neural network; \(x_{i}\) i represents the \(i\)-th element in the input vector \(x\), \(s_{i}\) represents the score of the \(i\)-th class; \(K\) represents the length of the input vector, that is, the number of classes in the classification task. The output of the softmax function, \(p_{i}\), represents the probability of the \(i\)-th class.

[0078] Step S41: The Transformer consists of three groups of encoder-decoders. Each encoder is composed of a self-attention layer and a feed-forward neural network layer, and the decoder is composed of a self-attention layer, an encoder-decoder attention layer, and a feed-forward neural network layer.

[0079] Step S5: Perform weighted summation on the EEG feature maps of the convolutional layer in step S3 through gradient-weighted class activation mapping to generate a saliency map of the input image, highlighting the signal channels that the model focuses on when making decisions, and revealing the relationship between brain lesions and channel signals. The calculation process is as follows:

[0080] Calculate the gradient of class \(c\) with respect to the feature map \(A\) k :

[0081] Perform global average pooling on the gradient to obtain the weight

[0082] Multiply the feature map \(A\) k by the weight and perform weighted summation to obtain the saliency map:

[0083] In the formula, \(A_{k}\) k represents the \(k\)-th feature map of the convolutional layer; \(y_{c}\) c is the output score of class \(c\); \(Z\) is the product of the width and height of the feature map \(A\) k ; ReLU is the rectified linear unit, which is used to filter out the negative part.

[0084] Step S6: Build a stroke rehabilitation evaluation database and software system.

[0085] Step S61: In MySQL, use the Creat statement to create tables EEG, EMG, FMI, and Feature, which respectively correspond to the electroencephalogram data, electromyogram data, model evaluation results, and the significant relationship between the corresponding model evaluation results and features and the mapping relationship of brain lesions. The identity information of the patient is used as the primary key in each table, and the corresponding foreign key relationships are set.

[0086] Step S62: Use mysql-connector in Python to connect to the MySQL database and insert data through the INSERT statement in the SQL statement.

[0087] Step S63: Use Visual Studio software to create the front-end pages based on the.NET Framework framework and the MVVM (Model-View-ViewModel) architecture, including user registration, login, rehabilitation panel, online consultation, etc. pages, and uniformly adopt WPF forms and XAML language.

[0088] Step S64: Use the C# language to establish the software background of the rehabilitation evaluation system, connect the database created in Step S61 through the MySql.Data.MySqlClient toolkit, and feedback it to the front-end page in the form of lists and waveform diagrams.

[0089] The above examples are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal brain and myoelectric fusion stroke rehabilitation assessment model evaluation method, characterized in that The following steps are involved: Step 1: Preprocess EEG and EMG signals: The 59-channel EEG signal was downsampled to 250Hz, and the 50Hz power frequency interference was removed by an adaptive filter. A 0.1-80Hz bandpass filter was performed. The ECG, EOG and head movement interference signals were removed based on independent component analysis. The myoelectric interference in the EEG signal was removed by canonical correlation analysis. The EEG signal was segmented according to the timestamps of different task segments and labeled accordingly. The DC bias of the 14-channel EMG signal was removed, and 10-450Hz bandpass filtering, full-wave rectification and down-sampling to 250Hz were performed. The timestamp of the EMG signal corresponding to the task point was calculated according to the sampling rate and timestamp of the synchronously collected EEG signal. The formula is: Where T emg,i represents the i-th timestamp of the electromyographic signal, S emg Represents the sampling rate of the electromyographic signal, S eeg represents the sampling rate of EEG signal, T eeg,i Represents the i-th timestamp of the EEG signal; Step 2: Calculate the mutual information between EEG and EMG channels and extract correlation features: Calculate the information entropy H of channel X x : Among them, n represents different n regional spaces, p i Represents the probability density of the i-th region among these n regions; Calculate the joint information entropy H of the two channels X and Y X,Y , the formula is as follows: Mutual information MI between channel X and channel Y XY The calculation formula is: Among them, H X represents the information entropy of channel X, H Y represents the information entropy of channel Y, H XY represents the joint information entropy between channels XY, and p(i,j) represents the marginal probability density of X=i, Y=j; Based on the mutual information, the correlation features between EEG channels are extracted to generate a 59×59 dimensional symmetric feature matrix P, whose elements satisfy m(i,j)=m(j,i)m(i,j)=m(j,i), and the matrix form is: Based on the mutual information, the correlation features between the electromyographic channels are extracted to generate a 14×14 dimensional symmetric feature matrix Q, whose elements satisfy n(i,j)=n(j,i)n(i,j)=n(j,i), and the matrix form is: Step 3: Fusion of multimodal features: The two feature matrices P and Q obtained in step 2 are respectively sent to the convolutional neural network model to learn the spatial-temporal features between them. The multi-dimensional tensors of the EEG and EMG signals output by the convolutional layer are flattened into two one-dimensional vectors by the Flatten layer, and then the two vectors are concatenated by the concat function to achieve feature-level fusion; The convolutional neural network model of EEG signals includes a temporal convolution layer, a depth convolution layer, a point convolution layer, two batch normalization layers and two maximum pooling layers. The convolution kernel size is 3, the pooling kernel size is 2, and the ELU activation function is used. The calculation formula is as follows: In the above formula, x represents the vector output after convolution; α is a hyperparameter with a default value of 1. The convolutional neural network model of electromyographic signals includes two one-dimensional convolutional layers, a batch normalization layer, and two maximum pooling layers. The RELU activation function is used, and the calculation formula is as follows: f(x)=max(0,x), x is the weighted sum output of a layer of the neural network; The one-dimensional feature vector of the EEG signal and the one-dimensional feature vector of the EMG signal flattened by the Flatten layer are concatenated together through the concat function to form a new fused feature vector; Step 4: Optimize features through Transformer self-attention mechanism: A three-layer encoder-decoder structure is adopted. The encoder consists of a multi-head self-attention layer with 8 heads and a feedforward neural network layer, and the decoder consists of a self-attention layer, an encoder-decoder attention layer, and a feedforward neural network layer. The obtained fusion feature vector is input into the Transformer self-attention mechanism to further learn the temporal-spatial features of the fusion data, and then input into the fully connected layer for classification. The fully connected layer uses the Softmax activation function, and the calculation formula is as follows: Where x represents the input vector, which represents the output of the neural network; i represents the th element in the input vector x, which represents the score of the i-th category; K represents the length of the input vector, that is, the number of categories in the classification task; the output of the softmax function represents the probability of the i-th category; Step 5: Feature interpretability analysis: The EEG feature map of the convolutional layer in step S3 is weighted and summed by gradient weighted class activation mapping to generate a saliency map of the input image, highlighting the signal channels that the model focuses on when making decisions and revealing the relationship between brain lesions and channel signals. The calculation process is as follows: Calculate category c relative to feature map A F The gradient is: Perform global average pooling on the gradient to get the weight The feature map A F With weight Weighted summation gives the saliency map: In the formula, A F represents the kth feature map of the convolutional layer; y K is the output score of category c; Z is the feature map A F The product of the width and height of ; ReLU is a rectified linear unit, which is used to filter out negative values; Step 6: Build a stroke rehabilitation assessment system: In MySQL, the tables EEG, EMG, FMI, and Feature are created through the Create statement to correspond to the EEG data, EMG data, model evaluation results, and the significance relationship between the corresponding model evaluation results and the mapping relationship between features and brain lesions. In each table, the patient's identity information is the primary key and the corresponding foreign key relationship is set; Use mysql-connector in Python to connect to the MySQL database and insert data through INSERT in the SQL statement; Use Visual Studio software to create front-end pages based on the .NET Framework and MVVM (Model-View-ViewModel) architecture, including user registration, login, rehabilitation panel, online consultation and other pages, uniformly using WPF forms and XAML language; The C# language is used to establish the software background of the rehabilitation assessment system. The database created in the background is connected through the MySql.Data.MySqlClient toolkit and fed back to the front-end page in the form of lists and waveform graphs.

Images