A method for recognizing motor imagery gestures based on electroencephalogram and electromyogram multimodal fusion

Abstract

The present invention discloses a method for recognizing motor imagery gestures based on the multimodal fusion of electroencephalogram (EEG) and electromyogram (EMG). It uses FBCSP to extract features from EEG. First, a filter bank is used to decompose the filtered EEG signals to extract signal features in different frequency bands. For each frequency band, the common spatial pattern analysis method is adopted. By finding a pair of projection matrices, the signals are projected from the original space to a new space, so that the variance of different categories of EEG signals in the new space is maximized or minimized. The EMG signals of two channels, namely the anterior group and the posterior group of the forearm muscles, are extracted. After preprocessing the signals, the mean values, ranges and the mean ratio of the two channels are extracted respectively. Finally, the EEG features and EMG features are concatenated and fused, and an SVR algorithm is used to train a model and deploy it on the server. The present invention makes full use of the complementarity of information between the multi-bioelectric signals, namely EMG and EEG, and significantly improves the classification effect of motor imagery recognition.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer recognition, and relates to a method for recognizing motor imagery gestures based on electroencephalogram and electromyogram multimodal fusion. Background Art

[0002] Brain-Machine Interface (BMI) technology is an exciting field that combines neuroscience, engineering, and computer science, aiming to establish a direct communication channel between the human brain and external devices. The development of brain-computer interfaces has great potential in the field of medical rehabilitation. This article will focus on an important research direction in the field of brain-computer rehabilitation, namely the motor imagery classification algorithm for electroencephalogram and surface electromyogram multimodal fusion.

[0003] Stroke is a common and serious neurological disease, usually caused by an interruption of blood flow to the brain, resulting in damage or death of brain cells. Stroke can lead to muscle function loss or impaired motor intention. Considering that non-invasive electroencephalogram signals are easily interfered by the scalp, hair, and tissues, there are limitations in terms of accuracy and information content, and it is impossible to deeply record deeper neural activities. Therefore, traditional brain-computer upper limb rehabilitation systems often only perform binary classification for the left and right hands, which has a certain effect on stroke recovery. However, to promote further recovery of the nervous system, more refined motor imagery is required, such as different gesture movements of the same hand.

[0004] For healthy people without any neuromuscular system diseases, when taking electromyograms at the positions of the extensor pollicis brevis and the extensor indicis proprius near the wrist joint and the extensor digitorum communis near the elbow joint on the back of the forearm, the accuracy rate of 16-classification of multi-finger movements can reach over 80%. In stroke patients, the changes in surface electromyograms may be affected by the damaged nervous system. Since stroke may cause neuron damage or loss of function, the electromyogram changes during motor imagery in patients may be limited. However, some studies have shown that even in stroke patients, weak changes in muscle electromyograms may still exist. This may be because some neural pathways can still transmit control signals of motor imagery, although they are affected to a certain extent. Based on this, this article proposes a method for classifying motor imagery by fusing electroencephalogram and surface electromyogram, and classifies according to the weak electromyograms of stroke patients to assist the electroencephalogram signals of motor imagery, so as to improve the accuracy rate of multi-gesture classification. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies and improvement needs of existing hand rehabilitation devices.

[0006] In a first aspect, the working process of a method for recognizing motor imagery gestures based on electroencephalogram and electromyogram multimodal fusion of the present invention is as follows:

[0007] Step 1: Collect electroencephalogram (EEG) data and electromyogram (EMG) data to obtain a dataset.

[0008] Step 2: Preprocess the EEG data and EMG data.

[0009] Step 3: Extract features from the EEG data and EMG data.

[0010] Feature extraction from the EEG data includes decomposition into multiple frequency bands through a filter and common spatial pattern feature extraction for the EEG signals in each frequency band. Feature extraction from the EMG data includes extracting the mean, range, and mean ratio of the EMG data. The obtained EEG features are concatenated with the EMG feature X sEMG to obtain the feature data X j .

[0011] Step 4: Construct a classification model and use the feature data X obtained in Step 3 j to train the classification model.

[0012] Step 5: During the process of the subject performing motor imagery, collect EEG data and EMG data of the subject; the obtained EEG data and EMG data are preprocessed and the features obtained after feature extraction are concatenated and then input into the classification model for recognition, and the classification model outputs the type of motor imagery of the subject.

[0013] Preferably, for any frequency band k, the specific process of extracting CSP features is as follows:

[0014] Construct the covariance matrix of the i-th sample as follows:

[0015]

[0016] Construct the average covariance matrix of the j-th class as:

[0017]

[0018] where is the number of samples, and tr(·) is the trace of the matrix

[0019] Preferably, the EMG feature X sEMG is collected in Step 3 as follows:

[0020]

[0021] where X sEMGi is the EMG feature of the i-th sample; mean(·) is the operation of taking the mean; max(·) is the operation of taking the maximum value; min(·) is the operation of taking the minimum value; X' i,0 、X' i,1They are the EMG data of the two channels of the i-th sample respectively.

[0022] Preferably, in step 3, the EEG features are collected as follows: for each movement j, a corresponding binary classification model is established respectively; each binary classification model includes a spatial filter for feature extraction The spatial filter corresponding to each motor imagery type Performs feature extraction on the preprocessed EEG data to obtain EEG features; the EEG features corresponding to each motor imagery type are respectively concatenated with the EMG features, and a corresponding training set is obtained for each motor imagery type.

[0023] Preferably, the classification model described in step 4 uses an SVR model; the number of motor imagery types is s; s≥3; each motor imagery type corresponds to an SVR model. Each SVR model is trained using the training set corresponding to its motor imagery type.

[0024] Preferably, the spatial filter is constructed And the process of performing feature extraction on the EEG data is as follows:

[0025] (1) Process the labels of each sample in the dataset into binary classification labels corresponding to each binary classification model.

[0026] (2) Decompose the EEG signals in step 2 into different frequency bands:

[0027] (3) Extract CSP features from the data of each frequency band respectively;

[0028] (4) Construct a composite covariance matrix And perform eigenvalue decomposition to obtain the whitening matrix P:

[0029] P = (Λc) -1 / 2 (U c ) T

[0030] where U c is the eigenvector matrix, and Λ c is the diagonal matrix composed of eigenvalues.

[0031] (5) Construct the initial spatial filter W as follows:

[0032] W = P T U

[0033] (6) Select the first 2 and the last 2 spatial filters in the initial spatial filter W to form the spatial filter of frequency band k

[0034]

[0035] Combining the filters of all frequency bands, the final spatial filter for the j-th class is And using the spatial filter Construct electroencephalogram features as follows:

[0036]

[0037] where diag(·) is to process the diagonal elements. X is the electroencephalogram signal.

[0038] Preferably, the electroencephalogram signal is decomposed into different frequency bands by a Chebyshev type II filter.

[0039] Preferably, there are three types of the above-mentioned motor imagery, namely rest, grasping movement and stretching movement.

[0040] Preferably, the electroencephalogram acquisition device used in step 1 has 24 channels and a sampling rate of 300 Hz. The electromyogram data is collected from two channels, corresponding to the anterior group (flexor) and posterior group (extensor) of the forearm muscles respectively, with a sampling rate of 1000 Hz.

[0041] Preferably, in step 2, the preprocessing process of the electroencephalogram data is as follows: select three channels C3, CZ, and C4, first downsample to 250 Hz, and then perform band-pass filtering from 3 to 48 Hz. The filter is a fir filter.

[0042] Preferably, in step 2, the preprocessing process of the electromyogram data is as follows: first downsample to 250 Hz, and then process it using a sliding window. The window size is 10 and the sliding step is 1. Mean normalization is performed within each sliding window; then, the data obtained from the first sliding window processing is continued to be processed using a sliding window. The window size is 20 and the sliding step is 1. Calculate the mean value within each sliding window; if the obtained mean value is greater than the threshold, retain the signal of the corresponding sampling point, otherwise the signal of this sampling point is 0.

[0043] In a second aspect, the present invention provides a hand rehabilitation device, including an electroencephalogram acquisition device, an electromyogram acquisition device, a classifier, and a hand rehabilitation device. The electroencephalogram acquisition device is used to collect the electroencephalogram data of the user; the electromyogram acquisition device is used to collect the electromyogram data of the user; the classifier is used to identify the motor imagery type of the user through the electroencephalogram data and the electromyogram data by using the aforementioned motor imagery gesture recognition method. The hand rehabilitation device is used to drive the hand of the user to perform a movement consistent with the recognition result of the classifier.

[0044] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the memory stores the computer program; the processor executes the aforementioned method for recognizing motor imagery gestures.

[0045] In a fourth aspect, the present invention provides a readable storage medium storing a computer program; when the computer program is executed by a processor, it is used to implement the aforementioned method for recognizing motor imagery gestures.

[0046] The beneficial effects of the present invention are as follows:

[0047] 1. The present invention makes full use of the complementarity of information between multi-bioelectrical signals, namely electromyogram and electroencephalogram. Compared with using only electroencephalogram or only electromyogram, the classification effect of the present invention for motor imagery recognition based on multi-modal fusion is significantly improved.

[0048] 2. The present invention uses FBCSP to extract features from electroencephalogram. First, a filter bank is used to decompose the filtered electroencephalogram signal (EEG) to extract signal features in different frequency bands. For each frequency band, a common spatial pattern analysis method is adopted. By finding a pair of projection matrices, the signal is projected from the original space to a new space, so that the variance of electroencephalogram signals of different classes is maximized or minimized in the new space. In this way, the CSP method can extract features that can distinguish different classes.

[0049] 3. The present invention proposes a motor imagery (MI) brain-computer interface based on a multi-modal fusion algorithm of electroencephalogram (EEG) and surface electromyogram (sEMG), and applies it to a hand rehabilitation device; during rehabilitation, the user needs to perform motor imagery of grasping or stretching the palm according to the prompt. After the system acquires the electroencephalogram (EEG) and electromyogram (sEMG) signals, they are sent to the model for processing to obtain the classification result and at the same time control the user's rehabilitation glove to perform corresponding rehabilitation actions, enabling the user to actively participate in rehabilitation instead of simply and passively completing the rehabilitation actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is the working flow chart of the present invention;

[0051] Figure 2 is the schematic diagram of the process of collecting electroencephalogram and electromyogram data once in step 1 of the present invention; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] The following will describe the present invention in detail with reference to the accompanying drawings.

[0053] As Figure 1 shown, the working flow of a method for recognizing motor imagery gestures based on multi-modal fusion of electroencephalogram and electromyogram is as follows:

[0054] Step 1: Acquisition of EEG data and EMG data

[0055] First, the subject wears the EEG and EMG acquisition devices. The EEG acquisition device has a total of 24 channels with a sampling rate of 300 Hz. The two channel positions of the EMG are respectively in the anterior group and posterior group of the forearm muscles, with a sampling rate of 1000 Hz. As Figure 2 shown, the subject prepares for motor imagery according to the system prompt for 0 - 3 seconds, performs motor imagery for 3 - 6 seconds, and rests for 6 - 8 seconds. Classification label data will also be recorded during the experiment, with a total of 3 actions: rest, stretch, and grasp.

[0056] Step 2: Data preprocessing

[0057] The EEG preprocessing process is as follows: Select three channels C3, CZ, and C4, first downsample to 250 Hz, and then perform band - pass filtering from 3 to 48 Hz. The filter is a fir filter.

[0058] The sEMG preprocessing process is as follows: First downsample to 250 Hz, then use a sliding window for processing. The window size is 10 and the sliding step is 1. Mean normalization is performed within each sliding window; then, the data obtained from the first sliding window processing is continued to be processed with a sliding window. The window size is 20 and the sliding step is 1, and the mean within each sliding window is calculated; if the obtained mean is greater than the threshold, the corresponding sampling point signal is retained, otherwise the sampling point signal is 0.

[0059] For the preprocessed EEG data and sEMG data, the data of 3 - 6 seconds of a single sampling is intercepted for subsequent feature extraction and training.

[0060] Step 3: Extract features of EEG and sEMG data

[0061] 3 - 1. EEG feature acquisition.

[0062] Since the CSP algorithm is for binary classification tasks, for any action j, a binary classification model is trained. At the final decision, the three classification results are compared to obtain the final classification result; each of the three binary classification models contains a spatial filter for feature extraction

[0063] The training process of the binary classification model corresponding to any action j is as follows:

[0064] 3 - 1 - 1. Process the action label into 0, 1 label: y j = eq(y true , j); when a = b, eq(a, b)=1, otherwise eq(a, b)=0.

[0065] Decompose the EEG signal in step 2 into different frequency bands using a Chebyshev type II filter:

[0066]

[0067] where, is the Chebyshev type II filter; high and low are the passband edge frequencies of the band-pass filter. i = 1, 2, 3,..., m. m is the number of frequency bands, which is 9 in this embodiment; the size of the data fb_x obtained after filtering each channel is: (n_bands, n_trails, n_channels, n_times). n_bands is the number of frequency bands, n_trails is the number of experimental samples, n_channels is the number of EEG channels, and n_times is the number of samples per sample.

[0068] 3-1-2. Extract CSP features from the data of each frequency band; for the data of any frequency band k N t is the total number of samples, N c is the total number of channels, N s is the number of sampling points.

[0069] For any frequency band k, the specific process of extracting CSP features is as follows:

[0070] Construct the covariance matrix of the i-th sample as:

[0071]

[0072] Construct the average covariance matrix of the j-th class as:

[0073]

[0074] where, is the set of sample indices labeled j, is the number of samples, and tr(·) is the trace of the matrix.

[0075] Next, construct the composite covariance matrix and perform eigenvalue decomposition to construct the whitening matrix P:

[0076]

[0077] P = (Λ c ) -1 / 2 (U c ) T

[0078] Among them, U c is the eigenvector matrix, and A c is the diagonal matrix composed of eigenvalues. P is the whitening matrix, such that holds.

[0079]

[0080] Among them, I is the identity matrix. S 1 , S 2 are two intermediate variables.

[0081] Perform eigenvalue decomposition on S 1 , S 2 to obtain the initial spatial filter W:

[0082] S 1 = UΛ 1 (U) T

[0083] S 2 = UΛ 2 (U) T

[0084] W = P T U

[0085] Select the first 2 and the last 2 spatial filters in the initial spatial filter W to form the spatial filter for frequency band k

[0086]

[0087] Combine the filters of all frequency bands to obtain the final spatial filter for the j-th class as And use the spatial filter to construct the EEG features as follows:

[0088]

[0089] Among them, diag(·) is the operation of taking diagonal elements.

[0090] 3-2. EMG feature acquisition.

[0091] For sEMG data N t is the total number of samples, N c is the total number of channels, and N s is the number of sampling points.

[0092] For sample i, the collected EMG features X sEMGi are as follows:

[0093]

[0094] Among them, mean(·) is the operation of taking the mean value; max(·) is the operation of taking the maximum value; min(·) is the operation of taking the minimum value; V' i,0 and X' i,1 are the EMG data of two channels of the i-th sample respectively.

[0095] 3-3. Concatenate the extracted EEG features with the EMG feature X sEMg to obtain the feature data X j , which is used as the training set of class j.

[0096] Step 4: Train a classification model and deploy it on the server

[0097] Using the training sets corresponding to the three classes, train three binary classifiers SVR_CLS j (·) respectively by using the SVR classification algorithm. Deploy the obtained three binary classifiers SVR_CLS j (·) on the backend server.

[0098] Step 5: Conduct online rehabilitation training

[0099] The subject wears a rehabilitation glove, an EEG cap and an EMG acquisition device. The two channels of the EMG acquisition device are worn on the anterior group and the posterior group of the forearm muscles respectively and are connected to the server. The front-end system starts the rehabilitation training and prompts the subject to perform motor imagery of grasping or stretching. The single motor imagery is 8 seconds. After the server receives 8 seconds of data, it first preprocesses the data using the method in Step 2, and then uses the three spatial filters obtained in Step 3 to extract features from the obtained EEG data respectively, obtaining three EEG features, and using the method in Step 3-2 to collect EMG features;

[0100] Concatenate the EMG features with the three EEG features respectively to obtain three classification features; input the three classification features into the corresponding three binary classifiers SVR_CLS j (·) respectively, and obtain the output results res = [res1, res2, res3] of the three binary classifiers SVR_CLS j (·); the prediction result is argmin(res). argmin(·) is the index that returns the minimum value.

[0101] The server sends a grasping or stretching instruction to the rehabilitation glove according to the prediction result to help the patient perform hand movements. After the action is completed, the glove returns to the relaxed state and the next action is performed.

[0102] In the hand rehabilitation for stroke patients, the present invention makes full use of the complementarity of information between multi-bioelectric signals, i.e., electromyogram and electroencephalogram. Compared with using only electroencephalogram or only electromyogram, the classification effect of multi-modal fusion has been significantly improved.

[0103] In this embodiment, the average accuracy rate of motor imagery classification after fusing electroencephalogram and electromyogram features is 96%; in contrast, when using only electroencephalogram, the average accuracy rate of 3-classification is 45%; when using only electromyogram, the average accuracy rate of 3-classification is 80%. This shows that the motor imagery recognition method provided by the present invention helps to improve the accuracy rate of motor imagery recognition.

[0104] In some embodiments, the recognition method provided by the present invention is used to recognize the type of motor imagery of a user when the user is training through an assisted movement device, so that the assisted movement device performs corresponding actions according to the recognition result of the motor imagery type, promoting the user to actively participate in the rehabilitation training, rather than passively accepting the movement of the training instrument. This is more conducive to reconstructing the user's motor nerves, giving the user a stronger sense of participation in the rehabilitation training, and facilitating the continuous progress of the rehabilitation training process.

Claims

1. A method for motion imagination gesture recognition based on electroencephalogram and electromyogram multimodal fusion, characterized in that: It includes the following steps: Step 1: Collect electroencephalogram (EEG) data and electromyogram (EMG) data to obtain a dataset; two channels of EMG data are collected, corresponding to the anterior and posterior groups of forearm muscles respectively; Step 2: Preprocess the EEG data and EMG data; Step 3: Extract features from the EEG data and EMG data; The feature extraction of the EEG data includes decomposing it into multiple frequency bands through a filter, and extracting common spatial pattern (CSP) features for the EEG signals in each frequency band; the feature extraction of the EMG data includes extracting the mean, range, and mean ratio of the EMG data; the mean ratio is the ratio of the means of the EMG data of the two channels; The obtained EEG features are concatenated with the EMG feature X sEMG to obtain the feature data X j ; The acquisition of EEG features is as follows: for each action j, a corresponding binary classification model is established respectively; each binary classification model includes a spatial filter for feature extraction The spatial filter corresponding to each type of motor imagery All perform feature extraction on the preprocessed EEG data to obtain EEG features; the EEG features corresponding to each type of motor imagery are respectively concatenated with EMG features, and a corresponding training set is obtained for each type of motor imagery; Step 4: Construct a classification model and use the feature data X obtained in Step 3 j Train the classification model; the classification model uses an SVR model; the number of motor imagery types is s; s≥3; each motor imagery type corresponds to an SVR model; each SVR model is trained using the training set corresponding to its motor imagery type; Step 5: During the process of the subject performing motor imagery, collect the EEG data and EMG data of the subject; the obtained EEG data and EMG data are input into a classification model for recognition after the features obtained through preprocessing and feature extraction are spliced, and the classification model outputs the motor imagery type of the subject.

2. The method for recognizing motor imagery gestures based on electroencephalogram and electromyogram multimodal fusion according to claim 1, wherein: For any frequency band k, the specific process of extracting CSP features is as follows: Construct the covariance matrix of the i-th sample as follows: Construct the average covariance matrix of the jth class as: wherein, is the number of samples, and tr(·) is the trace of a matrix Collect the myoelectric feature X in step 3 sEMG as follows: Among them, X sEMGi is the electromyogram feature of the i-th sample; mean(·) is the operation of taking the mean value; max(·) is the operation of taking the maximum value; min(·) is the operation of taking the minimum value; X' i,0 , X' i,1 are the electromyogram data of two channels of the i-th sample respectively.

3. A method for recognizing motor imagery gestures based on electroencephalogram and electromyogram multimodal fusion according to claim 1, characterized in that: Construct a spatial filter The process of extracting features from EEG data is as follows: (1) Process the labels of each sample in the dataset into binary labels corresponding to each binary classification model; (2) Decompose the EEG signals in Step 2 into different frequency bands: (3) Extract CSP features for the data in each frequency band respectively; (4) Construct a composite covariance matrix and perform eigenvalue decomposition to obtain the whitening matrix P: P = (Λ c ) -1 / 2 (U c ) T Among them, U c is the eigenvector matrix, and Λ c is the diagonal matrix composed of eigenvalues; (5) Construct the initial spatial filter W as follows: W = P T U (6) Select the first two and the last two spatial filters in the initial spatial filter W to form the spatial filter of frequency band k Combining the filters of all frequency bands, the final spatial filter for the j-th class is and using the spatial filter to construct the EEG features as follows: where, diag(·) is the process of taking diagonal elements; X is the EEG signal.

4. A method for recognizing motor imagery gestures based on electroencephalogram and electromyogram multimodal fusion according to claim 1, characterized in that: There are three types of the motor imagery types, namely rest, grasping movement, and stretching movement.

5. A method for recognizing motor imagery gestures based on electroencephalogram and electromyogram multimodal fusion according to claim 1, characterized in that: In Step 2, the preprocessing process of the EEG data is as follows: Select three channels of C3, CZ, and C4, first downsample to 250 Hz, then perform band-pass filtering from 3 to 48 Hz, and the filter is a fir filter; first downsample to 250 Hz, then process it using a sliding window, the window size is 10, the sliding step is 1, and mean normalization is performed within each sliding window; then, continue to process the data obtained from the first sliding window processing using a sliding window, the window size is 20, the sliding step is 1, and calculate the mean within each sliding window; if the obtained mean is greater than the threshold, retain the signal of the corresponding sampling point, otherwise the signal of this sampling point is 0.

6. A hand rehabilitation device, comprising an electroencephalogram acquisition device, a classifier, and a hand rehabilitation device; characterized in that: It also includes an EMG acquisition device; the EEG acquisition device is used to collect the EEG data of the user; the EMG acquisition device is used to collect the EMG data of the user; the classifier is used to identify the motor imagery type of the user through the EEG data and EMG data by using a method for recognizing motor imagery gestures based on multi-modal fusion of EEG and EMG as described in any one of claims 1-5; the hand rehabilitation device is used to drive the hand of the user to perform movements consistent with the recognition result of the classifier.

7. A computer device, characterized in that: It includes a memory, a processor, and a computer program stored on the memory and executable on the processor, the memory stores the computer program; the processor executes a method for recognizing motor imagery gestures based on multi-modal fusion of EEG and EMG as described in any one of claims 1-5.

8. A readable storage medium stores a computer program; characterized in that: When the computer program is executed by the processor, it is used to implement a method for recognizing motor imagery gestures based on multi-modal fusion of EEG and EMG as described in any one of claims 1-5.

Images