A motor imagery electroencephalogram signal classification method and system based on deep learning and a storage medium

Abstract

A kind of motor imagination electroencephalogram signal classification method, system and storage medium based on deep learning, the present application belongs to the field of brain-computer interface, specifically related to motor imagination electroencephalogram signal classification method, system and storage medium based on deep learning.The purpose of the present application is to solve the problem of low motor imagination electroencephalogram signal classification recognition accuracy caused by low signal-to-noise ratio of electroencephalogram signal.Process is: obtaining two-dimensional original electroencephalogram signal feature vector;Signal feature vector is selected by channel;EEG-TCN-LSTM convolutional neural network is constructed;Obtain trained EEG-TCN-LSTM;Obtain the two-dimensional electroencephalogram signal feature vector to be tested of subject, the two-dimensional original electroencephalogram signal feature vector is selected by channel, obtain new feature vector set, two-dimensional vector is input into trained EEG-TCN-LSTM, and trained EEG-TCN-LSTM outputs the prediction probability of each motor imagination category.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer interface, and specifically relates to a motor imagery EEG signal classification method, system and storage medium based on deep learning. Background Art

[0002] As we all know, the brain is the most complex biological organ in the human body, supporting a variety of complex human functions such as cognition, perception, emotion, memory, and movement, and playing a vital role in physiological and psychological processes. Research on brain science is also increasing, with the brain-computer interface (BCI) being a key area of ​​brain science and brain-inspired research. BCI refers to a new type of real-time communication and control system established between the biological (human or animal) brain and external devices or the environment, thereby enabling direct interaction between the brain and external devices.

[0003] After nearly 50 years of research, brain-computer interface (BCI) technology has continued to develop and improve. Research directions have gradually become distinct based on the EEG signal acquisition method and signal type. Among them, BCI systems based on motor imagery (MI) EEG signals have been a hot topic of research. Motor imagery is a spontaneous EEG signal that can be acquired using electroencephalogram (EEG) equipment, making it simple and easy to implement. Subjects spontaneously imagine certain movements based on on-screen prompts without actually performing them. During this process, the brain's motor cortex produces specific EEG signals with relatively high amplitudes, known as motor imagery EEG signals. The study of motor imagery EEG signals is of great significance in fields such as neurorehabilitation and BCI. By decoding the subject's motor intentions and converting the recognition results into commands to control external devices, "mind control" can be achieved, offering a novel treatment option for patients with limited mobility.

[0004] By detecting the user's brain activity, brain-computer interface systems can monitor, replace, improve / restore, enhance, and supplement other areas such as healthcare, education, entertainment, smart homes, and engineering. Brain-computer interface systems based on motor imagery EEG signals also have many applications in these fields: healthcare, education, and engineering.

[0005] In summary, motor imagery classification technology plays a significant role in many fields and has become a cutting-edge technology that countries are actively investing in. Furthermore, motor imagery classification technology is an interdisciplinary and complex system engineering project with significant scientific value and broad application prospects. Summary of the Invention

[0006] The purpose of the present invention is to solve the problem of low accuracy in classification and recognition of motor imagery EEG signals due to low signal-to-noise ratio of EEG signals, and to propose a motor imagery EEG signal classification method, system and storage medium based on deep learning.

[0007] A motor imagery EEG signal classification method based on deep learning has the following specific process:

[0008] Step 1: Obtain the original EEG signal dataset;

[0009] Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector;

[0010] Step 2: Perform channel selection on the original two-dimensional EEG signal feature vector to obtain a new two-dimensional EEG signal feature vector set;

[0011] Step 3: Build an EEG-TCN-LSTM convolutional neural network;

[0012] The EEG-TCN-LSTM convolutional neural network is a multi-domain feature fusion network;

[0013] The EEG-TCN-LSTM convolutional neural network includes: improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN, and classifier;

[0014] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depthwise convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0015] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, and ELU activation function;

[0016] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0017] TCN residual block 1 includes:

[0018] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0019] TCN residual block 2 includes:

[0020] a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0021] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0022] The classifier consists of a fully connected layer and the first Softmax activation function layer in sequence;

[0023] The working process of the EEG-TCN-LSTM convolutional neural network is:

[0024] The new two-dimensional EEG signal feature vector obtained in step 2 is input into the improved EEGNet convolutional neural network, and the improved EEGNet convolutional neural network outputs the two-dimensional EEG frequency-spatial domain feature X fs ;

[0025] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved TCN temporal convolutional neural network, and the improved TCN temporal convolutional neural network outputs the two-dimensional EEG time domain feature X t ;

[0026] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved LSTM recurrent neural network RNN, and the improved LSTM recurrent neural network RNN ​​outputs the two-dimensional EEG time domain feature X l ;

[0027] The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0028] Flatten the concatenated frequency-time-space domain feature X to obtain feature X′;

[0029] The feature X′ is input into the classifier, and the classifier outputs the predicted probability of each motor imagery category;

[0030] Step 4: Obtain the trained EEG-TCN-LSTM convolutional neural network;

[0031] Step 5: Obtain the subject's EEG signal to be tested;

[0032] Preprocessing the EEG signal data to be measured into a two-dimensional EEG signal feature vector;

[0033] Perform channel selection on the original two-dimensional EEG signal feature vector to obtain a new two-dimensional EEG signal feature vector set;

[0034] The two-dimensional vector is input into the trained EEG-TCN-LSTM convolutional neural network, and the trained EEG-TCN-LSTM convolutional neural network outputs the predicted probability of each motor imagery category.

[0035] A motor imagery EEG signal classification system based on deep learning is used to implement a motor imagery EEG signal classification method based on deep learning.

[0036] A computer storage medium stores at least one instruction, which is loaded and executed by a processor to implement the motor imagery EEG signal classification method based on deep learning.

[0037] The beneficial effects of the present invention are:

[0038] The present invention designs and implements a motor imagery EEG signal classification method and system based on deep learning. The ReliefF-MR algorithm is used to perform channel selection on the two-dimensional original EEG signal feature vector. An improved EEGNet convolutional neural network is used to extract frequency and spatial domain features from the preprocessed EEG signal. Then, an improved TCN temporal convolutional neural network is combined with an improved LSTM recurrent neural network (RNN) to superimpose the two time domain features extracted from the EEG signal. A feedforward neural network classifier is used for final intention recognition, thereby improving the accuracy of motor imagery EEG signal classification and recognition.

[0039] The RelidfF-MR channel selection algorithm designed in this invention realizes the optimal channel selection of motor imagery EEG signals;

[0040] The network model designed by the present invention has better classification effect than other network models such as FBCSP;

[0041] The method of the present invention was evaluated on the BCI2008 IV-2a dataset. The results showed that the classification accuracy of the method of the present invention was 73.07%, which can effectively complete the classification of EEG signals and enable the classification effect to be visualized on the web page. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the present invention;

[0043] Figure 2 It is an improved EEGNet convolutional neural network architecture diagram;

[0044] Figure 3 It is an improved TCN temporal convolutional neural network architecture diagram;

[0045] Figure 4a It is an improved LSTM recurrent neural network RNN ​​graph;

[0046] Figure 4b This is the LSTM unit structure diagram;

[0047] Figure 5 This is a detailed diagram of the EEG-TCN-LSTM convolutional neural network structure;

[0048] Figure 6 This is a comparison chart of classification accuracy of different channel selection algorithms;

[0049] Figure 7 This is a comparison chart of the optimal number of channels for different channel selection algorithms;

[0050] Figure 8 This is a histogram of the average decoding accuracy and Kappa coefficient of subjects with different models in the ablation experiment;

[0051] Figure 9a This is the confusion matrix diagram of subject A3 under the EEG-TCN-LSTM convolutional neural network model;

[0052] Figure 9b This is the confusion matrix diagram of subject A7 under the EEG-TCN-LSTM convolutional neural network model;

[0053] Figure 10a This is the t-SNE visualization result of subject A3;

[0054] Figure 10b This is the t-SNE visualization result of subject A7;

[0055] Figure 11 This is the login interface of the motor imagery EEG signal classification system based on deep learning;

[0056] Figure 12 This is the interface diagram of the channel selection algorithm of the motor imagery EEG signal classification system based on deep learning;

[0057] Figure 13a This is the main interface diagram of the feature extraction and classification system of motor imagery EEG signal classification based on deep learning;

[0058] Figure 13b This is the interface diagram of feature extraction and classification details of the motor imagery EEG signal classification system based on deep learning. DETAILED DESCRIPTION

[0059] Specific implementation method 1: Combination Figure 1 This embodiment describes a method for classifying motor imagery EEG signals based on deep learning. The specific process is as follows:

[0060] Step 1: Obtain the original EEG signal dataset;

[0061] Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector;

[0062] Step 2: In order to make full use of the effective information of the electrode channels and reduce the influence of irrelevant channels on the classification results, and further improve the classification accuracy of motor imagery signals, channel selection is performed on the original dataset;

[0063] The ReliefF-MR algorithm is used to select channels of the original two-dimensional EEG signal feature vectors to obtain a new two-dimensional EEG signal feature vector set;

[0064] Step 3: Build an EEG-TCN-LSTM convolutional neural network;

[0065] The EEG-TCN-LSTM convolutional neural network is a multi-domain feature fusion network;

[0066] The EEG-TCN-LSTM convolutional neural network includes: improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN, and classifier;

[0067] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depthwise convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0068] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, and ELU activation function;

[0069] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0070] TCN residual block 1 includes:

[0071] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0072] TCN residual block 2 includes:

[0073] a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0074] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0075] The classifier consists of a fully connected layer and the first Softmax activation function layer in sequence;

[0076] The working process of the EEG-TCN-LSTM convolutional neural network is:

[0077] The new two-dimensional EEG signal feature vector obtained in step 2 is input into the improved EEGNet convolutional neural network, and the improved EEGNet convolutional neural network outputs the two-dimensional EEG frequency-spatial domain feature X fs ;

[0078] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved TCN temporal convolutional neural network, and the improved TCN temporal convolutional neural network outputs the two-dimensional EEG time domain feature X t ;

[0079] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved LSTM recurrent neural network RNN, and the improved LSTM recurrent neural network RNN ​​outputs the two-dimensional EEG time domain feature X l ;

[0080] The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0081] Flatten the concatenated frequency-time-space domain feature X to obtain feature X′;

[0082] The feature X′ is input into the classifier, and the classifier outputs the predicted probability of each motor imagery category;

[0083] Step 4: Obtain the trained EEG-TCN-LSTM convolutional neural network;

[0084] Step 5: Obtain the subject's EEG signal (the subject uses 22 Ag / AgCl electrodes (the electrode spacing is 3.5 cm) to record the subject's EEG signal;);

[0085] Preprocessing the EEG signal data to be measured into a two-dimensional EEG signal feature vector;

[0086] The ReliefF-MR algorithm is used to select channels of the original two-dimensional EEG signal feature vectors to obtain a new two-dimensional EEG signal feature vector set;

[0087] The two-dimensional vector is input into the trained EEG-TCN-LSTM convolutional neural network, and the trained EEG-TCN-LSTM convolutional neural network outputs the predicted probability of each motor imagery category.

[0088] Specific embodiment 2: This embodiment differs from specific embodiment 1 in that the original EEG signal dataset is obtained in step 1;

[0089] Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector;

[0090] The specific process is:

[0091] Obtain the original EEG signal dataset from the BCI2008 IV-2a database (public dataset);

[0092] The MNE library is used to process the raw EEG signal into a two-dimensional raw EEG signal feature vector.

[0093] Step 1.1: The original EEG data set consists of EEG data of 9 subjects;

[0094] Each subject performed two experiments on two different days; each experiment consisted of six runs with a short break between runs. One run consisted of 48 trials (12 for each of the four possible categories), for a total of 288 trials per experiment.

[0095] Each subject sat in a comfortable armchair in front of a computer screen, and EEG was recorded using 22 Ag / AgCl electrodes (3.5 cm apart). All signals were recorded monopolarly, with the left mastoid serving as the reference electrode and the right mastoid serving as the ground electrode. The signals were sampled at 250 Hz and band-pass filtered from 0.5 Hz to 100 Hz. The amplifier sensitivity was set to 100 μV, and a 50 Hz notch filter was enabled to suppress power supply noise.

[0096] At the beginning of each experiment, approximately 5 minutes of recording were performed to estimate the effects of the electrooculogram (EOG). The recording was divided into three parts: (1) two minutes of eyes open (looking at a fixation cross on the screen), (2) one minute of eyes closed, and (3) one minute of eye movements;

[0097] At the beginning of a trial, a fixation cross appears on a black screen. In addition, a short sound prompt is played. Two seconds later, an arrow prompt appears on the screen, pointing to the left, right, down, or up (corresponding to four categories: left hand, right hand, feet, or tongue), and the arrow stays on the screen for 1.25 seconds. This prompts the subject to perform the required motor imagery task. No feedback is provided during the trial. The subject is asked to continue the motor imagery task until the fixation cross disappears from the screen. This is followed by a short break, when the screen turns black again and the next experiment is carried out;

[0098] Step 1.2: Each subject generates two EEG signal samples. The EEG signal data is represented as a two-dimensional vector. The signal feature dimension is [1, 22, 1125], where 22 is the number of electrode channels in the original EEG signal, 1125 is the number of time points, and 1 is the expanded dimension of the channel number.

[0099] Next, input is given to the EEGNet structure, TCN structure, and LSTM structure for feature learning.

[0100] Other steps and parameters are the same as those in the first embodiment.

[0101] Specific embodiment three: This embodiment differs from specific embodiments one or two in that, in step 2, in order to make full use of the effective information of the electrode channels while reducing the influence of irrelevant channels on the classification results and further improve the classification accuracy of motor imagery signals, channel selection is performed on the original data set;

[0102] The ReliefF-MR algorithm is used to perform channel selection on the two-dimensional original EEG signal feature vector obtained in step 1 to obtain a new two-dimensional EEG signal feature vector set;

[0103] In step 3, verify the improvement of channel selection in the proposed improved deep learning model;

[0104] The specific process is:

[0105] Step 2.1: Calculate the correlation statistic d of the two-dimensional original EEG signal feature vectors using the ReliefF algorithm, sort the two-dimensional original EEG signal feature vectors in ascending order according to the correlation statistic, and remove all two-dimensional original EEG signal feature vectors with correlation statistics less than 0 to obtain the set A of two-dimensional original EEG signal feature vectors;

[0106] Step 2.2: Calculate the redundancy R1 of the two-dimensional original EEG signal feature vector set A;

[0107] Selecting the eigenvector t with the smallest correlation statistic in the two-dimensional original EEG signal eigenvector set A, removing the eigenvector t from the two-dimensional original EEG signal eigenvector set A to obtain the two-dimensional original EEG signal eigenvector set B; calculating the redundancy R2 of the two-dimensional original EEG signal eigenvector set B;

[0108] Compare the redundancy R1 and redundancy R2:

[0109] If the redundancy R2 is less than the redundancy R1, it means that the redundancy cannot be reduced, and the two-dimensional original EEG signal feature vector set A is the optimal feature set, which is used as the new two-dimensional EEG signal feature vector set;

[0110] If the redundancy R2 is greater than or equal to the redundancy R1, the two-dimensional original EEG signal feature vector set B is used as the new two-dimensional original EEG signal feature vector set A, and step 2.2 is repeated.

[0111] Other steps and parameters are the same as those in the first or second embodiment.

[0112] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the processing of improving the EEGNet convolutional neural network in step 3 is as follows:

[0113] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depthwise convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0114] Improve the output data of the i′th layer of the EEGNet convolutional neural network with X i′ Indicates that i′=1,2,3,...,16, the first layer is the input layer, the second layer is the one-dimensional convolution layer, the third layer is the first activation function layer, the fourth layer is the first batch normalization layer, the fifth layer is the depthwise convolution layer, the sixth layer is the second activation function layer, the seventh layer is the second batch normalization layer, the eighth layer is the third activation function layer, the ninth layer is the first average pooling layer, the tenth layer is the first Dropout layer, the eleventh layer is the separable convolution layer, the twelfth layer is the fourth activation function layer, the thirteenth layer is the third batch normalization layer, the fourteenth layer is the fifth activation function layer, the fifteenth layer is the second average pooling layer, and the sixteenth layer is the second Dropout layer;

[0115] 1) Input the two-dimensional EEG signal feature vector obtained in step 2 into the input layer of the improved EEGNet convolutional neural network. The shape of the EEG signal output by the input layer is [1, C, T];

[0116] 1 is the number of characteristic channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125);

[0117] The number of electrode channels C depends on the feature vector extracted from the original EEG signal by the ReliefF-MR algorithm. For the BCI2008 IV-2a dataset, C is 22;

[0118] 2) Input the EEG signal output by the input layer into the one-dimensional convolution layer, and the one-dimensional convolution layer outputs the feature X2; input the feature X2 output by the one-dimensional convolution layer into the first activation function layer, and the first activation function layer outputs the feature; specifically:

[0119] Set the one-dimensional convolution layer to a convolution filter with 8 kernels, kernel size [1,32] and stride size [1,1], where stride represents the x and y movement distances of the kernel.

[0120] Set the padding method to zero padding of the same shape, which keeps the sample shape unchanged in the convolution calculation;

[0121] The EEG signal output by the input layer is input into the one-dimensional convolution layer. The one-dimensional convolution layer outputs the feature X2. The shape of the feature X2 is [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125).

[0122] The one-dimensional convolution layer output feature X2 is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0123] The mathematical expression of the Linear activation function is as follows:

[0124] f(x)=x

[0125] 3) Input the output feature of the first activation function layer into the first batch normalization layer, and the first batch normalization layer outputs feature X3; specifically:

[0126] Perform batch normalization on the output features of the first activation function layer, and obtain the shape of feature X3 as [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125);

[0127] 4) Input the output feature X3 of the first batch normalization layer into the depth convolution layer, and the depth convolution layer outputs the feature X4; input the output feature X4 of the depth convolution layer into the second activation function layer, and the second activation function layer outputs the feature; specifically:

[0128] Set the depthwise convolutional layer to a convolution filter with 2 kernels, kernel size [C, 1], and stride size [1, 1], where stride represents the x and y movement distances of the kernel.

[0129] Set the padding method to zero padding of the same shape, which keeps the sample shape unchanged in the convolution calculation;

[0130] The output feature X3 of the first batch normalization layer is input into the depth convolution layer, and the depth convolution layer outputs feature X4. The shape of feature X4 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0131] The output feature X4 of the deep convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0132] 5) Input the output feature of the second activation function layer into the second batch normalization layer, and the second batch normalization layer outputs feature X5; specifically:

[0133] Perform batch normalization on the output features of the second activation function layer, and obtain the shape of feature X5 as [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (e.g., 1125);

[0134] 6) Input the output feature X5 of the second batch normalization layer into the third activation function layer, and the third activation function layer outputs feature X6; specifically:

[0135] Use the ELU activation function to process the second batch normalization layer output feature X5, and the shape of the feature X6 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0136] The mathematical expression of the activation function ELU is as follows:

[0137]

[0138] 7) Input the output feature X6 of the third activation function layer into the first average pooling layer, and the first average pooling layer outputs feature X7; specifically:

[0139] Set the pooling window size of the first average pooling layer to [1,8] and the stride size to [1,8];

[0140] The output feature X6 of the third activation function layer is subjected to the first average pooling layer operation, and the shape of the feature X7 obtained is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140);

[0141] 8) Input the output feature X7 of the first average pooling layer into the first Dropout layer, and the first Dropout layer outputs feature X8; specifically:

[0142] Set the Dropout probability to 0.2;

[0143] The first average pooling layer outputs feature X7, which is then processed by the first Dropout layer to obtain feature X8 with a shape of [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140).

[0144] 9) Input the output feature X8 of the first Dropout layer into the separable convolution layer, and the separable convolution layer outputs X9; input the output feature X9 of the separation convolution layer into the fourth activation function layer, and the fourth activation function layer outputs the feature;

[0145] Specifically:

[0146] Set the separable convolution layer to a convolution filter with 16 kernels, a kernel size of [1,16], and a stride size of [1,1]. The stride represents the x- and y-movement distances of the kernel.

[0147] The output feature X8 of the first Dropout layer is input into the separable convolution layer, and the separable convolution layer outputs X9. The shape of feature X9 is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140).

[0148] The output feature X9 of the separation convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0149] 10) Input the output features of the fourth activation function layer into the third batch normalization layer, and the third batch normalization layer outputs features X 10 Specifically:

[0150] Perform batch normalization on the output features of the fourth activation function layer to obtain feature X 10 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g. 140);

[0151] 11) Output feature X of the third batch normalization layer 10 Input the fifth activation function layer, the fifth activation function layer outputs feature X 11 Specifically:

[0152] The third batch normalization layer outputs feature X 10 Input ELU activation function layer, ELU activation function layer output feature X 11 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width;

[0153] 12) Output feature X of the fifth activation function layer 11 Input the second average pooling layer, the second average pooling layer outputs feature X 12 Specifically:

[0154] Set the pooling window size of the second average pooling layer to [1,8] and the stride size to [1,8];

[0155] The fifth activation function layer outputs feature X 11 After the second average pooling layer operation, feature X is obtained 12 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width (e.g. 17);

[0156] 13) Output feature X of the second average pooling layer 12 Input the second Dropout layer, the second Dropout layer outputs feature X 13 Specifically:

[0157] Set the Dropout probability to 0.2;

[0158] The second average pooling layer outputs feature X 12 After the second Dropout layer operation, feature X is obtained 13 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0159] Feature X 13 As the frequency-spatial domain feature X extracted by the improved EEGNet convolutional neural network fs , called the two-dimensional EEG frequency-space domain feature X fs .

[0160] In the basic model, adjust the number of convolutional layers, pooling layers, and learning rate, identify the optimal network structure through classification accuracy, loss rate, ROC curve, and confusion matrix, and analyze and compare the experimental results;

[0161] The other steps and parameters are the same as those in the first to third embodiments.

[0162] Specific embodiment 5: This embodiment differs from any one of specific embodiments 1 to 4 in that the processing process of the improved TCN temporal convolutional neural network is as follows:

[0163] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, and ELU activation function;

[0164] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0165] Improve the output data of the i-th layer of the TCN temporal convolutional neural network structure with X i , i=1,2,3, the first layer is the input layer, the second layer is the TCN residual module, and the third layer is the ELU activation function;

[0166] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0167] TCN residual block 1 includes:

[0168] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0169] TCN residual block 2 includes:

[0170] a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0171] The mathematical expression of dilated causal convolution is as follows:

[0172]

[0173] Where d is the dilation factor, k is the convolution kernel size, sd·i represents the past direction, and x∈R n It is a one-dimensional sequence input, filter f:{0,…,k-1}→R.

[0174] The mathematical expression of the activation function ReLU is as follows:

[0175]

[0176] 1) Improve the two-dimensional EEG frequency-spatial domain feature X output by the EEGNet convolutional neural network fs Input the input layer of the improved TCN temporal convolutional neural network, and the shape of the EEG signal output by the input layer is [16, 1, T / 64];

[0177] 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0178] 2) Input the EEG signal output by the input layer in 1) into the TCN residual module. The shape of the EEG signal output by the TCN residual module is [12, T / 64]. Input the EEG signal output by the TCN residual module into the ELU activation function. The ELU activation function outputs features. 12 is the number of channels and T / 64 is the width.

[0179] The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t ;

[0180] Specifically:

[0181] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0182] TCN residual block 1 includes:

[0183] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0184] TCN residual block 2 includes:

[0185] a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0186] The first dilated causal convolution layer, the second dilated causal convolution layer, the third dilated causal convolution layer, and the fourth dilated causal convolution layer are all set to use convolution filters with 12 convolution kernels and a convolution kernel size of [1, 4].

[0187] The EEG signal output by the input layer in 1) is input into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]. The EEG signal output by the TCN residual module is input into the ELU activation function, and the ELU activation function outputs features.

[0188] The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t .

[0189] In the basic model, the number of convolutional layers, pooling layers and learning rate are adjusted, and the optimal network model for extracting time domain features is found through classification accuracy, loss rate, ROC curve and confusion matrix.

[0190] The other steps and parameters are the same as those in the first to fourth embodiments.

[0191] Specific embodiment 6: This embodiment differs from any one of specific embodiments 1 to 5 in that the processing process of the improved LSTM recurrent neural network RNN ​​is as follows:

[0192] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0193] Improve the output data of the i″th layer of the LSTM recurrent neural network RNN ​​with X i″ Indicates that i″=1,2,3, the first layer is the input layer, the second layer is the LSTM layer, and the third layer is the Dropout layer;

[0194] 1) The two-dimensional EEG frequency-spatial domain feature X fs Input the input layer of the improved LSTM recurrent neural network RNN. The shape of the EEG signal output by the input layer is [16, 1, T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0195] 2) Input the EEG signal output by the input layer in 1) into the LSTM layer. The shape of the EEG signal output by the LSTM layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically:

[0196] Set the number of neurons in the hidden layer of the LSTM module to 20;

[0197] The EEG signal output from the input layer in 1) is processed through the LSTM layer. The resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width.

[0198] 3) Input the EEG signal output by the LSTM layer in 2) into the seventh Dropout layer. The shape of the EEG signal output by the seventh Dropout layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically:

[0199] Set the seventh Dropout probability to 0.5;

[0200] The EEG signal output by the LSTM layer in 2) is passed through the seventh Dropout layer. The resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width.

[0201] The seventh Dropout layer outputs the EEG signal as the time domain feature extracted by the improved LSTM recurrent neural network RNN, which is called the two-dimensional EEG time domain feature X l ;

[0202] This method can automatically learn the time domain features of two-dimensional EEG signals through the LSTM structure.

[0203] In the basic model, the number of convolutional layers, pooling layers, and learning rate are adjusted, and the optimal network structure is distinguished through classification accuracy, loss rate, ROC curve, and confusion matrix, and the experimental results are analyzed and compared.

[0204] The other steps and parameters are the same as those in the first to fifth embodiments.

[0205] Specific embodiment 7: This embodiment differs from any one of specific embodiments 1 to 6 in that the processing process of the classifier is as follows:

[0206] The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0207] The shape of the concatenated frequency-time-space domain feature X is [32, T / 64], where 32 is the number of channels and T / 64 is the width;

[0208] The concatenated frequency-time-space domain feature X is tensor-flattened to obtain feature X'. The shape of feature X' is [544,1], where 544 is the number of channels and 1 is the width.

[0209] The classifier consists of a fully connected layer and the first Softmax activation function layer in sequence;

[0210] The output data of the i′′ layer of the classifier structure is represented by X i″′ Indicates that i″′=1,2, the first layer is the fully connected layer, and the second layer is the first Softmax activation function layer;

[0211] 1) Input feature X' into the fully connected layer of the classifier. The fully connected layer outputs data in the shape of [4], where 4 is the unnormalized score of each motor imagery category.

[0212] 2) Input the output data of the fully connected layer into the first Softmax activation function layer of the classifier. The output data shape of the first Softmax activation function layer is [4], where 4 is the predicted probability of each motor imagery category;

[0213] 4 represents the left hand, right hand, feet, and tongue;

[0214] The mathematical expression of the activation function Softmax is as follows:

[0215]

[0216] Where Z is the input vector, σ(Z) i is the i-th element in the output vector, and e is a natural constant;

[0217] Let Z1 and Z2 represent the output results of the fully connected layer and the activation function layer respectively, then the calculation of Z1 and Z2 can be expressed as:

[0218] Z1=WX'+b

[0219] Where W represents the weight matrix, the shape is [4,544]; b represents the bias vector, the shape is [4,1];

[0220]

[0221] The other steps and parameters are the same as those in the first to sixth embodiments.

[0222] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that the trained EEG-TCN-LSTM convolutional neural network is obtained in step 4; the specific process is as follows:

[0223] Cross entropy is used as the loss function, and the loss function is optimized by the Adam optimizer until convergence to obtain the trained EEG-TCN-LSTM convolutional neural network.

[0224] The other steps and parameters are the same as those in the first to seventh embodiments.

[0225] Specific embodiment nine: In this embodiment, a motor imagery EEG signal classification system based on deep learning is used to execute the motor imagery EEG signal classification method based on deep learning described in any one of embodiments one to eight.

[0226] Specific embodiment ten: This embodiment is a computer storage medium, which stores at least one instruction. The at least one instruction is loaded and executed by a processor to implement the deep learning-based motor imagery EEG signal classification method described in any one of specific embodiments one to specific embodiments eight.

[0227] It should be understood that the instructions include computer program products, software, or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system or other electronic device. Computer storage media may include readable media on which instructions are stored, and may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions.

[0228] Example

[0229] Example 1:

[0230] A motor imagery EEG signal classification method based on deep learning has the following specific process:

[0231] Step 1: Obtain the original EEG signal dataset;

[0232] Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector;

[0233] The specific process is:

[0234] Obtain the original EEG signal dataset from the BCI2008 IV-2a database (public dataset);

[0235] The MNE library is used to process the raw EEG signal into a two-dimensional raw EEG signal feature vector.

[0236] Step 1.1: The original EEG data set consists of EEG data of 9 subjects;

[0237] Each subject performed two experiments on two different days; each experiment consisted of six runs with a short break between runs. One run consisted of 48 trials (12 for each of the four possible categories), for a total of 288 trials per experiment.

[0238] Each subject sat in a comfortable armchair in front of a computer screen, and EEG was recorded using 22 Ag / AgCl electrodes (3.5 cm apart). All signals were recorded monopolarly, with the left mastoid serving as the reference electrode and the right mastoid serving as the ground electrode. The signals were sampled at 250 Hz and band-pass filtered from 0.5 Hz to 100 Hz. The amplifier sensitivity was set to 100 μV, and a 50 Hz notch filter was enabled to suppress power supply noise.

[0239] At the beginning of each experiment, approximately 5 minutes of recording were performed to estimate the effects of the electrooculogram (EOG). The recording was divided into three parts: (1) two minutes of eyes open (looking at a fixation cross on the screen), (2) one minute of eyes closed, and (3) one minute of eye movements;

[0240] At the beginning of a trial, a fixation cross appears on a black screen. In addition, a short sound prompt is played. Two seconds later, an arrow prompt appears on the screen, pointing to the left, right, down, or up (corresponding to four categories: left hand, right hand, feet, or tongue), and the arrow stays on the screen for 1.25 seconds. This prompts the subject to perform the required motor imagery task. No feedback is provided during the trial. The subject is asked to continue the motor imagery task until the fixation cross disappears from the screen. This is followed by a short break, when the screen turns black again and the next experiment is carried out;

[0241] Step 1.2: Each subject generates two EEG signal samples. The EEG signal data is represented as a two-dimensional vector. The signal feature dimension is [1, 22, 1125], where 22 is the number of electrode channels in the original EEG signal, 1125 is the number of time points, and 1 is the expanded dimension of the channel number.

[0242] Next, input is given to the EEGNet structure, TCN structure, and LSTM structure for feature learning.

[0243] Step 2: In order to make full use of the effective information of the electrode channels and reduce the influence of irrelevant channels on the classification results, and further improve the classification accuracy of motor imagery signals, channel selection is performed on the original dataset;

[0244] The ReliefF-MR algorithm is used to select channels of the original two-dimensional EEG signal feature vectors to obtain a new two-dimensional EEG signal feature vector set;

[0245] The specific process is:

[0246] Step 2.1: Calculate the correlation statistic d of the two-dimensional original EEG signal feature vectors using the ReliefF algorithm, sort the two-dimensional original EEG signal feature vectors in ascending order according to the correlation statistic, and remove all two-dimensional original EEG signal feature vectors with correlation statistics less than 0 to obtain the set A of two-dimensional original EEG signal feature vectors;

[0247] Step 2.2: Calculate the redundancy R1 of the two-dimensional original EEG signal feature vector set A;

[0248] Selecting the eigenvector t with the smallest correlation statistic in the two-dimensional original EEG signal eigenvector set A, removing the eigenvector t from the two-dimensional original EEG signal eigenvector set A to obtain the two-dimensional original EEG signal eigenvector set B; calculating the redundancy R2 of the two-dimensional original EEG signal eigenvector set B;

[0249] Compare the redundancy R1 and redundancy R2:

[0250] If the redundancy R2 is less than the redundancy R1, it means that the redundancy cannot be reduced, and the two-dimensional original EEG signal feature vector set A is the optimal feature set, which is used as the new two-dimensional EEG signal feature vector set;

[0251] If the redundancy R2 is greater than or equal to the redundancy R1, the two-dimensional original EEG signal feature vector set B is used as the new two-dimensional original EEG signal feature vector set A, and step 2.2 is repeated.

[0252] The ReliefF-MR algorithm is as follows:

[0253]

[0254]

[0255] After channel selection of the data, the classification effects on the neural network before and after channel selection are compared.

[0256] Step 3: Build an EEG-TCN-LSTM convolutional neural network;

[0257] The EEG-TCN-LSTM convolutional neural network is a multi-domain feature fusion network;

[0258] The EEG-TCN-LSTM convolutional neural network includes: improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN, and classifier;

[0259] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depthwise convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0260] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, and ELU activation function;

[0261] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0262] TCN residual block 1 includes:

[0263] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0264] TCN residual block 2 includes:

[0265] a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0266] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0267] The classifier consists of a fully connected layer and the first Softmax activation function layer in sequence;

[0268] The working process of the EEG-TCN-LSTM convolutional neural network is:

[0269] The new two-dimensional EEG signal feature vector obtained in step 2 is input into the improved EEGNet convolutional neural network, and the improved EEGNet convolutional neural network outputs the two-dimensional EEG frequency-spatial domain feature X fs ;

[0270] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved TCN temporal convolutional neural network, and the improved TCN temporal convolutional neural network outputs the two-dimensional EEG time domain feature X t ;

[0271] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved LSTM recurrent neural network RNN, and the improved LSTM recurrent neural network RNN ​​outputs the two-dimensional EEG time domain feature X l ;

[0272] The two-dimensional EEG time domain feature X t and the two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0273] Flatten the concatenated frequency-time-space domain feature X to obtain feature X';

[0274] The feature X' is input into the classifier, and the classifier outputs the predicted probability of each motor imagery category;

[0275] The processing process of the improved EEGNet convolutional neural network is as follows:

[0276] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depthwise convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0277] Improve the output data of the i′th layer of the EEGNet convolutional neural network with X i′ Indicates that i′=1,2,3,...,16, the first layer is the input layer, the second layer is the one-dimensional convolution layer, the third layer is the first activation function layer, the fourth layer is the first batch normalization layer, the fifth layer is the depthwise convolution layer, the sixth layer is the second activation function layer, the seventh layer is the second batch normalization layer, the eighth layer is the third activation function layer, the ninth layer is the first average pooling layer, the tenth layer is the first Dropout layer, the eleventh layer is the separable convolution layer, the twelfth layer is the fourth activation function layer, the thirteenth layer is the third batch normalization layer, the fourteenth layer is the fifth activation function layer, the fifteenth layer is the second average pooling layer, and the sixteenth layer is the second Dropout layer;

[0278] 1) Input the two-dimensional EEG signal feature vector obtained in step 2 into the input layer of the improved EEGNet convolutional neural network. The shape of the EEG signal output by the input layer is [1, C, T];

[0279] 1 is the number of characteristic channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125);

[0280] The number of electrode channels C depends on the feature vector extracted from the original EEG signal by the ReliefF-MR algorithm. For the BCI2008 IV-2a dataset, C is 22;

[0281] 2) Input the EEG signal output by the input layer into the one-dimensional convolution layer, and the one-dimensional convolution layer outputs the feature X2; input the feature X2 output by the one-dimensional convolution layer into the first activation function layer, and the first activation function layer outputs the feature; specifically:

[0282] Set the one-dimensional convolution layer to a convolution filter with 8 kernels, kernel size [1,32] and stride size [1,1], where stride represents the x and y movement distances of the kernel.

[0283] Set the padding method to zero padding of the same shape, which keeps the sample shape unchanged in the convolution calculation;

[0284] The EEG signal output by the input layer is input into the one-dimensional convolution layer. The one-dimensional convolution layer outputs the feature X2. The shape of the feature X2 is [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125).

[0285] The one-dimensional convolution layer output feature X2 is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0286] The mathematical expression of the Linear activation function is as follows:

[0287] f(x)=x

[0288] 3) Input the output feature of the first activation function layer into the first batch normalization layer, and the first batch normalization layer outputs feature X3; specifically:

[0289] Perform batch normalization on the output features of the first activation function layer, and obtain the shape of feature X3 as [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125);

[0290] 4) Input the output feature X3 of the first batch normalization layer into the depth convolution layer, and the depth convolution layer outputs the feature X4; input the output feature X4 of the depth convolution layer into the second activation function layer, and the second activation function layer outputs the feature; specifically:

[0291] Set the depthwise convolutional layer to a convolution filter with 2 kernels, kernel size [C, 1], and stride size [1, 1], where stride represents the x and y movement distances of the kernel.

[0292] Set the padding method to zero padding of the same shape, which keeps the sample shape unchanged in the convolution calculation;

[0293] The output feature X3 of the first batch normalization layer is input into the depth convolution layer, and the depth convolution layer outputs feature X4. The shape of feature X4 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0294] The output feature X4 of the deep convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0295] 5) Input the output feature of the second activation function layer into the second batch normalization layer, and the second batch normalization layer outputs feature X5; specifically:

[0296] Perform batch normalization on the output features of the second activation function layer, and obtain the shape of feature X5 as [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (e.g., 1125);

[0297] 6) Input the output feature X5 of the second batch normalization layer into the third activation function layer, and the third activation function layer outputs feature X6; specifically:

[0298] Use the ELU activation function to process the second batch normalization layer output feature X5, and the shape of the feature X6 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0299] The mathematical expression of the activation function ELU is as follows:

[0300]

[0301] 7) Input the output feature X6 of the third activation function layer into the first average pooling layer, and the first average pooling layer outputs feature X7; specifically:

[0302] Set the pooling window size of the first average pooling layer to [1,8] and the stride size to [1,8];

[0303] The output feature X6 of the third activation function layer is subjected to the first average pooling layer operation, and the shape of the feature X7 obtained is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140);

[0304] 8) Input the output feature X7 of the first average pooling layer into the first Dropout layer, and the first Dropout layer outputs feature X8; specifically:

[0305] Set the Dropout probability to 0.2;

[0306] The first average pooling layer outputs feature X7, which is then processed by the first Dropout layer to obtain feature X8 with a shape of [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140).

[0307] 9) Input the output feature X8 of the first Dropout layer into the separable convolution layer, and the separable convolution layer outputs X9; input the output feature X9 of the separation convolution layer into the fourth activation function layer, and the fourth activation function layer outputs the feature;

[0308] Specifically:

[0309] Set the separable convolution layer to a convolution filter with 16 kernels, a kernel size of [1,16], and a stride size of [1,1]. The stride represents the x- and y-movement distances of the kernel.

[0310] The output feature X8 of the first Dropout layer is input into the separable convolution layer, and the separable convolution layer outputs X9. The shape of feature X9 is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140).

[0311] The output feature X9 of the separation convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0312] 10) Input the output features of the fourth activation function layer into the third batch normalization layer, and the third batch normalization layer outputs features X 10 Specifically:

[0313] Perform batch normalization on the output features of the fourth activation function layer to obtain feature X 10 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g. 140);

[0314] 11) Output feature X of the third batch normalization layer 10 Input the fifth activation function layer, the fifth activation function layer outputs feature X 11 Specifically:

[0315] The third batch normalization layer outputs feature X 10 Input ELU activation function layer, ELU activation function layer output feature X 11 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width;

[0316] 12) Output feature X of the fifth activation function layer 11 Input the second average pooling layer, the second average pooling layer outputs feature X 12 Specifically:

[0317] Set the pooling window size of the second average pooling layer to [1,8] and the stride size to [1,8];

[0318] The fifth activation function layer outputs feature X 11 After the second average pooling layer operation, feature X is obtained 12 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width (e.g. 17);

[0319] 13) Output feature X of the second average pooling layer 12 Input the second Dropout layer, the second Dropout layer outputs feature X 13 Specifically:

[0320] Set the Dropout probability to 0.2;

[0321] The second average pooling layer outputs feature X 12 After the second Dropout layer operation, feature X is obtained 13The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0322] Feature X 13 As the frequency-spatial domain feature X extracted by the improved EEGNet convolutional neural network fs , called the two-dimensional EEG frequency-space domain feature X fs .

[0323] In the basic model, adjust the number of convolutional layers, pooling layers, and learning rate, identify the optimal network structure through classification accuracy, loss rate, ROC curve, and confusion matrix, and analyze and compare the experimental results;

[0324] The processing process of the improved TCN temporal convolutional neural network is as follows:

[0325] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, and ELU activation function;

[0326] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0327] Improve the output data of the i-th layer of the TCN temporal convolutional neural network structure with X i , i=1,2,3, the first layer is the input layer, the second layer is the TCN residual module, and the third layer is the ELU activation function;

[0328] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0329] TCN residual block 1 includes:

[0330] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0331] TCN residual block 2 includes:

[0332] a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0333] The mathematical expression of dilated causal convolution is as follows:

[0334]

[0335] Where d is the dilation factor, k is the convolution kernel size, sd·i represents the past direction, and x∈R n It is a one-dimensional sequence input, filter f:{0,…,k-1}→R.

[0336] The mathematical expression of the activation function ReLU is as follows:

[0337]

[0338] 1) Improve the two-dimensional EEG frequency-spatial domain feature X output by the EEGNet convolutional neural network fs Input the input layer of the improved TCN temporal convolutional neural network, and the shape of the EEG signal output by the input layer is [16, 1, T / 64];

[0339] 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0340] 2) Input the EEG signal output by the input layer in 1) into the TCN residual module. The shape of the EEG signal output by the TCN residual module is [12, T / 64]. Input the EEG signal output by the TCN residual module into the ELU activation function. The ELU activation function outputs features. 12 is the number of channels and T / 64 is the width.

[0341] The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t ;

[0342] Specifically:

[0343] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0344] TCN residual block 1 includes:

[0345] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0346] TCN residual block 2 includes:

[0347] a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0348] The first dilated causal convolution layer, the second dilated causal convolution layer, the third dilated causal convolution layer, and the fourth dilated causal convolution layer are all set to use convolution filters with 12 convolution kernels and a convolution kernel size of [1, 4].

[0349] The EEG signal output by the input layer in 1) is input into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]. The EEG signal output by the TCN residual module is input into the ELU activation function, and the ELU activation function outputs features.

[0350] The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t .

[0351] In the basic model, the number of convolutional layers, pooling layers and learning rate are adjusted, and the optimal network model for extracting time domain features is found through classification accuracy, loss rate, ROC curve and confusion matrix.

[0352] The processing process of the improved LSTM recurrent neural network RNN ​​is as follows:

[0353] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0354] Improve the output data of the i″th layer of the LSTM recurrent neural network RNN ​​with X i″ Indicates that i″=1,2,3, the first layer is the input layer, the second layer is the LSTM layer, and the third layer is the Dropout layer;

[0355] 1) The two-dimensional EEG frequency-spatial domain feature X fs Input the input layer of the improved LSTM recurrent neural network RNN. The shape of the EEG signal output by the input layer is [16, 1, T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0356] 2) Input the EEG signal output by the input layer in 1) into the LSTM layer. The shape of the EEG signal output by the LSTM layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically:

[0357] Set the number of neurons in the hidden layer of the LSTM module to 20;

[0358] The EEG signal output from the input layer in 1) is processed through the LSTM layer. The resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width.

[0359] 3) Input the EEG signal output by the LSTM layer in 2) into the seventh Dropout layer. The shape of the EEG signal output by the seventh Dropout layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically:

[0360] Set the seventh Dropout probability to 0.5;

[0361] The EEG signal output by the LSTM layer in 2) is passed through the seventh Dropout layer. The resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width.

[0362] The seventh Dropout layer outputs the EEG signal as the time domain feature extracted by the improved LSTM recurrent neural network RNN, which is called the two-dimensional EEG time domain feature X l ;

[0363] This method can automatically learn the time domain features of two-dimensional EEG signals through the LSTM structure.

[0364] In the basic model, the number of convolutional layers, pooling layers, and learning rate are adjusted, and the optimal network structure is distinguished through classification accuracy, loss rate, ROC curve, and confusion matrix, and the experimental results are analyzed and compared.

[0365] The processing process of the classifier is:

[0366] The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0367] The shape of the concatenated frequency-time-space domain feature X is [32, T / 64], where 32 is the number of channels and T / 64 is the width;

[0368] The concatenated frequency-time-space domain feature X is tensor-flattened to obtain feature X'. The shape of feature X' is [544,1], where 544 is the number of channels and 1 is the width.

[0369] The classifier consists of a fully connected layer and the first Softmax activation function layer in sequence;

[0370] The output data of the i′′ layer of the classifier structure is represented by X i″′ Indicates that i″′=1,2, the first layer is the fully connected layer, and the second layer is the first Softmax activation function layer;

[0371] 1) Input feature X' into the fully connected layer of the classifier. The fully connected layer outputs data in the shape of [4], where 4 is the unnormalized score of each motor imagery category.

[0372] 2) Input the output data of the fully connected layer into the first Softmax activation function layer of the classifier. The output data shape of the first Softmax activation function layer is [4], where 4 is the predicted probability of each motor imagery category;

[0373] 4 represents the left hand, right hand, feet, and tongue;

[0374] The mathematical expression of the activation function Softmax is as follows:

[0375]

[0376] Where Z is the input vector, σ(Z) i is the i-th element in the output vector, and e is a natural constant;

[0377] Let Z1 and Z2 represent the output results of the fully connected layer and the activation function layer respectively, then the calculation of Z1 and Z2 can be expressed as:

[0378] Z1=WX'+b

[0379] Where W represents the weight matrix, the shape is [4,544]; b represents the bias vector, the shape is [4,1];

[0380]

[0381] Step 4: Obtain the trained EEG-TCN-LSTM convolutional neural network; the specific process is:

[0382] Cross entropy is used as the loss function, and the loss function is optimized by the Adam optimizer until convergence to obtain the trained EEG-TCN-LSTM convolutional neural network.

[0383] Step 5: Obtain the subject's EEG signal (the subject uses 22 Ag / AgCl electrodes (the electrode spacing is 3.5 cm) to record the subject's EEG signal;);

[0384] Preprocessing the EEG signal data to be measured into a two-dimensional EEG signal feature vector;

[0385] The ReliefF-MR algorithm is used to select channels of the original two-dimensional EEG signal feature vectors to obtain a new two-dimensional EEG signal feature vector set;

[0386] The two-dimensional vector is input into the trained EEG-TCN-LSTM convolutional neural network, and the trained EEG-TCN-LSTM convolutional neural network outputs the predicted probability of each motor imagery category.

[0387] Example 2:

[0388] To fully utilize the effective information from the electrode channels while reducing the impact of irrelevant channels on the classification results, and further improve the accuracy of motor imagery signal classification, channel selection was performed on the original dataset. The ReliefF-MR algorithm was used to select channels from the two-dimensional raw EEG signal feature vectors. The improved channel selection effect was verified in the improved deep learning model proposed in step 3.

[0389] The improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN ​​and classifier are combined into a complete network structure to form the EEG-TCN-LSTM convolutional neural network. The details of the EEG-TCN-LSTM convolutional neural network structure are shown in the figure below. Figure 5 ;

[0390] After channel selection, the classification accuracy and optimal number of channels of the EEG-TCN-LSTM convolutional neural network model optimized by the ReliefF-MR algorithm, the ReliefF algorithm, and the mRMR algorithm were compared with the EEG-TCN-LSTM convolutional neural network model without channel selection optimization.

[0391] 1: Compare the classification effects before and after channel selection through the accuracy curve; the experimental results are as follows Figure 6 , you can see Figure 6 The ReliefF-MR algorithm (black solid line) was used to select channels for the input network dataset, which significantly improved the performance in the EEG-TCN-LSTMt network.

[0392] 2: Compare the optimal channel number before and after channel selection through the optimal channel number curve; the experimental results are as follows Figure 7 , you can see Figure 7 The number of data channels input to the network can be significantly reduced by the ReliefF-MR algorithm (black solid line);

[0393] Example 3:

[0394] EEG signals are a special type of time series data, and their characteristic information has three dimensions: time domain, frequency domain, and spatial domain. Therefore, EEG signal classification algorithms require a method that can fully extract the multidimensional characteristics of EEG signals;

[0395] 1: The improved EEGNet convolutional neural network constructed in step 3 first extracts the frequency features of the motor imagery signal from the time axis through two-dimensional convolution. Then, deep convolution is used to learn spatial filters of specific frequencies to extract channel-related spatial domain features. Finally, depthwise separable convolution is used to output the final feature map through depthwise convolution and pointwise convolution. The frequency domain features of the motor imagery signal are generated by one-dimensional convolution and depthwise classifiable convolution, the spatial domain features are generated based on depthwise convolution, and the time domain features are generated according to the time dimension.

[0396] 2: The output feature map of step 3 still contains time information, so the feature vectors obtained by the improved EEGNet convolutional neural network are input in parallel into the improved TCN temporal convolutional neural network and the improved LSTM recurrent neural network RNN ​​to extract time domain features. The improved TCN temporal convolutional neural network introduces dilated causal convolution and residual connection to obtain time domain feature information across time steps; the improved LSTM recurrent neural network RNN ​​learns the long-term and short-term related feature information of the motor imagery signal. The local features extracted by the improved TCN temporal convolutional neural network and the improved LSTM recurrent neural network RNN ​​are then spliced ​​and fused to form a global information feature map containing the time, frequency and spatial domains of the motor imagery signal;

[0397] The network model consisting of the improved EEGNet convolutional neural network, the improved TCN temporal convolutional neural network, the improved LSTM recurrent neural network RNN ​​and the classifier is named the EEG-TCN-LSTM convolutional neural network model. In order to evaluate the impact of the improved EEGNet convolutional neural network, the improved TCN temporal convolutional neural network and the improved LSTM recurrent neural network RNN ​​on the overall performance of the EEG-TCN-LSTM convolutional neural network model, an ablation experiment was designed to analyze the results.

[0398] 1: We selected EEGNet as the baseline experimental model, and added TCN modules or LSTM modules to it, and added these two modules in parallel to conduct ablation experiments;

[0399] 2: Experimental results are as follows Figure 8, we can see that the improved EEGNet convolutional neural network's classification effect is average, but the improved EEGNet convolutional neural network with the serial addition of the TCN module + the improved TCN temporal convolutional neural network is able to extract effective identification information containing temporal features from the output feature map of the improved EEGNet convolutional neural network across time steps, improving the average accuracy to 70.93%. Similarly, the improved EEGNet convolutional neural network with the serial addition of the LSTM module + the improved LSTM recurrent neural network RNN ​​can further focus on the long-term and short-term temporal features of the motor imagery signal, improving the average accuracy by 14%. The EEGNet convolutional neural network + the improved TCN temporal convolutional neural network + the improved LSTM recurrent neural network RNN ​​achieves the highest average accuracy of 73.07%.

[0400] 3: Such as Figure 9a 、 9b The confusion matrices represent the prediction results of subject A3 and subject A7 under the EEG-TCN-LSTM model. It can be seen that in the confusion matrix of the four categories, the model has little effect on the classification effect of each type of motor imagery task. It can classify each type of motor imagery task indiscriminately with a low error rate, indicating that the extracted features are stable and effective.

[0401] 4: To further verify the effectiveness of the model in the motor imagery classification task, the t-Distributed Stochastic Neighbor Embedding (t-SNE) method is used to visualize the EEG features extracted by the EEG-TCN-LSTM model. Figure 10a 、 10b As shown in the figure, the four colors represent the four types of motor imagery tasks to which the EEG features belong. It can be seen that after the feature extraction of the EEG-TCN-LSTM model, the EEG features of different categories have a high concentration, indicating that the recognition effect of the model proposed in this paper is very good;

[0402] A login interface for a motor imagery EEG signal classification system based on deep learning Figure 11 As shown, the channel selection algorithm interface is as follows Figure 12 As shown, the feature extraction and classification interface is as follows Figure 13a 、 Figure 13b shown.

[0403] The present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A motor imagery EEG signal classification method based on deep learning, characterized by: The specific process of the method is: Step 1: Obtain the original EEG signal dataset; Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector; Step 2: Perform channel selection on the original two-dimensional EEG signal feature vector to obtain a new two-dimensional EEG signal feature vector set; Step 3: Build an EEG-TCN-LSTM convolutional neural network; The EEG-TCN-LSTM convolutional neural network is a multi-domain feature fusion network; The EEG-TCN-LSTM convolutional neural network includes: improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN, and classifier; The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depthwise convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer; The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, and ELU activation function; The TCN residual module includes: TCN residual block 1 and TCN residual block 2; TCN residual block 1 includes: The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer; TCN residual block 2 includes: a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer; The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence; The classifier consists of a fully connected layer and the first Softmax activation function layer in sequence; The working process of the EEG-TCN-LSTM convolutional neural network is: The new two-dimensional EEG signal feature vector obtained in step 2 is input into the improved EEGNet convolutional neural network, and the improved EEGNet convolutional neural network outputs the two-dimensional EEG frequency-spatial domain feature X fs ; Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved TCN temporal convolutional neural network, and the improved TCN temporal convolutional neural network outputs the two-dimensional EEG time domain feature X t ; Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved LSTM recurrent neural network RNN, and the improved LSTM recurrent neural network RNN ​​outputs the two-dimensional EEG time domain feature X l ; The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X; Flatten the concatenated frequency-time-space domain feature X to obtain feature X′; The feature X′ is input into the classifier, and the classifier outputs the predicted probability of each motor imagery category; Step 4: Obtain the trained EEG-TCN-LSTM convolutional neural network; Step 5: Obtain the subject's EEG signal to be tested; Preprocessing the EEG signal data to be measured into a two-dimensional EEG signal feature vector; Perform channel selection on the original two-dimensional EEG signal feature vector to obtain a new two-dimensional EEG signal feature vector set; The two-dimensional vector is input into the trained EEG-TCN-LSTM convolutional neural network, and the trained EEG-TCN-LSTM convolutional neural network outputs the predicted probability of each motor imagery category.

2. The method for classifying motor imagery EEG signals based on deep learning according to claim 1, characterized in that: In step 1, the original EEG signal dataset is obtained; Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector; The specific process is: Obtain the original EEG signal dataset from the BCI2008 IV-2a database; The MNE library is used to process the raw EEG signal into a two-dimensional raw EEG signal feature vector.

3. The method for classifying motor imagery EEG signals based on deep learning according to claim 2, characterized in that: In step 2, channel selection is performed on the two-dimensional original EEG signal feature vector obtained in step 1 to obtain a new two-dimensional EEG signal feature vector set; The specific process is: Step 2.1: Calculate the correlation statistic d of the two-dimensional original EEG signal feature vectors using the ReliefF algorithm, sort the two-dimensional original EEG signal feature vectors in ascending order according to the correlation statistic, and remove all two-dimensional original EEG signal feature vectors with correlation statistics less than 0 to obtain the set A of two-dimensional original EEG signal feature vectors; Step 2.2: Calculate the redundancy R1 of the two-dimensional original EEG signal feature vector set A; Selecting the eigenvector t with the smallest correlation statistic in the two-dimensional original EEG signal eigenvector set A, removing the eigenvector t from the two-dimensional original EEG signal eigenvector set A to obtain the two-dimensional original EEG signal eigenvector set B; calculating the redundancy R2 of the two-dimensional original EEG signal eigenvector set B; Compare the redundancy R1 and redundancy R2: If the redundancy R2 is less than the redundancy R1, it means that the redundancy cannot be reduced, and the two-dimensional original EEG signal feature vector set A is the optimal feature set, which is used as the new two-dimensional EEG signal feature vector set; If the redundancy R2 is greater than or equal to the redundancy R1, the two-dimensional original EEG signal feature vector set B is used as the new two-dimensional original EEG signal feature vector set A, and step 2.2 is repeated.

4. The method for classifying motor imagery EEG signals based on deep learning according to claim 3, characterized in that: The processing process of improving the EEGNet convolutional neural network in step 3 is as follows: 1) Input the two-dimensional EEG signal feature vector obtained in step 2 into the input layer of the improved EEGNet convolutional neural network. The shape of the EEG signal output by the input layer is [1, C, T]; 1 is the number of characteristic channels, C is the number of electrode channels, and T is the number of time points; 2) Input the EEG signal output by the input layer into the one-dimensional convolution layer, and the one-dimensional convolution layer outputs the feature X2; input the feature X2 output by the one-dimensional convolution layer into the first activation function layer, and the first activation function layer outputs the feature; Specifically: Set the one-dimensional convolution layer to a convolution filter with 8 convolution kernels, a convolution kernel size of [1,32], and a stride size of [1,1]; Set the filling method to zero filling of the same shape; The EEG signal output by the input layer is input into the one-dimensional convolution layer, and the one-dimensional convolution layer outputs the feature X2. The shape of the feature X2 is [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points. The one-dimensional convolution layer output feature X2 is input into the Linear activation function layer, and the Linear activation function layer outputs the feature; 3) Input the output feature of the first activation function layer into the first batch normalization layer, and the first batch normalization layer outputs feature X3; specifically: Perform batch normalization on the output features of the first activation function layer, and obtain the shape of feature X3 as [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points; 4) Input the output feature X3 of the first batch normalization layer into the depth convolution layer, and the depth convolution layer outputs the feature X4; input the output feature X4 of the depth convolution layer into the second activation function layer, and the second activation function layer outputs the feature; Specifically: Set the depthwise convolutional layer to a convolution filter with 2 kernels, kernel size [C, 1], and stride size [1, 1]. Set the filling method to zero filling of the same shape; The output feature X3 of the first batch normalization layer is input into the depth convolution layer, and the depth convolution layer outputs feature X4. The shape of feature X4 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width; The output feature X4 of the deep convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature; 5) Input the output feature of the second activation function layer into the second batch normalization layer, and the second batch normalization layer outputs feature X5; specifically: The output features of the second activation function layer are batch normalized to obtain the shape of feature X5 as [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width; 6) Input the output feature X5 of the second batch normalization layer into the third activation function layer, and the third activation function layer outputs feature X6; specifically: Use the ELU activation function to process the second batch normalization layer output feature X5, and the shape of the feature X6 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width; 7) Input the output feature X6 of the third activation function layer into the first average pooling layer, and the first average pooling layer outputs feature X7; specifically: Set the pooling window size of the first average pooling layer to [1,8] and the stride size to [1,8]; The output feature X6 of the third activation function layer is processed by the first average pooling layer, and the shape of the feature X7 is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width; 8) Input the output feature X7 of the first average pooling layer into the first Dropout layer, and the first Dropout layer outputs feature X8; specifically: Set the Dropout probability to 0.2; The first average pooling layer outputs feature X7, which is then passed through the first Dropout layer to obtain feature X8 with a shape of [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width. 9) Input the output feature X8 of the first Dropout layer into the separable convolution layer, and the separable convolution layer outputs X9; input the output feature X9 of the separation convolution layer into the fourth activation function layer, and the fourth activation function layer outputs the feature; Specifically: Set the separable convolution layer to a convolution filter with 16 kernels, a kernel size of [1,16], and a stride size of [1,1]; The output feature X8 of the first Dropout layer is input into the separable convolution layer, and the separable convolution layer outputs X9. The shape of feature X9 is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width; The output feature X9 of the separation convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature; 10) Input the output features of the fourth activation function layer into the third batch normalization layer, and the third batch normalization layer outputs features X 10 Specifically: Perform batch normalization on the output features of the fourth activation function layer to obtain feature X 10 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width; 11) Output feature X of the third batch normalization layer 10 Input the fifth activation function layer, the fifth activation function layer outputs feature X 11 Specifically: The third batch normalization layer outputs feature X 10 Input ELU activation function layer, ELU activation function layer output feature X 11 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width; 12) Output feature X of the fifth activation function layer 11 Input the second average pooling layer, the second average pooling layer outputs feature X 12 Specifically: Set the pooling window size of the second average pooling layer to [1,8] and the stride size to [1,8]; The fifth activation function layer outputs feature X 11 After the second average pooling layer operation, feature X is obtained 12 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width; 13) Output feature X of the second average pooling layer 12 Input the second Dropout layer, the second Dropout layer outputs feature X 13 Specifically: Set the Dropout probability to 0.2; The second average pooling layer outputs feature X 12 After the second Dropout layer operation, feature X is obtained 13 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width; Feature X 13 As the frequency-spatial domain feature X extracted by the improved EEGNet convolutional neural network fs , called the two-dimensional EEG frequency-space domain feature X fs .

5. The method for classifying motor imagery EEG signals based on deep learning according to claim 4, characterized in that: The processing process of the improved TCN temporal convolutional neural network is as follows: 1) Improve the two-dimensional EEG frequency-spatial domain feature X output by the EEGNet convolutional neural network fs Input the input layer of the improved TCN temporal convolutional neural network, and the shape of the EEG signal output by the input layer is [16, 1, T / 64]; 16 is the number of channels, 1 is the height, and T / 64 is the width; 2) Input the EEG signal output by the input layer in 1) into the TCN residual module. The shape of the EEG signal output by the TCN residual module is [12, T / 64]. Input the EEG signal output by the TCN residual module into the ELU activation function. The ELU activation function outputs features. 12 is the number of channels, T / 64 is the width; The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t ; Specifically: The TCN residual module includes: TCN residual block 1 and TCN residual block 2; TCN residual block 1 includes: The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer; TCN residual block 2 includes: a third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer; The first dilated causal convolution layer, the second dilated causal convolution layer, the third dilated causal convolution layer, and the fourth dilated causal convolution layer are all set to use convolution filters with 12 convolution kernels and a convolution kernel size of [1, 4]. The EEG signal output by the input layer in 1) is input into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]. The EEG signal output by the TCN residual module is input into the ELU activation function, and the ELU activation function outputs features. The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t .

6. The method for classifying motor imagery EEG signals based on deep learning according to claim 5, characterized in that: The processing process of the improved LSTM recurrent neural network RNN ​​is as follows: 1) The two-dimensional EEG frequency-spatial domain feature X fs Input the input layer of the improved LSTM recurrent neural network RNN. The shape of the EEG signal output by the input layer is [16, 1, T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width; 2) Input the EEG signal output by the input layer in 1) into the LSTM layer. The shape of the EEG signal output by the LSTM layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically: Set the number of neurons in the hidden layer of the LSTM module to 20; The EEG signal output from the input layer in 1) is processed through the LSTM layer. The resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width. 3) Input the EEG signal output by the LSTM layer in 2) into the seventh Dropout layer. The shape of the EEG signal output by the seventh Dropout layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically: Set the seventh Dropout probability to 0.5; The EEG signal output by the LSTM layer in 2) is passed through the seventh Dropout layer. The resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width. The seventh Dropout layer outputs the EEG signal as the time domain feature extracted by the improved LSTM recurrent neural network RNN, which is called the two-dimensional EEG time domain feature X l .

7. The method for classifying motor imagery EEG signals based on deep learning according to claim 6, characterized in that: The processing process of the classifier is: 1) Input feature X' into the fully connected layer of the classifier. The fully connected layer outputs data in the shape of [4], where 4 is the unnormalized score of each motor imagery category. 2) Input the output data of the fully connected layer into the first Softmax activation function layer of the classifier. The output data shape of the first Softmax activation function layer is [4], where 4 is the predicted probability of each motor imagery category; 4 represents left hand, right hand, feet, and tongue.

8. The method for classifying motor imagery EEG signals based on deep learning according to claim 7, characterized in that: In step 4, the trained EEG-TCN-LSTM convolutional neural network is obtained; the specific process is as follows: Cross entropy is used as the loss function, and the loss function is optimized by the Adam optimizer until convergence to obtain the trained EEG-TCN-LSTM convolutional neural network.

9. A motor imagery EEG signal classification system based on deep learning, characterized by: The system is used to execute the motor imagery EEG signal classification method based on deep learning as described in any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the deep learning-based motor imagery EEG signal classification method according to any one of claims 1 to 8.

Images