Motor imagery electroencephalogram signal classification method and system based on deep learning and storage medium

Abstract

The invention discloses a motor imagery electroencephalogram signal classification method and system based on deep learning and a storage medium, belongs to the field of brain-computer interfaces, and particularly relates to the motor imagery electroencephalogram signal classification method and system based on deep learning and the storage medium. The objective of the invention is to solve the problem of low motor imagery electroencephalogram signal classification and recognition accuracy caused by low signal-to-noise ratio of electroencephalogram signals and the like. The method comprises the following steps: acquiring a two-dimensional original electroencephalogram signal feature vector; performing channel selection on the signal feature vector; an EEG-TCN-LSTM convolutional neural network is constructed; the trained EEG-TCN-LSTM is obtained; the method comprises the following steps: acquiring a to-be-tested two-dimensional electroencephalogram signal feature vector of a subject, performing channel selection on the two-dimensional original electroencephalogram signal feature vector to obtain a new feature vector set, inputting a two-dimensional vector into a trained EEG-TCN-LSTM, and outputting a prediction probability of each motor imagery category by the trained EEG-TCN-LSTM.

Description

Technical Field

[0001] The present invention belongs to the field of brain-computer interface, and specifically relates to a motor imagery EEG signal classification method, system and storage medium based on deep learning. Background Art

[0002] As we all know, the brain is the most complex biological organ in the human body, supporting various complex human cognition, perception, emotion, memory, movement and other functions, and playing an important role in physiological and psychological processes. There are more and more studies on brain science, among which the brain-computer interface (BCI) is the key direction of brain science and brain-like research. The brain-computer interface refers to a new type of real-time communication and control system established between the biological (human or animal) brain and external devices or the environment, thereby realizing direct interaction between the brain and external devices.

[0003] After nearly 50 years of research, brain-computer interface technology has been continuously developed and improved, and the research direction of brain-computer interface has gradually been clearly distinguished according to the acquisition method and signal type of EEG signals. Among them, the brain-computer interface system based on motor imagery (MI) EEG signals has always been a hot topic of research. Motor imagery is one of the spontaneous EEG signals. The signal can be collected using electroencephalogram (EEG) equipment, which is simple to operate and easy to implement. The subject spontaneously imagines a certain action according to the prompts on the screen without actually moving. In this process, the motor cortex of the brain will produce a specific EEG signal with a relatively obvious amplitude, namely the motor imagery EEG signal. The study of motor imagery EEG signals is of great significance in the fields of neurorehabilitation and brain-computer interface. By decoding the subject's movement intention and converting the recognition result into a command to control the external device, "mind control" can be achieved, which provides a new treatment method for patients with limited mobility.

[0004] By detecting the user's brain neural activity, the brain-computer interface system can monitor, replace, improve / restore, enhance and supplement in other aspects such as medical treatment, education, entertainment, smart home, engineering, etc. The brain-computer interface system based on motor imagery EEG signals also has many applications in these fields: medical field, education field, engineering field, etc.

[0005] In summary, motor imagery classification technology plays an important role in many fields and has become a cutting-edge technology that countries are competing to invest in research and development. At the same time, motor imagery classification technology is an interdisciplinary research field, a complex system engineering, with great scientific research value and broad application prospects. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of low accuracy in classification and recognition of motor imagery EEG signals due to low signal-to-noise ratio of EEG signals, and to propose a motor imagery EEG signal classification method, system and storage medium based on deep learning.

[0007] A motor imagery EEG signal classification method based on deep learning. The specific process is as follows:

[0008] Step 1: Obtain the original EEG signal dataset;

[0009] Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector;

[0010] Step 2: Perform channel selection on the original two-dimensional EEG signal feature vector to obtain a new two-dimensional EEG signal feature vector set;

[0011] Step 3: Construct EEG-TCN-LSTM convolutional neural network;

[0012] The EEG-TCN-LSTM convolutional neural network is a multi-domain feature fusion network;

[0013] The EEG-TCN-LSTM convolutional neural network includes: improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN, and classifier;

[0014] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depth convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0015] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, ELU activation function;

[0016] The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence;

[0017] TCN residual block 1 includes:

[0018] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0019] TCN residual block 2 includes:

[0020] A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0021] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0022] The classifier includes a fully connected layer and a first Softmax activation function layer in sequence;

[0023] The working process of the EEG-TCN-LSTM convolutional neural network is:

[0024] The new two-dimensional EEG signal feature vector obtained in step 2 is input into the improved EEGNet convolutional neural network, and the improved EEGNet convolutional neural network outputs a two-dimensional EEG frequency-spatial domain feature X fs ;

[0025] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved TCN temporal convolutional neural network, and the improved TCN temporal convolutional neural network outputs the two-dimensional EEG time domain feature X t ;

[0026] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved LSTM recurrent neural network RNN, and the improved LSTM recurrent neural network RNN ​​outputs the two-dimensional EEG time domain feature X l ;

[0027] The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0028] Flatten the concatenated frequency-time-space domain feature X to obtain feature X′;

[0029] The feature X′ is input into the classifier, and the classifier outputs the predicted probability of each motor imagery category;

[0030] Step 4: Obtain the trained EEG-TCN-LSTM convolutional neural network;

[0031] Step 5: Obtain the subject's EEG signal to be tested;

[0032] Preprocessing the EEG signal data to be tested into a two-dimensional EEG signal feature vector;

[0033] Perform channel selection on the original two-dimensional EEG signal feature vector to obtain a new two-dimensional EEG signal feature vector set;

[0034] The two-dimensional vector is input into the trained EEG-TCN-LSTM convolutional neural network, and the trained EEG-TCN-LSTM convolutional neural network outputs the predicted probability of each motor imagery category.

[0035] A motor imagery EEG signal classification system based on deep learning is used to execute a motor imagery EEG signal classification method based on deep learning.

[0036] A computer storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the motor imagery EEG signal classification method based on deep learning.

[0037] The beneficial effects of the present invention are:

[0038] The present invention designs and implements a motor imagery EEG signal classification method and system based on deep learning, adopts the ReliefF-MR algorithm to perform channel selection on the feature vector of the two-dimensional original EEG signal, extracts the frequency and spatial domain features of the preprocessed EEG signal through an improved EEGNet convolutional neural network, and then superimposes the two time domain features extracted from the EEG signal through an improved TCN temporal convolutional neural network combined with an improved LSTM recurrent neural network RNN; uses a feedforward neural network classifier to perform the final intention recognition, thereby improving the classification and recognition accuracy of motor imagery EEG signals.

[0039] The RelidfF-MR channel selection algorithm designed in the present invention realizes the optimal channel selection of motor imagery EEG signals;

[0040] The network model designed by the present invention has better classification effect than other network models such as FBCSP;

[0041] The method of the present invention is evaluated on the BCI2008 IV-2a data set. The results show that the classification accuracy of the method of the present invention is 73.07%, which can effectively complete the classification of EEG signals and enable the classification effect to be visualized on the web page. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of the present invention;

[0043] Figure 2 It is an improved EEGNet convolutional neural network architecture diagram;

[0044] Figure 3 It is an improved TCN temporal convolutional neural network architecture diagram;

[0045] Figure 4a It is an improved LSTM recurrent neural network RNN ​​graph;

[0046] Figure 4b This is the LSTM unit structure diagram;

[0047] Figure 5 This is the detailed diagram of the EEG-TCN-LSTM convolutional neural network structure;

[0048] Figure 6 This is a comparison chart of classification accuracy of different channel selection algorithms;

[0049] Figure 7 This is a comparison chart of the optimal number of channels for different channel selection algorithms;

[0050] Figure 8 It is the bar graph of the average decoding accuracy and Kappa coefficient of the subjects with different models in the ablation experiment;

[0051] Figure 9a This is the confusion matrix diagram of subject A3 under the EEG-TCN-LSTM convolutional neural network model;

[0052] Figure 9b This is the confusion matrix diagram of subject A7 under the EEG-TCN-LSTM convolutional neural network model;

[0053] Fig.10a This is the t-SNE visualization result of subject A3;

[0054] Fig.10b This is the t-SNE visualization result of subject A7;

[0055] Fig.11 This is the login interface of the motor imagery EEG signal classification system based on deep learning;

[0056] Fig.12 This is the interface diagram of the channel selection algorithm of the motor imagery EEG signal classification system based on deep learning;

[0057] Fig.13a This is the main interface diagram of feature extraction and classification of the motor imagery EEG signal classification system based on deep learning;

[0058] Fig.13b This is the interface diagram of feature extraction and classification details of the motor imagery EEG signal classification system based on deep learning. DETAILED DESCRIPTION

[0059] Specific implementation method 1: Combination Figure 1 This embodiment describes a method for classifying motor imagery EEG signals based on deep learning. The specific process is as follows:

[0060] Step 1: Obtain the original EEG signal dataset;

[0061] Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector;

[0062] Step 2: In order to make full use of the effective information of the electrode channels and reduce the influence of irrelevant channels on the classification results, and further improve the classification accuracy of motor imagery signals, channel selection is performed on the original data set;

[0063] The ReliefF-MR algorithm is used to select channels of the original two-dimensional EEG signal feature vectors to obtain a new set of two-dimensional EEG signal feature vectors.

[0064] Step 3: Construct EEG-TCN-LSTM convolutional neural network;

[0065] The EEG-TCN-LSTM convolutional neural network is a multi-domain feature fusion network;

[0066] The EEG-TCN-LSTM convolutional neural network includes: improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN, and classifier;

[0067] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depth convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0068] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, ELU activation function;

[0069] The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence;

[0070] TCN residual block 1 includes:

[0071] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0072] TCN residual block 2 includes:

[0073] A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0074] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0075] The classifier includes a fully connected layer and a first Softmax activation function layer in sequence;

[0076] The working process of the EEG-TCN-LSTM convolutional neural network is:

[0077] The new two-dimensional EEG signal feature vector obtained in step 2 is input into the improved EEGNet convolutional neural network, and the improved EEGNet convolutional neural network outputs a two-dimensional EEG frequency-spatial domain feature X fs ;

[0078] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved TCN temporal convolutional neural network, and the improved TCN temporal convolutional neural network outputs the two-dimensional EEG time domain feature X t ;

[0079] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved LSTM recurrent neural network RNN, and the improved LSTM recurrent neural network RNN ​​outputs the two-dimensional EEG time domain feature X l ;

[0080] The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0081] Flatten the concatenated frequency-time-space domain feature X to obtain feature X′;

[0082] The feature X′ is input into the classifier, and the classifier outputs the predicted probability of each motor imagery category;

[0083] Step 4: Obtain the trained EEG-TCN-LSTM convolutional neural network;

[0084] Step 5: Obtain the subject's EEG signal to be tested (the subject uses 22 Ag / AgCl electrodes (the electrode spacing is 3.5 cm) to record the subject's EEG signal to be tested;);

[0085] Preprocessing the EEG signal data to be tested into a two-dimensional EEG signal feature vector;

[0086] The ReliefF-MR algorithm is used to select channels of the original two-dimensional EEG signal feature vectors to obtain a new set of two-dimensional EEG signal feature vectors.

[0087] The two-dimensional vector is input into the trained EEG-TCN-LSTM convolutional neural network, and the trained EEG-TCN-LSTM convolutional neural network outputs the predicted probability of each motor imagery category.

[0088] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the original EEG signal data set is obtained in step 1;

[0089] Preprocessing the raw EEG signal data into a two-dimensional raw EEG signal feature vector;

[0090] The specific process is:

[0091] Obtain the original EEG signal dataset from the BCI2008 IV-2a database (public dataset);

[0092] The MNE library is used to process the raw EEG signal into a two-dimensional raw EEG signal feature vector.

[0093] Step 1.1: The original EEG data set consists of EEG data of 9 subjects;

[0094] Each subject performed two experiments on two different days; each experiment consisted of 6 runs with a short break between each run. One run consisted of 48 trials (12 for each of the four possible categories), for a total of 288 trials per experiment;

[0095] Each subject sat in a comfortable armchair in front of a computer screen, and EEG was recorded using 22 Ag / AgCl electrodes (electrode spacing of 3.5 cm); all signals were recorded in a monopolar manner, with the left mastoid as the reference electrode and the right mastoid as the ground electrode; the signal was sampled at 250 Hz and band-pass filtered from 0.5 Hz to 100 Hz; the amplifier sensitivity was set to 100 μV, and a 50 Hz notch filter was enabled to suppress power supply noise;

[0096] At the beginning of each experiment, approximately 5 minutes of recordings were performed to estimate the effects of the electrooculogram (EOG). The recordings were divided into three parts: (1) 2 minutes with eyes open (looking at a fixation cross on the screen), (2) 1 minute with eyes closed, and (3) 1 minute with eye movements;

[0097] At the beginning of a trial, a fixation cross appears on a black screen. In addition, a brief audio prompt is played. Two seconds later, an arrow prompt appears on the screen, pointing to the left, right, down, or up (corresponding to four categories: left hand, right hand, feet, or tongue), and the arrow remains on the screen for 1.25 seconds. This prompts the subject to perform the required motor imagery task. No feedback is provided during the trial. The subject is required to continue the motor imagery task until the fixation cross disappears from the screen. This is followed by a short break, when the screen turns black again and the next experiment is carried out;

[0098] Step 1.2: Each subject will generate two EEG signal samples, and the EEG signal data is represented as a two-dimensional vector; the signal feature dimension is [1, 22, 1125], where 22 is the number of electrode channels in the original EEG signal, 1125 is the number of time points, and 1 is the expanded dimension of the channel number;

[0099] Next, input is given to the EEGNet structure, TCN structure, and LSTM structure for feature learning.

[0100] The other steps and parameters are the same as those in the first embodiment.

[0101] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that, in step 2, in order to make full use of the effective information of the electrode channel and reduce the influence of irrelevant channels on the classification results, and further improve the classification accuracy of motor imagery signals, channel selection is performed on the original data set;

[0102] The ReliefF-MR algorithm is used to perform channel selection on the two-dimensional original EEG signal feature vector obtained in step 1 to obtain a new two-dimensional EEG signal feature vector set;

[0103] Verify the improvement of channel selection effect in the proposed improved deep learning model in step 3;

[0104] The specific process is:

[0105] Step 2.1: Calculate the correlation statistic d of the two-dimensional original EEG signal feature vector using the ReliefF algorithm, sort the two-dimensional original EEG signal feature vectors from small to large according to the correlation statistic, remove all the two-dimensional original EEG signal feature vectors whose correlation statistics are less than 0, and obtain the two-dimensional original EEG signal feature vector set A;

[0106] Step 2.2: Calculate the redundancy R1 of the two-dimensional original EEG signal feature vector set A;

[0107] Select the feature vector t with the smallest correlation statistic in the two-dimensional original EEG signal feature vector set A, remove the feature vector t from the two-dimensional original EEG signal feature vector set A, and obtain the two-dimensional original EEG signal feature vector set B; calculate the redundancy R2 of the two-dimensional original EEG signal feature vector set B;

[0108] Compare the size of redundancy R1 and redundancy R2:

[0109] If the redundancy R2 is less than the redundancy R1, it means that the redundancy cannot be reduced, and the two-dimensional original EEG signal feature vector set A is the optimal feature set, which is used as the new two-dimensional EEG signal feature vector set;

[0110] If the redundancy R2 is greater than or equal to the redundancy R1, the two-dimensional original EEG signal feature vector set B is used as the new two-dimensional original EEG signal feature vector set A, and step 2.2 is repeated.

[0111] The other steps and parameters are the same as those in the first or second embodiment.

[0112] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that the processing process of improving the EEGNet convolutional neural network in step 3 is:

[0113] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depth convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0114] Improve the output data of the i′th layer of the EEGNet convolutional neural network with X i′ Indicates that i′=1,2,3,...,16, the first layer is the input layer, the second layer is the one-dimensional convolution layer, the third layer is the first activation function layer, the fourth layer is the first batch normalization layer, the fifth layer is the depthwise convolution layer, the sixth layer is the second activation function layer, the seventh layer is the second batch normalization layer, the eighth layer is the third activation function layer, the ninth layer is the first average pooling layer, the tenth layer is the first Dropout layer, the eleventh layer is the separable convolution layer, the twelfth layer is the fourth activation function layer, the thirteenth layer is the third batch normalization layer, the fourteenth layer is the fifth activation function layer, the fifteenth layer is the second average pooling layer, and the sixteenth layer is the second Dropout layer;

[0115] 1) Input the two-dimensional EEG signal feature vector obtained in step 2 into the input layer of the improved EEGNet convolutional neural network, and the shape of the EEG signal output by the input layer is [1, C, T];

[0116] 1 is the number of characteristic channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125);

[0117] The number of electrode channels C depends on the feature vector extracted from the original EEG signal by the ReliefF-MR algorithm. For the BCI2008 IV-2a dataset, C is 22;

[0118] 2) Input the EEG signal output by the input layer into the one-dimensional convolution layer, and the one-dimensional convolution layer outputs the feature X2; input the feature X2 output by the one-dimensional convolution layer into the first activation function layer, and the first activation function layer outputs the feature; specifically:

[0119] Set the one-dimensional convolution layer to a convolution filter with 8 convolution kernels, a convolution kernel size of [1,32], and a stride size of [1,1]. The stride represents the x- and y-movement distances of the convolution kernel.

[0120] Set the padding method to zero padding of the same shape, which keeps the sample shape unchanged in the convolution calculation;

[0121] The EEG signal output by the input layer is input into the one-dimensional convolution layer. The one-dimensional convolution layer outputs feature X2. The shape of feature X2 is [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points (such as 1125).

[0122] The one-dimensional convolution layer output feature X2 is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0123] The mathematical expression of the Linear activation function is as follows:

[0124] f(x)=x

[0125] 3) Input the output feature of the first activation function layer into the first batch normalization layer, and the first batch normalization layer outputs feature X3; specifically:

[0126] The output features of the first activation function layer are batch normalized to obtain feature X3 with a shape of [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125);

[0127] 4) Input the output feature X3 of the first batch normalization layer into the deep convolution layer, and the deep convolution layer outputs the feature X4; input the output feature X4 of the deep convolution layer into the second activation function layer, and the second activation function layer outputs the feature; specifically:

[0128] Set the depth convolutional layer to a convolution filter with 2 kernels, kernel size [C, 1], and stride size [1, 1], where the stride represents the x and y movement distances of the kernel.

[0129] Set the padding method to zero padding of the same shape, which keeps the sample shape unchanged in the convolution calculation;

[0130] The output feature X3 of the first batch normalization layer is input into the deep convolution layer, and the deep convolution layer outputs feature X4. The shape of feature X4 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0131] The output feature X4 of the deep convolutional layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0132] 5) Input the output feature of the second activation function layer into the second batch normalization layer, and the second batch normalization layer outputs feature X5; specifically:

[0133] Perform batch normalization on the output features of the second activation function layer, and obtain the shape of feature X5 as [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0134] 6) Input the output feature X5 of the second batch normalization layer into the third activation function layer, and the third activation function layer outputs feature X6; specifically:

[0135] Use the ELU activation function to process the second batch normalization layer output feature X5, and the shape of feature X6 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0136] The mathematical expression of the activation function ELU is as follows:

[0137]

[0138] 7) Input the output feature X6 of the third activation function layer into the first average pooling layer, and the first average pooling layer outputs feature X7; specifically:

[0139] Set the pooling window size of the first average pooling layer to [1,8] and the stride size to [1,8];

[0140] The output feature X6 of the third activation function layer is subjected to the first average pooling layer operation, and the shape of the feature X7 obtained is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140);

[0141] 8) Input the output feature X7 of the first average pooling layer into the first Dropout layer, and the first Dropout layer outputs feature X8; specifically:

[0142] Set the Dropout probability to 0.2;

[0143] The first average pooling layer outputs feature X7, which is then processed by the first Dropout layer to obtain feature X8 of shape [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140).

[0144] 9) Input the output feature X8 of the first Dropout layer into the separable convolution layer, and the separable convolution layer outputs X9; input the output feature X9 of the separation convolution layer into the fourth activation function layer, and the fourth activation function layer outputs the feature;

[0145] Specifically:

[0146] Set the separable convolution layer to a convolution filter with 16 convolution kernels, a convolution kernel size of [1,16], and a stride size of [1,1]. The stride represents the x- and y-movement distances of the convolution kernel.

[0147] The output feature X8 of the first Dropout layer is input into the separable convolution layer, and the separable convolution layer outputs X9. The shape of feature X9 is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g. 140);

[0148] The output feature X9 of the separation convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0149] 10) Input the output feature of the fourth activation function layer into the third batch normalization layer, and the third batch normalization layer outputs feature X 10 ; Specifically:

[0150] Perform batch normalization on the output features of the fourth activation function layer to obtain feature X 10 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g. 140);

[0151] 11) Output feature X of the third batch normalization layer 10 Input the fifth activation function layer, the fifth activation function layer outputs feature X 11 ; Specifically:

[0152] The third batch normalization layer outputs feature X 10 Input ELU activation function layer, ELU activation function layer output feature X 11 The shape is [16,1,T / 8], 16 is the number of channels, 1 is the height, and T / 8 is the width;

[0153] 12) Output feature X of the fifth activation function layer 11 Input the second average pooling layer, the second average pooling layer outputs feature X 12 ; Specifically:

[0154] Set the pooling window size of the second average pooling layer to [1,8] and the stride size to [1,8];

[0155] The fifth activation function layer outputs feature X 11 After the second average pooling layer operation, feature X is obtained 12 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width (e.g. 17);

[0156] 13) Output feature X of the second average pooling layer 12 Input the second Dropout layer, the second Dropout layer outputs feature X 13 ; Specifically:

[0157] Set the Dropout probability to 0.2;

[0158] The second average pooling layer outputs feature X 12 After the second Dropout layer operation, we get feature X 13 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0159] Feature X 13 As the frequency-spatial domain feature X extracted by the improved EEGNet convolutional neural network fs , called the two-dimensional EEG frequency-space domain feature X fs .

[0160] In the basic model, adjust the number of convolutional layers, pooling layers and learning rate, distinguish the optimal network structure through classification accuracy, loss rate, ROC curve and confusion matrix, and analyze and compare the experimental results;

[0161] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0162] Specific implementation mode 5: This implementation mode is different from the specific implementation modes 1 to 4 in that the processing process of the improved TCN temporal convolutional neural network is:

[0163] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, ELU activation function;

[0164] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0165] Improve the output data of the i-th layer of the TCN temporal convolutional neural network structure with X i , i=1,2,3, the first layer is the input layer, the second layer is the TCN residual module, and the third layer is the ELU activation function;

[0166] The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence;

[0167] TCN residual block 1 includes:

[0168] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0169] TCN residual block 2 includes:

[0170] A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0171] The mathematical expression of dilated causal convolution is as follows:

[0172]

[0173] Where d is the dilation factor, k is the convolution kernel size, sd·i represents the past direction, and x∈R n Is a one-dimensional sequence input, filter f:{0,…,k-1}→R.

[0174] The mathematical expression of the activation function ReLU is as follows:

[0175]

[0176] 1) Improve the two-dimensional EEG frequency-spatial domain feature X output by the EEGNet convolutional neural network fs Input the input layer of the improved TCN temporal convolutional neural network, and the shape of the EEG signal output by the input layer is [16, 1, T / 64];

[0177] 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0178] 2) Input the EEG signal output by the input layer in 1) into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]; Input the EEG signal output by the TCN residual module into the ELU activation function, and the ELU activation function outputs features; 12 is the number of channels, and T / 64 is the width;

[0179] The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t ;

[0180] Specifically:

[0181] The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence;

[0182] TCN residual block 1 includes:

[0183] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0184] TCN residual block 2 includes:

[0185] A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0186] The first dilated causal convolution layer, the second dilated causal convolution layer, the third dilated causal convolution layer, and the fourth dilated causal convolution layer are all set to use convolution filters with 12 convolution kernels and a convolution kernel size of [1,4];

[0187] The EEG signal output by the input layer in 1) is input into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]; the EEG signal output by the TCN residual module is input into the ELU activation function, and the ELU activation function outputs features;

[0188] The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t .

[0189] In the basic model, the number of convolutional layers, pooling layers and learning rate are adjusted, and the optimal network model for extracting time domain features is found through classification accuracy, loss rate, ROC curve and confusion matrix.

[0190] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0191] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the processing process of the improved LSTM recurrent neural network RNN ​​is:

[0192] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0193] Improve the output data of the i″th layer of the LSTM recurrent neural network RNN ​​with X i″ Indicates that i″=1,2,3, the first layer is the input layer, the second layer is the LSTM layer, and the third layer is the Dropout layer;

[0194] 1) The two-dimensional EEG frequency-spatial domain feature X fs Input the input layer of the improved LSTM recurrent neural network RNN. The shape of the EEG signal output by the input layer is [16, 1, T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0195] 2) Input the EEG signal output by the input layer in 1) into the LSTM layer. The shape of the EEG signal output by the LSTM layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically:

[0196] Set the number of neurons in the hidden layer of the LSTM module to 20;

[0197] The EEG signal output by the input layer in 1) is processed by the LSTM layer, and the resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width;

[0198] 3) Input the EEG signal output by the LSTM layer in 2) into the seventh Dropout layer. The shape of the EEG signal output by the seventh Dropout layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width; specifically:

[0199] Set the seventh Dropout probability to 0.5;

[0200] The EEG signal output by the LSTM layer in 2) is passed through the seventh Dropout layer, and the shape of the obtained EEG signal is [20, T / 64], where 20 is the number of channels and T / 64 is the width;

[0201] The seventh Dropout layer outputs the EEG signal as the time domain feature extracted by the improved LSTM recurrent neural network RNN, which is called the two-dimensional EEG time domain feature X l ;

[0202] This method can automatically learn the time domain features of two-dimensional EEG signals through the LSTM structure.

[0203] In the basic model, the number of convolutional layers, pooling layers and learning rate are adjusted, and the optimal network structure is distinguished through classification accuracy, loss rate, ROC curve and confusion matrix, and the experimental results are analyzed and compared.

[0204] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0205] Specific implementation method 7: This implementation method is different from any one of the specific implementation methods 1 to 6 in that the processing process of the classifier is:

[0206] The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0207] The shape of the concatenated frequency-time-space domain feature X is [32, T / 64], where 32 is the number of channels and T / 64 is the width;

[0208] The concatenated frequency-time-space domain feature X is tensor flattened to obtain feature X'. The shape of feature X' is [544,1], where 544 is the number of channels and 1 is the width.

[0209] The classifier includes a fully connected layer and a first Softmax activation function layer in sequence;

[0210] The output data of the i′′ layer of the classifier structure is represented by X i″′ Indicates that i″′=1,2, the first layer is the fully connected layer, and the second layer is the first Softmax activation function layer;

[0211] 1) Input feature X' into the fully connected layer of the classifier. The fully connected layer outputs data in the shape of [4], where 4 is the unnormalized score of each motor imagery category.

[0212] 2) Input the output data of the fully connected layer into the first Softmax activation function layer of the classifier. The output data shape of the first Softmax activation function layer is [4], where 4 is the predicted probability of each motor imagery category;

[0213] 4 represents left hand, right hand, feet, and tongue;

[0214] The mathematical expression of the activation function Softmax is as follows:

[0215]

[0216] Where Z is the input vector, σ(Z) i is the i-th element in the output vector, and e is a natural constant;

[0217] Let Z1 and Z2 represent the output results of the fully connected layer and the activation function layer respectively, then the calculation of Z1 and Z2 can be expressed as:

[0218] Z1=WX'+b

[0219] Where W represents the weight matrix, the shape is [4,544]; b represents the bias vector, the shape is [4,1];

[0220]

[0221] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0222] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the trained EEG-TCN-LSTM convolutional neural network is obtained in step 4; the specific process is as follows:

[0223] The cross entropy is used as the loss function, and the loss function is optimized by the Adam optimizer until convergence to obtain the trained EEG-TCN-LSTM convolutional neural network.

[0224] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0225] Specific implementation method nine: In this implementation method, a motor imagery EEG signal classification system based on deep learning is used to execute the motor imagery EEG signal classification method based on deep learning described in any one of implementation methods one to eight.

[0226] Specific embodiment ten: This embodiment is a computer storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the deep learning-based motor imagery EEG signal classification method described in any one of specific embodiments one to eight.

[0227] It should be understood that the instructions include computer program products, software or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system, or other electronic device. Computer storage media may include readable media on which instructions are stored, which may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions.

[0228] Example

[0229] Embodiment 1:

[0230] A motor imagery EEG signal classification method based on deep learning. The specific process is as follows:

[0231] Step 1: Obtain the original EEG signal dataset;

[0232] Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector;

[0233] The specific process is:

[0234] Obtain the original EEG signal dataset from the BCI2008 IV-2a database (public dataset);

[0235] The MNE library is used to process the raw EEG signal into a two-dimensional raw EEG signal feature vector.

[0236] Step 1.1: The original EEG data set consists of EEG data of 9 subjects;

[0237] Each subject performed two experiments on two different days; each experiment consisted of 6 runs with a short break between each run. One run consisted of 48 trials (12 for each of the four possible categories), for a total of 288 trials per experiment;

[0238] Each subject sat in a comfortable armchair in front of a computer screen, and EEG was recorded using 22 Ag / AgCl electrodes (electrode spacing of 3.5 cm); all signals were recorded in a monopolar manner, with the left mastoid as the reference electrode and the right mastoid as the ground electrode; the signal was sampled at 250 Hz and band-pass filtered from 0.5 Hz to 100 Hz; the amplifier sensitivity was set to 100 μV, and a 50 Hz notch filter was enabled to suppress power supply noise;

[0239] At the beginning of each experiment, approximately 5 minutes of recordings were performed to estimate the effects of the electrooculogram (EOG). The recordings were divided into three parts: (1) 2 minutes with eyes open (looking at a fixation cross on the screen), (2) 1 minute with eyes closed, and (3) 1 minute with eye movements;

[0240] At the beginning of a trial, a fixation cross appears on a black screen. In addition, a brief audio prompt is played. Two seconds later, an arrow prompt appears on the screen, pointing to the left, right, down, or up (corresponding to four categories: left hand, right hand, feet, or tongue), and the arrow remains on the screen for 1.25 seconds. This prompts the subject to perform the required motor imagery task. No feedback is provided during the trial. The subject is required to continue the motor imagery task until the fixation cross disappears from the screen. This is followed by a short break, when the screen turns black again and the next experiment is carried out;

[0241] Step 1.2: Each subject will generate two EEG signal samples, and the EEG signal data is represented as a two-dimensional vector; the signal feature dimension is [1, 22, 1125], where 22 is the number of electrode channels in the original EEG signal, 1125 is the number of time points, and 1 is the expanded dimension of the channel number;

[0242] Next, input is given to the EEGNet structure, TCN structure, and LSTM structure for feature learning.

[0243] Step 2: In order to make full use of the effective information of the electrode channels and reduce the influence of irrelevant channels on the classification results, and further improve the classification accuracy of motor imagery signals, channel selection is performed on the original data set;

[0244] The ReliefF-MR algorithm is used to select channels of the original two-dimensional EEG signal feature vectors to obtain a new set of two-dimensional EEG signal feature vectors.

[0245] The specific process is:

[0246] Step 2.1: Calculate the correlation statistic d of the two-dimensional original EEG signal feature vector using the ReliefF algorithm, sort the two-dimensional original EEG signal feature vectors from small to large according to the correlation statistic, remove all the two-dimensional original EEG signal feature vectors whose correlation statistics are less than 0, and obtain the two-dimensional original EEG signal feature vector set A;

[0247] Step 2.2: Calculate the redundancy R1 of the two-dimensional original EEG signal feature vector set A;

[0248] Select the feature vector t with the smallest correlation statistic in the two-dimensional original EEG signal feature vector set A, remove the feature vector t from the two-dimensional original EEG signal feature vector set A, and obtain the two-dimensional original EEG signal feature vector set B; calculate the redundancy R2 of the two-dimensional original EEG signal feature vector set B;

[0249] Compare the size of redundancy R1 and redundancy R2:

[0250] If the redundancy R2 is less than the redundancy R1, it means that the redundancy cannot be reduced, and the two-dimensional original EEG signal feature vector set A is the optimal feature set, which is used as the new two-dimensional EEG signal feature vector set;

[0251] If the redundancy R2 is greater than or equal to the redundancy R1, the two-dimensional original EEG signal feature vector set B is used as the new two-dimensional original EEG signal feature vector set A, and step 2.2 is repeated.

[0252] The ReliefF-MR algorithm is as follows:

[0253]

[0254]

[0255] After channel selection of the data, the classification effects on the neural network before and after channel selection were compared.

[0256] Step 3: Construct EEG-TCN-LSTM convolutional neural network;

[0257] The EEG-TCN-LSTM convolutional neural network is a multi-domain feature fusion network;

[0258] The EEG-TCN-LSTM convolutional neural network includes: improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN, and classifier;

[0259] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depth convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0260] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, ELU activation function;

[0261] The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence;

[0262] TCN residual block 1 includes:

[0263] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0264] TCN residual block 2 includes:

[0265] A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0266] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0267] The classifier includes a fully connected layer and a first Softmax activation function layer in sequence;

[0268] The working process of the EEG-TCN-LSTM convolutional neural network is:

[0269] The new two-dimensional EEG signal feature vector obtained in step 2 is input into the improved EEGNet convolutional neural network, and the improved EEGNet convolutional neural network outputs a two-dimensional EEG frequency-spatial domain feature X fs ;

[0270] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved TCN temporal convolutional neural network, and the improved TCN temporal convolutional neural network outputs the two-dimensional EEG time domain feature X t ;

[0271] Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved LSTM recurrent neural network RNN, and the improved LSTM recurrent neural network RNN ​​outputs the two-dimensional EEG time domain feature X l ;

[0272] The two-dimensional EEG time domain feature X t and the two-dimensional EEG time domain feature X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0273] Flatten the concatenated frequency-time-space domain feature X to obtain feature X';

[0274] The feature X' is input into the classifier, and the classifier outputs the predicted probability of each motor imagery category;

[0275] The processing process of the improved EEGNet convolutional neural network is:

[0276] The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depth convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer;

[0277] Improve the output data of the i′th layer of the EEGNet convolutional neural network with X i′ Indicates that i′=1,2,3,...,16, the first layer is the input layer, the second layer is the one-dimensional convolution layer, the third layer is the first activation function layer, the fourth layer is the first batch normalization layer, the fifth layer is the depthwise convolution layer, the sixth layer is the second activation function layer, the seventh layer is the second batch normalization layer, the eighth layer is the third activation function layer, the ninth layer is the first average pooling layer, the tenth layer is the first Dropout layer, the eleventh layer is the separable convolution layer, the twelfth layer is the fourth activation function layer, the thirteenth layer is the third batch normalization layer, the fourteenth layer is the fifth activation function layer, the fifteenth layer is the second average pooling layer, and the sixteenth layer is the second Dropout layer;

[0278] 1) Input the two-dimensional EEG signal feature vector obtained in step 2 into the input layer of the improved EEGNet convolutional neural network, and the shape of the EEG signal output by the input layer is [1, C, T];

[0279] 1 is the number of characteristic channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125);

[0280] The number of electrode channels C depends on the feature vector extracted from the original EEG signal by the ReliefF-MR algorithm. For the BCI2008 IV-2a dataset, C is 22;

[0281] 2) Input the EEG signal output by the input layer into the one-dimensional convolution layer, and the one-dimensional convolution layer outputs the feature X2; input the feature X2 output by the one-dimensional convolution layer into the first activation function layer, and the first activation function layer outputs the feature; specifically:

[0282] Set the one-dimensional convolution layer to a convolution filter with 8 convolution kernels, a convolution kernel size of [1,32], and a stride size of [1,1]. The stride represents the x- and y-movement distances of the convolution kernel.

[0283] Set the padding method to zero padding of the same shape, which keeps the sample shape unchanged in the convolution calculation;

[0284] The EEG signal output by the input layer is input into the one-dimensional convolution layer. The one-dimensional convolution layer outputs feature X2. The shape of feature X2 is [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points (such as 1125).

[0285] The one-dimensional convolution layer output feature X2 is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0286] The mathematical expression of the Linear activation function is as follows:

[0287] f(x)=x

[0288] 3) Input the output feature of the first activation function layer into the first batch normalization layer, and the first batch normalization layer outputs feature X3; specifically:

[0289] The output features of the first activation function layer are batch normalized to obtain feature X3 with a shape of [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points (e.g., 1125);

[0290] 4) Input the output feature X3 of the first batch normalization layer into the deep convolution layer, and the deep convolution layer outputs the feature X4; input the output feature X4 of the deep convolution layer into the second activation function layer, and the second activation function layer outputs the feature; specifically:

[0291] Set the depth convolutional layer to a convolution filter with 2 kernels, kernel size [C, 1], and stride size [1, 1], where the stride represents the x and y movement distances of the kernel.

[0292] Set the padding method to zero padding of the same shape, which keeps the sample shape unchanged in the convolution calculation;

[0293] The output feature X3 of the first batch normalization layer is input into the deep convolution layer, and the deep convolution layer outputs feature X4. The shape of feature X4 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0294] The output feature X4 of the deep convolutional layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0295] 5) Input the output feature of the second activation function layer into the second batch normalization layer, and the second batch normalization layer outputs feature X5; specifically:

[0296] Perform batch normalization on the output features of the second activation function layer, and obtain the shape of feature X5 as [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0297] 6) Input the output feature X5 of the second batch normalization layer into the third activation function layer, and the third activation function layer outputs feature X6; specifically:

[0298] Use the ELU activation function to process the second batch normalization layer output feature X5, and the shape of feature X6 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width (such as 1125);

[0299] The mathematical expression of the activation function ELU is as follows:

[0300]

[0301] 7) Input the output feature X6 of the third activation function layer into the first average pooling layer, and the first average pooling layer outputs feature X7; specifically:

[0302] Set the pooling window size of the first average pooling layer to [1,8] and the stride size to [1,8];

[0303] The output feature X6 of the third activation function layer is subjected to the first average pooling layer operation, and the shape of the feature X7 obtained is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140);

[0304] 8) Input the output feature X7 of the first average pooling layer into the first Dropout layer, and the first Dropout layer outputs feature X8; specifically:

[0305] Set the Dropout probability to 0.2;

[0306] The first average pooling layer outputs feature X7, which is then processed by the first Dropout layer to obtain feature X8 of shape [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g., 140).

[0307] 9) Input the output feature X8 of the first Dropout layer into the separable convolution layer, and the separable convolution layer outputs X9; input the output feature X9 of the separation convolution layer into the fourth activation function layer, and the fourth activation function layer outputs the feature;

[0308] Specifically:

[0309] Set the separable convolution layer to a convolution filter with 16 convolution kernels, a convolution kernel size of [1,16], and a stride size of [1,1]. The stride represents the x- and y-movement distances of the convolution kernel.

[0310] The output feature X8 of the first Dropout layer is input into the separable convolution layer, and the separable convolution layer outputs X9. The shape of feature X9 is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g. 140);

[0311] The output feature X9 of the separation convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature;

[0312] 10) Input the output feature of the fourth activation function layer into the third batch normalization layer, and the third batch normalization layer outputs feature X 10 ; Specifically:

[0313] Perform batch normalization on the output features of the fourth activation function layer to obtain feature X 10 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width (e.g. 140);

[0314] 11) Output feature X of the third batch normalization layer 10 Input the fifth activation function layer, the fifth activation function layer outputs feature X 11 ; Specifically:

[0315] The third batch normalization layer outputs feature X 10 Input ELU activation function layer, ELU activation function layer output feature X 11 The shape is [16,1,T / 8], 16 is the number of channels, 1 is the height, and T / 8 is the width;

[0316] 12) Output feature X of the fifth activation function layer 11 Input the second average pooling layer, the second average pooling layer outputs feature X 12 ; Specifically:

[0317] Set the pooling window size of the second average pooling layer to [1,8] and the stride size to [1,8];

[0318] The fifth activation function layer outputs feature X 11 After the second average pooling layer operation, feature X is obtained 12 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width (e.g. 17);

[0319] 13) Output feature X of the second average pooling layer 12 Input the second Dropout layer, the second Dropout layer outputs feature X 13 ; Specifically:

[0320] Set the Dropout probability to 0.2;

[0321] The second average pooling layer outputs feature X 12 After the second Dropout layer operation, we get feature X 13The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0322] Feature X 13 As the frequency-spatial domain feature X extracted by the improved EEGNet convolutional neural network fs , called the two-dimensional EEG frequency-space domain feature X fs .

[0323] In the basic model, adjust the number of convolutional layers, pooling layers and learning rate, distinguish the optimal network structure through classification accuracy, loss rate, ROC curve and confusion matrix, and analyze and compare the experimental results;

[0324] The processing process of the improved TCN temporal convolutional neural network is as follows:

[0325] The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, ELU activation function;

[0326] The TCN residual module includes: TCN residual block 1 and TCN residual block 2;

[0327] Improve the output data of the i-th layer of the TCN temporal convolutional neural network structure with X i , i=1,2,3, the first layer is the input layer, the second layer is the TCN residual module, and the third layer is the ELU activation function;

[0328] The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence;

[0329] TCN residual block 1 includes:

[0330] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0331] TCN residual block 2 includes:

[0332] A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0333] The mathematical expression of dilated causal convolution is as follows:

[0334]

[0335] Where d is the dilation factor, k is the convolution kernel size, sd·i represents the past direction, and x∈R n Is a one-dimensional sequence input, filter f:{0,…,k-1}→R.

[0336] The mathematical expression of the activation function ReLU is as follows:

[0337]

[0338] 1) Improve the two-dimensional EEG frequency-spatial domain feature X output by the EEGNet convolutional neural network fs Input the input layer of the improved TCN temporal convolutional neural network, and the shape of the EEG signal output by the input layer is [16, 1, T / 64];

[0339] 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0340] 2) Input the EEG signal output by the input layer in 1) into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]; Input the EEG signal output by the TCN residual module into the ELU activation function, and the ELU activation function outputs features; 12 is the number of channels, and T / 64 is the width;

[0341] The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t ;

[0342] Specifically:

[0343] The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence;

[0344] TCN residual block 1 includes:

[0345] The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer;

[0346] TCN residual block 2 includes:

[0347] A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer;

[0348] The first dilated causal convolution layer, the second dilated causal convolution layer, the third dilated causal convolution layer, and the fourth dilated causal convolution layer are all set to use convolution filters with 12 convolution kernels and a convolution kernel size of [1,4];

[0349] The EEG signal output by the input layer in 1) is input into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]; the EEG signal output by the TCN residual module is input into the ELU activation function, and the ELU activation function outputs features;

[0350] The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t .

[0351] In the basic model, the number of convolutional layers, pooling layers and learning rate are adjusted, and the optimal network model for extracting time domain features is found through classification accuracy, loss rate, ROC curve and confusion matrix.

[0352] The processing process of the improved LSTM recurrent neural network RNN ​​is:

[0353] The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence;

[0354] Improve the output data of the i″th layer of the LSTM recurrent neural network RNN ​​with X i″ Indicates that i″=1,2,3, the first layer is the input layer, the second layer is the LSTM layer, and the third layer is the Dropout layer;

[0355] 1) The two-dimensional EEG frequency-spatial domain feature X fs Input the input layer of the improved LSTM recurrent neural network RNN. The shape of the EEG signal output by the input layer is [16, 1, T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width;

[0356] 2) Input the EEG signal output by the input layer in 1) into the LSTM layer. The shape of the EEG signal output by the LSTM layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically:

[0357] Set the number of neurons in the hidden layer of the LSTM module to 20;

[0358] The EEG signal output by the input layer in 1) is processed by the LSTM layer, and the resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width;

[0359] 3) Input the EEG signal output by the LSTM layer in 2) into the seventh Dropout layer. The shape of the EEG signal output by the seventh Dropout layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width; specifically:

[0360] Set the seventh Dropout probability to 0.5;

[0361] The EEG signal output by the LSTM layer in 2) is passed through the seventh Dropout layer, and the resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width;

[0362] The seventh Dropout layer outputs the EEG signal as the time domain feature extracted by the improved LSTM recurrent neural network RNN, which is called the two-dimensional EEG time domain feature X l ;

[0363] This method can automatically learn the time domain features of two-dimensional EEG signals through the LSTM structure.

[0364] In the basic model, the number of convolutional layers, pooling layers and learning rate are adjusted, and the optimal network structure is distinguished through classification accuracy, loss rate, ROC curve and confusion matrix, and the experimental results are analyzed and compared.

[0365] The processing process of the classifier is:

[0366] The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X;

[0367] The shape of the concatenated frequency-time-space domain feature X is [32, T / 64], where 32 is the number of channels and T / 64 is the width;

[0368] The concatenated frequency-time-space domain feature X is tensor flattened to obtain feature X'. The shape of feature X' is [544,1], where 544 is the number of channels and 1 is the width.

[0369] The classifier includes a fully connected layer and a first Softmax activation function layer in sequence;

[0370] The output data of the i′′ layer of the classifier structure is represented by X i″′ Indicates that i″′=1,2, the first layer is the fully connected layer, and the second layer is the first Softmax activation function layer;

[0371] 1) Input feature X' into the fully connected layer of the classifier. The fully connected layer outputs data in the shape of [4], where 4 is the unnormalized score of each motor imagery category.

[0372] 2) Input the output data of the fully connected layer into the first Softmax activation function layer of the classifier. The output data shape of the first Softmax activation function layer is [4], where 4 is the predicted probability of each motor imagery category;

[0373] 4 represents left hand, right hand, feet, and tongue;

[0374] The mathematical expression of the activation function Softmax is as follows:

[0375]

[0376] Where Z is the input vector, σ(Z) i is the i-th element in the output vector, and e is a natural constant;

[0377] Let Z1 and Z2 represent the output results of the fully connected layer and the activation function layer respectively, then the calculation of Z1 and Z2 can be expressed as:

[0378] Z1=WX'+b

[0379] Where W represents the weight matrix, the shape is [4,544]; b represents the bias vector, the shape is [4,1];

[0380]

[0381] Step 4: Obtain the trained EEG-TCN-LSTM convolutional neural network; the specific process is:

[0382] The cross entropy is used as the loss function, and the loss function is optimized by the Adam optimizer until convergence to obtain the trained EEG-TCN-LSTM convolutional neural network.

[0383] Step 5: Obtain the subject's EEG signal to be tested (the subject uses 22 Ag / AgCl electrodes (the electrode spacing is 3.5 cm) to record the subject's EEG signal to be tested;);

[0384] Preprocessing the EEG signal data to be tested into a two-dimensional EEG signal feature vector;

[0385] The ReliefF-MR algorithm is used to select channels of the original two-dimensional EEG signal feature vectors to obtain a new set of two-dimensional EEG signal feature vectors.

[0386] The two-dimensional vector is input into the trained EEG-TCN-LSTM convolutional neural network, and the trained EEG-TCN-LSTM convolutional neural network outputs the predicted probability of each motor imagery category.

[0387] Embodiment 2:

[0388] In order to make full use of the effective information of the electrode channels and reduce the influence of irrelevant channels on the classification results, and further improve the classification accuracy of motor imagery signals, channel selection is performed on the original data set. The ReliefF-MR algorithm is used to select channels for the two-dimensional original EEG signal feature vector. The effect of channel selection is verified in the improved deep learning model proposed in step 3;

[0389] The improved EEGNet convolutional neural network, the improved TCN temporal convolutional neural network, the improved LSTM recurrent neural network RNN ​​and the classifier form a complete network structure to form the EEG-TCN-LSTM convolutional neural network. The details of the EEG-TCN-LSTM convolutional neural network structure are shown in the figure below. Figure 5 ;

[0390] After channel selection, the classification accuracy and optimal number of channels of the EEG-TCN-LSTM convolutional neural network model optimized by the ReliefF-MR algorithm, the ReliefF algorithm, and the mRMR algorithm and the EEG-TCN-LSTM convolutional neural network model without channel selection optimization were compared.

[0391] 1: Compare the classification effects before and after channel selection through the accuracy curve; the experimental results are as follows Figure 6 , you can see Figure 6 The ReliefF-MR algorithm (black solid line) is used to select channels for the data set input to the network, which significantly improves the effect in the EEG-TCN-LSTMt network.

[0392] 2: Compare the optimal number of channels before and after channel selection through the optimal channel number curve; the experimental results are as follows Figure 7 , you can see Figure 7 The number of data channels input to the network can be significantly reduced by using the ReliefF-MR algorithm (black solid line);

[0393] Embodiment 3:

[0394] EEG signal is a special kind of time series data, and its characteristic information has three dimensions: time domain, frequency domain, and space domain. Therefore, the EEG signal classification algorithm requires a method that can fully extract the multi-dimensional characteristics of EEG signals;

[0395] 1: The improved EEGNet convolutional neural network constructed in step 3 first extracts the frequency features of the motor imagery signal from the time axis through two-dimensional convolution; then uses deep convolution to learn the spatial filter of a specific frequency to extract the spatial domain features related to the channel; finally, the deep separable convolution outputs the final feature map through deep convolution and point convolution. The frequency domain features of the motor imagery signal are generated by one-dimensional convolution and deep classifiable convolution, the spatial domain features are generated based on deep convolution, and the time domain features are generated according to the time dimension;

[0396] 2: The output feature map of step 3 still contains time information, so the feature vectors obtained by the improved EEGNet convolutional neural network are input into the improved TCN time convolutional neural network and the improved LSTM recurrent neural network RNN ​​in parallel to extract time domain features. The improved TCN time convolutional neural network introduces dilated causal convolution and residual connection to obtain time domain feature information across time steps; the improved LSTM recurrent neural network RNN ​​learns the long-term and short-term related feature information of motor imagery signals. Then the local features extracted by the improved TCN time convolutional neural network and the improved LSTM recurrent neural network RNN ​​are spliced ​​and fused to form a global information feature map containing the time, frequency and spatial domains of motor imagery signals;

[0397] The network model consisting of the improved EEGNet convolutional neural network, the improved TCN temporal convolutional neural network, the improved LSTM recurrent neural network RNN ​​and the classifier is named the EEG-TCN-LSTM convolutional neural network model. In order to evaluate the impact of the improved EEGNet convolutional neural network, the improved TCN temporal convolutional neural network and the improved LSTM recurrent neural network RNN ​​on the overall performance in the EEG-TCN-LSTM convolutional neural network model, an ablation experiment is designed to analyze the results.

[0398] 1: EEGNet was selected as the benchmark experimental model, and TCN modules or LSTM modules were added to it, and these two modules were added in parallel to conduct ablation experiments;

[0399] 2: Experimental results are as follows Figure 8, we can see that the classification effect of the improved EEGNet convolutional neural network is average, but the improved EEGNet convolutional neural network with the TCN module added in series + the improved TCN time convolutional neural network can extract effective recognition information containing time domain features from the output feature map of the improved EEGNet convolutional neural network across time steps, and the average accuracy rate is increased to 70.93%. Similarly, the improved EEGNet convolutional neural network with the LSTM module added in series + the improved LSTM recurrent neural network RNN ​​can also further focus on the long-term and short-term time domain features of the motor imagery signal, and the average accuracy rate is increased by 14%, while the average accuracy rate of the EEGNet convolutional neural network + the improved TCN time convolutional neural network + the improved LSTM recurrent neural network RNN ​​reaches the highest, that is, 73.07%;

[0400] 3: If Figure 9a , 9b The confusion matrices represent the prediction results of subject A3 and subject A7 under the EEG-TCN-LSTM model. It can be seen that in the confusion matrix of the four categories, the model has little effect on the classification effect of each type of motor imagery task. It can classify each type of motor imagery task indiscriminately, and the misjudgment rate is low, indicating that the extracted features are stable and effective.

[0401] 4: To further verify the effectiveness of the model in the motor imagery classification task, the t-Distributed Stochastic Neighbor Embedding (t-SNE) method is used to visualize the EEG features extracted by the EEG-TCN-LSTM model. Fig.10a , 10b As shown in the figure, the four colors in the figure represent the four types of motor imagery tasks to which the EEG features belong. It can be seen that after the feature extraction of the EEG-TCN-LSTM model, the EEG features of different categories have a high concentration, indicating that the recognition effect of the model proposed in this paper is very good;

[0402] The login interface of a motor imagery EEG signal classification system based on deep learning Fig.11 As shown, the channel selection algorithm interface is as follows Fig.12 As shown, the feature extraction and classification interface is as follows Fig.13a , Fig.13b shown.

[0403] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A motor imagery EEG signal classification method based on deep learning, characterized by: The specific process of the method is: Step 1: Obtain the original EEG signal dataset; Preprocessing the original EEG signal data into a two-dimensional original EEG signal feature vector; Step 2: Perform channel selection on the original two-dimensional EEG signal feature vector to obtain a new two-dimensional EEG signal feature vector set; Step 3: Construct EEG-TCN-LSTM convolutional neural network; The EEG-TCN-LSTM convolutional neural network is a multi-domain feature fusion network; The EEG-TCN-LSTM convolutional neural network includes: improved EEGNet convolutional neural network, improved TCN temporal convolutional neural network, improved LSTM recurrent neural network RNN, and classifier; The improved EEGNet convolutional neural network includes: input layer, one-dimensional convolution layer, first activation function layer, first batch normalization layer, depth convolution layer, second activation function layer, second batch normalization layer, third activation function layer, first average pooling layer, first Dropout layer, separable convolution layer, fourth activation function layer, third batch normalization layer, fifth activation function layer, second average pooling layer and second Dropout layer; The improved TCN temporal convolutional neural network includes: input layer, TCN residual module, ELU activation function; The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence; TCN residual block 1 includes: The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer; TCN residual block 2 includes: A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer; The improved LSTM recurrent neural network RNN ​​includes an input layer, an LSTM layer, and a seventh Dropout layer in sequence; The classifier includes a fully connected layer and a first Softmax activation function layer in sequence; The working process of the EEG-TCN-LSTM convolutional neural network is: The new two-dimensional EEG signal feature vector obtained in step 2 is input into the improved EEGNet convolutional neural network, and the improved EEGNet convolutional neural network outputs a two-dimensional EEG frequency-spatial domain feature X fs ; Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved TCN temporal convolutional neural network, and the improved TCN temporal convolutional neural network outputs the two-dimensional EEG time domain feature X t ; Improve the two-dimensional EEG frequency-spatial domain features output by EEGNet convolutional neural network X fs Input the improved LSTM recurrent neural network RNN, and the improved LSTM recurrent neural network RNN ​​outputs the two-dimensional EEG time domain feature X l ; The improved TCN temporal convolutional neural network outputs two-dimensional EEG time domain features X t And the improved LSTM recurrent neural network RNN ​​outputs two-dimensional EEG time domain features X l Perform splicing to obtain the spliced ​​frequency-time-space domain feature X; Flatten the concatenated frequency-time-space domain feature X to obtain feature X′; The feature X′ is input into the classifier, and the classifier outputs the predicted probability of each motor imagery category; Step 4: Obtain the trained EEG-TCN-LSTM convolutional neural network; Step 5: Obtain the subject's EEG signal to be tested; Preprocessing the EEG signal data to be tested into a two-dimensional EEG signal feature vector; Perform channel selection on the original two-dimensional EEG signal feature vector to obtain a new two-dimensional EEG signal feature vector set; The two-dimensional vector is input into the trained EEG-TCN-LSTM convolutional neural network, and the trained EEG-TCN-LSTM convolutional neural network outputs the predicted probability of each motor imagery category.

2. The method for classifying motor imagery EEG signals based on deep learning according to claim 1, characterized in that: In the step 1, an original EEG signal data set is obtained; Preprocessing the raw EEG signal data into a two-dimensional raw EEG signal feature vector; The specific process is: Obtain the original EEG signal dataset from the BCI2008 IV-2a database; The MNE library is used to process the raw EEG signal into a two-dimensional raw EEG signal feature vector.

3. The method for classifying motor imagery EEG signals based on deep learning according to claim 2, characterized in that: In the step 2, channel selection is performed on the two-dimensional original EEG signal feature vector obtained in the step 1 to obtain a new two-dimensional EEG signal feature vector set; The specific process is: Step 2.1: Calculate the correlation statistic d of the two-dimensional original EEG signal feature vector using the ReliefF algorithm, sort the two-dimensional original EEG signal feature vectors from small to large according to the correlation statistic, remove all the two-dimensional original EEG signal feature vectors whose correlation statistics are less than 0, and obtain the two-dimensional original EEG signal feature vector set A; Step 2.2: Calculate the redundancy R1 of the two-dimensional original EEG signal feature vector set A; Select the feature vector t with the smallest correlation statistic in the two-dimensional original EEG signal feature vector set A, remove the feature vector t from the two-dimensional original EEG signal feature vector set A, and obtain the two-dimensional original EEG signal feature vector set B; calculate the redundancy R2 of the two-dimensional original EEG signal feature vector set B; Compare the size of redundancy R1 and redundancy R2: If the redundancy R2 is less than the redundancy R1, it means that the redundancy cannot be reduced, and the two-dimensional original EEG signal feature vector set A is the optimal feature set, which is used as the new two-dimensional EEG signal feature vector set; If the redundancy R2 is greater than or equal to the redundancy R1, the two-dimensional original EEG signal feature vector set B is used as the new two-dimensional original EEG signal feature vector set A, and step 2.2 is repeated.

4. The method for classifying motor imagery EEG signals based on deep learning according to claim 3, characterized in that: The processing process of improving the EEGNet convolutional neural network in step 3 is: 1) Input the two-dimensional EEG signal feature vector obtained in step 2 into the input layer of the improved EEGNet convolutional neural network, and the shape of the EEG signal output by the input layer is [1, C, T]; 1 is the number of characteristic channels, C is the number of electrode channels, and T is the number of time points; 2) Input the EEG signal output by the input layer into the one-dimensional convolution layer, and the one-dimensional convolution layer outputs the feature X2; input the feature X2 output by the one-dimensional convolution layer into the first activation function layer, and the first activation function layer outputs the feature; Specifically: Set the one-dimensional convolution layer to a convolution filter with 8 convolution kernels, a convolution kernel size of [1,32], and a stride size of [1,1]; Set the filling method to zero-fill with the same shape; The EEG signal output by the input layer is input into the one-dimensional convolution layer. The one-dimensional convolution layer outputs feature X2. The shape of feature X2 is [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points. The one-dimensional convolution layer output feature X2 is input into the Linear activation function layer, and the Linear activation function layer outputs the feature; 3) Input the output feature of the first activation function layer into the first batch normalization layer, and the first batch normalization layer outputs feature X3; specifically: The output features of the first activation function layer are batch normalized to obtain feature X3 with a shape of [8, C, T], where 8 is the number of feature channels, C is the number of electrode channels, and T is the number of time points; 4) Input the output feature X3 of the first batch normalization layer into the deep convolution layer, and the deep convolution layer outputs the feature X4; input the output feature X4 of the deep convolution layer into the second activation function layer, and the second activation function layer outputs the feature; Specifically: Set the depth convolution layer to a convolution filter with 2 kernels, kernel size [C, 1] and stride size [1, 1]; Set the filling method to zero-fill with the same shape; The output feature X3 of the first batch normalization layer is input into the deep convolution layer, and the deep convolution layer outputs feature X4. The shape of feature X4 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width; The output feature X4 of the deep convolutional layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature; 5) Input the output feature of the second activation function layer into the second batch normalization layer, and the second batch normalization layer outputs feature X5; specifically: The output features of the second activation function layer are batch normalized to obtain feature X5 with a shape of [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width; 6) Input the output feature X5 of the second batch normalization layer into the third activation function layer, and the third activation function layer outputs feature X6; specifically: The second batch normalization layer output feature X5 is processed using the ELU activation function, and the shape of feature X6 is [16, 1, T], where 16 is the number of channels, 1 is the height, and T is the width; 7) Input the output feature X6 of the third activation function layer into the first average pooling layer, and the first average pooling layer outputs feature X7; specifically: Set the pooling window size of the first average pooling layer to [1,8] and the stride size to [1,8]; The output feature X6 of the third activation function layer is processed by the first average pooling layer, and the shape of the feature X7 is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width; 8) Input the output feature X7 of the first average pooling layer into the first Dropout layer, and the first Dropout layer outputs feature X8; specifically: Set the Dropout probability to 0.2; The first average pooling layer outputs feature X7, which is then processed by the first Dropout layer to obtain feature X8 of shape [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width; 9) Input the output feature X8 of the first Dropout layer into the separable convolution layer, and the separable convolution layer outputs X9; input the output feature X9 of the separation convolution layer into the fourth activation function layer, and the fourth activation function layer outputs the feature; Specifically: Set the separable convolution layer to a convolution filter with 16 kernels, kernel size [1,16], and stride size [1,1]; The output feature X8 of the first Dropout layer is input into the separable convolution layer. The separable convolution layer outputs X9. The shape of feature X9 is [16, 1, T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width. The output feature X9 of the separation convolution layer is input into the Linear activation function layer, and the Linear activation function layer outputs the feature; 10) Input the output feature of the fourth activation function layer into the third batch normalization layer, and the third batch normalization layer outputs feature X 10 ; Specifically: Perform batch normalization on the output features of the fourth activation function layer to obtain feature X 10 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width; 11) Output feature X of the third batch normalization layer 10 Input the fifth activation function layer, the fifth activation function layer outputs feature X 11 ; Specifically: The third batch normalization layer outputs feature X 10 Input ELU activation function layer, ELU activation function layer output feature X 11 The shape is [16,1,T / 8], where 16 is the number of channels, 1 is the height, and T / 8 is the width; 12) Output feature X of the fifth activation function layer 11 Input the second average pooling layer, the second average pooling layer outputs feature X 12 ; Specifically: Set the pooling window size of the second average pooling layer to [1,8] and the stride size to [1,8]; The fifth activation function layer outputs feature X 11 After the second average pooling layer operation, feature X is obtained 12 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width; 13) Output feature X of the second average pooling layer 12 Input the second Dropout layer, the second Dropout layer outputs feature X 13 ; Specifically: Set the Dropout probability to 0.2; The second average pooling layer outputs feature X 12 After the second Dropout layer operation, we get feature X 13 The shape is [16,1,T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width; Feature X 13 As the frequency-spatial domain feature X extracted by the improved EEGNet convolutional neural network fs , called the two-dimensional EEG frequency-space domain feature X fs .

5. The method for classifying motor imagery EEG signals based on deep learning according to claim 4, characterized in that: The processing process of the improved TCN temporal convolutional neural network is as follows: 1) Improve the two-dimensional EEG frequency-spatial domain feature X output by the EEGNet convolutional neural network fs Input the input layer of the improved TCN temporal convolutional neural network, and the shape of the EEG signal output by the input layer is [16, 1, T / 64]; 16 is the number of channels, 1 is the height, and T / 64 is the width; 2) Input the EEG signal output by the input layer in 1) into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]; Input the EEG signal output by the TCN residual module into the ELU activation function, and the ELU activation function outputs features; 12 is the number of channels, T / 64 is the width; The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t ; Specifically: The TCN residual module includes: TCN residual block 1 and TCN residual block 2 in sequence; TCN residual block 1 includes: The first dilated causal convolution layer, the first weight normalization layer, the first ReLU activation function layer, the third Dropout layer, the second dilated causal convolution layer, the second weight normalization layer, the second ReLU activation function layer, and the fourth Dropout layer; TCN residual block 2 includes: A third dilated causal convolution layer, a third weight normalization layer, a third ReLU activation function layer, a fifth Dropout layer, a fourth dilated causal convolution layer, a fourth weight normalization layer, a fourth ReLU activation function layer, and a sixth Dropout layer; The first dilated causal convolution layer, the second dilated causal convolution layer, the third dilated causal convolution layer, and the fourth dilated causal convolution layer are all set to use convolution filters with 12 convolution kernels and a convolution kernel size of [1,4]; The EEG signal output by the input layer in 1) is input into the TCN residual module, and the shape of the EEG signal output by the TCN residual module is [12, T / 64]; the EEG signal output by the TCN residual module is input into the ELU activation function, and the ELU activation function outputs features; The output feature of the ELU activation function is used as the time domain feature of the improved TCN temporal convolutional neural network, which is called the two-dimensional EEG time domain feature X t .

6. The method for classifying motor imagery EEG signals based on deep learning according to claim 5, characterized in that: The processing process of the improved LSTM recurrent neural network RNN ​​is: 1) The two-dimensional EEG frequency-spatial domain feature X fs Input the input layer of the improved LSTM recurrent neural network RNN. The shape of the EEG signal output by the input layer is [16, 1, T / 64], where 16 is the number of channels, 1 is the height, and T / 64 is the width; 2) Input the EEG signal output by the input layer in 1) into the LSTM layer. The shape of the EEG signal output by the LSTM layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width. Specifically: Set the number of neurons in the hidden layer of the LSTM module to 20; The EEG signal output by the input layer in 1) is processed by the LSTM layer, and the resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width; 3) Input the EEG signal output by the LSTM layer in 2) into the seventh Dropout layer. The shape of the EEG signal output by the seventh Dropout layer is [20, T / 64], where 20 is the number of channels and T / 64 is the width; specifically: Set the seventh Dropout probability to 0.5; The EEG signal output by the LSTM layer in 2) is passed through the seventh Dropout layer, and the resulting EEG signal shape is [20, T / 64], where 20 is the number of channels and T / 64 is the width; The seventh Dropout layer outputs the EEG signal as the time domain feature extracted by the improved LSTM recurrent neural network RNN, which is called the two-dimensional EEG time domain feature X l .

7. The method for classifying motor imagery EEG signals based on deep learning according to claim 6, characterized in that: The processing process of the classifier is: 1) Input feature X' into the fully connected layer of the classifier. The fully connected layer outputs data in the shape of [4], where 4 is the unnormalized score of each motor imagery category. 2) Input the output data of the fully connected layer into the first Softmax activation function layer of the classifier. The output data shape of the first Softmax activation function layer is [4], where 4 is the predicted probability of each motor imagery category; 4 represents left hand, right hand, feet, and tongue.

8. The method for classifying motor imagery EEG signals based on deep learning according to claim 7, characterized in that: In step 4, the trained EEG-TCN-LSTM convolutional neural network is obtained; the specific process is: The cross entropy is used as the loss function, and the loss function is optimized by the Adam optimizer until convergence to obtain the trained EEG-TCN-LSTM convolutional neural network.

9. A motor imagery EEG signal classification system based on deep learning, characterized by: The system is used to execute the deep learning-based motor imagery EEG signal classification method described in any one of 1 to 8.

10. A computer storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the deep learning-based motor imagery EEG signal classification method described in any one of 1 to 8.

Images