Motor imagery eeg signal classification method based on parallel damscn-lstm

Abstract

The application discloses a motor imagery electroencephalogram signal classification method based on parallel DAMSCN-LSTM, which comprises the following steps: S1, pre-processing four types of motor imagery electroencephalogram signals, including removing electrooculogram and electromyogram and performing band-pass filtering; S2, extracting time features of the electroencephalogram signals by using LSTM; S3, extracting time-frequency features of the electroencephalogram signals in different scales by using DAMSCN, and simultaneously introducing spatial attention and channel attention modules; S4, extracting the multi-scale time-frequency features while introducing spatial attention mechanism and channel attention mechanism; S5, splicing the extracted multi-scale time-frequency features and time features, and then realizing feature classification by means of a full connection layer and a SoftMax layer. Finally, the method is verified on a public data set, and compared with related literatures, and the results show that the multi-class motor imagery electroencephalogram signal classification algorithm provided in the application has better classification results.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of signal processing and pattern recognition, and particularly relates to a motor imagery electroencephalogram signal classification method based on parallel DAMSCN-LSTM. BACKGROUND

[0002] Brain-computer interface (BCI) is a computer-based system that can realize direct communication between brain and external devices, which collects and examines brain signals and converts them into instructions, and communicates with output devices to perform the required response. With the introduction of complex algorithms, high-performance computers, edge and cloud computing, next-generation communication technologies, brain-computer interface is widely used in the fields of rehabilitation of neuromotor disorders, robot control and speech communication.

[0003] This paper mainly studies the motor imagery electroencephalogram signal, that is, when the subject imagines moving his right hand or left hand, or both feet, or any finger in the five fingers, tongue or any limb, the corresponding motor imagery electroencephalogram signal will be generated. The MI-BCI system is divided into five stages: signal acquisition, signal preprocessing, feature extraction, feature classification and device control. EEG signal acquisition, signal digitization and storage all belong to the signal acquisition stage. Signal filtering, cleaning and conversion are all part of the data preprocessing stage. In the feature extraction stage, discriminant features are extracted from the EEG signal data containing useful information. In the classification stage, the extracted features are used as input to train the machine learning model. The trained model can classify different signals and MI tasks. Finally, in the device control stage, the classified signals are converted into commands for controlling external devices.

[0004] With the development of deep learning (DL) in recent years, due to its ability to adapt to non-linear and non-stationary signals and automatically extract important feature information from electroencephalogram signals, it has shown superior performance in MI-EEG classification than traditional methods. However, due to the high dimensionality of electroencephalogram data, noise, channel correlation and artifacts, it is challenging to use DL to establish an ideal electroencephalogram classification model.

[0005] MI-EEG classification, as an increasingly popular field in the field of BCI, not only has great potential, but also has important applications such as robots, therapeutic applications, games, etc. Despite this, there are still great limitations in data processing and classification methods. Traditional convolutional neural networks often ignore time information when classifying motor imagery electroencephalogram signals, use single-scale convolution kernels, and cannot obtain rich features, resulting in poor classification performance. SUMMARY

[0006] To solve the above problems, the application provides a motor imagery electroencephalogram signal classification method based on parallel DAMSCN-LSTM.

[0007] The technical scheme adopted by the application is as follows: the motor imagery electroencephalogram signal classification method based on parallel DAMSCN-LSTM comprises the following steps:

[0008] S1, pre-processing four types of motor imagery electroencephalogram signals, including removing electrooculogram and electromyogram and performing band-pass filtering processing;

[0009] S2, extracting time features of the electroencephalogram signals by using LSTM;

[0010] S3, extracting time-frequency features of the electroencephalogram signals in different scales by using DAMSCN;

[0011] S4, introducing spatial attention mechanism and channel attention mechanism while extracting the multi-scale time-frequency features;

[0012] S5, splicing the extracted multi-scale time-frequency features and time features, and then realizing feature classification by means of a full connection layer and a SoftMax layer.

[0013] Further, the band-pass filtering processing comprises adopting 8-30Hz band-pass filtering processing on the original motor imagery electroencephalogram signals, so as to retain mu rhythm (8-13Hz) and beta rhythm (13-30Hz) related to motor imagery.

[0014] Further, the extracting time features of the electroencephalogram signals by using LSTM comprises performing dimension transposition on the electroencephalogram signals of three channels (C3, C4 and C z ), and then splicing into a one-dimensional vector, each LSTM unit processes information through three inputs: x t is the input of the current time step, h t-1 is the output of the previous LSTM unit, c t-1 is the cell state of the previous unit, the LSTM unit comprises three gate controls: a forgetting gate, an input gate and an output gate, the forgetting gate f t decides which information of the previous cell state is discarded by using a sigmoid function, the input gate i t decides whether the current state is updated by the current input, the output gate o t selects useful information from the current cell state and displays it as the output.

[0015] Further, at the end of the LSTM unit, a dropout layer is added.

[0016] Further, the use of DAMSCN extracts the time-frequency features of the electroencephalogram signals of different scales includes performing continuous wavelet processing on the electroencephalogram signals of the three channels, and splicing the obtained feature maps, and then inputting them into the DAMSCB for multi-scale feature extraction.

[0017] The input of the DAMSCB is H is the height of the feature map, W is the width of the feature map, C is the number of channels, and the formula for multi-scale feature extraction is as follows:

[0018]

[0019] In the formula, F l , l = 1, 2, 3, represents the features extracted by each branch with different receptive fields, and has the same size and depth as the input X, represents an inflation convolution layer based on a k x k convolution kernel; then pass into a batch normalization layer BN(·); finally use ReLu(·) as the activation function.

[0020] Further, the inflation convolution is obtained by inserting a gap between the kernel elements of the original convolution kernel w according to the inflation coefficient d, and the size of the two-dimensional inflation convolution kernel is defined as:

[0021] k' = (k - 1) d + 1

[0022] Where k is the size of the original convolution kernel, and k' is the effective kernel size for the inflation rate d.

[0023] Further, the spatial attention mechanism and the channel attention mechanism are added to each branch of the multi-scale feature extraction, and the obtained spatial attention map is:

[0024] SAM l = σ(C 5×5 ([MaxPool(F l ),AvgPool(F l )]))

[0025] Where C 5×5 (·) represents a convolution layer with a kernel size of 5, sigma(·) represents a Sigmoid function, MaxPool and AvgPool represent channel-level maximum pooling and average pooling operations respectively, and the output feature F l of each branch of the multi-scale feature extraction is element-wise multiplied by the spatial attention map SAM l to obtain the output of the spatial attention and input into the subsequent network layer, and the calculation formula is as follows:

[0026] The advantages and beneficial effects of the present application are as follows:

[0027] In view of the problem that time information is often ignored, a single scale convolution kernel is used, rich features cannot be obtained, and classification performance is poor when deep learning is used for classification of multi-class motor imagery electroencephalogram signals in the prior art, a motor imagery electroencephalogram signal classification algorithm based on parallel DAMSCN-LSTM is proposed. Through testing on a public data set, experiments show that the features of the time-frequency graph of the electroencephalogram signal are extracted by using multi-scale convolution, the spatial attention and channel attention modules are introduced, then the time features extracted by the LSTM are fused with the multi-scale time-frequency features, and thus the recognition accuracy of the multi-class motor imagery electroencephalogram signal is improved. Compared with other algorithms, the algorithm proposed in the present application has stronger robustness and higher accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a structural diagram of the present application;

[0029] Figure 2 is a key module DAMSCB structure of the DAMSCN branch;

[0030] Figure 3 is a specific spatial attention module;

[0031] Figure 4 is an LSTM unit structure diagram. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and in detail below with reference to the drawings in the embodiments of the present application. The described embodiments are only some of the embodiments of the present application.

[0033] As Figure 1 shown, the technical solutions of the present application include the following steps:

[0034] S1, four types of motor imagery electroencephalogram signals are preprocessed, including removing electrooculogram and electromyogram, and performing band-pass filtering. The original motor imagery electroencephalogram signal is subjected to electrooculogram and electromyogram removal, and is subjected to band-pass filtering with a band-pass of 8-30 Hz, so as to retain the mu rhythm (8-13 Hz) and beta rhythm (13-30 Hz) related to motor imagery.

[0035] S2, the time features of the electroencephalogram signal are extracted by using the LSTM. On the LSTM branch, the time features of the electroencephalogram signal are extracted by using the LSTM. zThe EEG data of the brain is first transposed in dimension and then spliced into a one-dimensional variable to improve the training speed of the LSTM. At the end of the LSTM branch, a dropout layer is added to prevent overfitting when training the model with small samples and increase the robustness of the model. LSTM controls the transmission of sequence information and learns long-term and short-term dependencies in sequence data through self-connected storage units that store network time states. As shown in Figure 4 , each LSTM unit processes information through three inputs: x t is the input of the current time step, h t-1 is the output of the previous LSTM unit, and c t-1 is the cell state of the previous unit. The LSTM unit is controlled by three gates: the forget gate, the input gate, and the output gate. The forget gate f t decides which information of the previous cell state to discard using the sigmoid function. The input gate i t decides whether the current state is updated by the current input. The output gate o t selects useful information from the current cell state and displays it as output. Each gate acts as a simple neural network, with W i , W f , W c , and W o as weights, and b i , b f , b c , and b o as the biases of the input gate, the forget gate, the temporary unit gate, and the output gate, respectively. The equations of the LSTM unit are as follows.

[0036] f t =σ(W f [h t-1 ,x t ]+b f )

[0037] i t =σ(W i [h t-1 ,x t ]+b i )

[0038]

[0039]

[0040] o t =σ(W o [h t-1 ,x t ]+b o )

[0041] ht =o t *tanh(c t )

[0042] σ(·) represents the Sigmoid function, and tanh is the hyperbolic tangent function. Input x represents the current time step. t The information provided indicates that the operator * represents the dot product of two vectors.

[0043] S3, as Figure 2 As shown, DAMSCN is used to extract time-frequency features of EEG signals at different scales. The EEG signals from the three channels are subjected to continuous wavelet processing, and the resulting feature maps are stitched together. Then, the data is input into DAMSCB for multi-scale feature extraction. To achieve different receptive fields, dilated convolution is employed in this invention. Dilated convolution is obtained by inserting gaps between kernels of the original convolution kernel w according to the dilation coefficient d. Mathematically, the size of a two-dimensional dilated convolution kernel can be defined as:

[0044] k'=(k-1)d+1

[0045] Where k is the original convolution kernel size, and k' is the effective kernel size for the expansion rate d.

[0046] The input for DAMSCB is H is the height of the feature map, W is the width of the feature map, and C is the number of channels. This represents a three-dimensional vector space. The formula for multi-scale feature extraction is as follows:

[0047]

[0048] Here F l l = 1, 2, 3, representing the features extracted by each branch with different receptive fields, having the same size and depth as the input X. This represents a dilated convolutional layer based on a k×k convolutional kernel; then it is passed into a batch normalized layer BN(·); finally, ReLU(·) is used as the activation function.

[0049] S4, while extracting multi-scale time-frequency features, introduces spatial attention and channel attention mechanisms, such as... Figure 2 As shown. This invention adds a spatial attention mechanism to each branch of multi-scale feature extraction, and its specific structure is as follows. Figure 3 As shown, the input feature maps are subjected to max pooling and average pooling operations respectively, and then the resulting feature maps are concatenated, followed by convolutional dimensionality reduction and the Sigmoid function. The final spatial attention map is as follows:

[0050] SAM l =σ(C 5×5([MaxPool(F l ),AvgPool(F l )]))

[0051] where C 5×5 (·) denotes a convolutional layer with kernel size 5, σ(·) denotes a Sigmoid function, MaxPool and AvgPool represent channel-wise max-pooling and average-pooling operations, respectively. The output feature F l of each branch of the multi-scale feature extraction is obtained. l Element-wise multiplication is performed to obtain the output of the spatial attention SAM which is input into the subsequent network layer, and the calculation formula is as follows:

[0052]

[0053] The feature maps of each branch are fused by using a splicing operation, and the result is as follows:

[0054]

[0055] It is noted that different sizes of features have different contributions to the final classification. In order to enhance the representation ability of multi-scale features, a channel attention mechanism is used to explicitly model the mutual dependence between feature channels to improve the quality of feature representation. The present application uses a squeeze-and-excitation block (SE) unit to improve the attention of the information feature mapping generated in the feature fusion process. The SE block mainly includes three steps of operation: squeezing, excitation and scale.

[0056] The first step is to compress the global spatial information by using global average pooling to obtain the importance of the feature map, which is defined as:

[0057]

[0058] where F sq (·) denotes a compression operation, m denotes a compression vector generated by the compression operation, and L is the size of the feature map, which is the product of the width and the height.

[0059] The second step is the excitation operation, which is defined as:

[0060] S=F ex (m,W)

[0061] where F ex (·) denotes an excitation operation, S is the output of the operation, which can depict the importance of different channel features, and W is an adjustment based on the scaling parameter (ratio) for the operation, which aims to reduce the number of channels to reduce the calculation amount.

[0062] Step 3 is to assign the feature map F ff The importance degree scale operation of each channel is to multiply the channel weight value S calculated by the SE module with the two-dimensional matrix of the corresponding channel of the original feature map respectively:

[0063] F out = F scale (F ff , S) = F ff · S

[0064] Wherein, · represents the matrix multiplication operation, F scale (·) represents the weight assignment operation.

[0065] S5, the extracted multi-scale time-frequency features and the features of the time features are spliced, and then the classification of the features is realized by means of the full connection layer and the SoftMax layer. First, the multi-scale time-frequency features extracted by the DAMSCN branch are flattened to obtain a one-dimensional vector, then the one-dimensional vector is spliced with the time features extracted by the LSTM branch, and finally the classification of the four types of motor imagery electroencephalogram signals is realized through two full connection layers and a SoftMax layer.

[0066] The above examples should be understood as only for illustrating the present application and not for limiting the protection scope of the present application. After reading the content of the present application, the skilled in the art can make various changes or modifications to the present application, and these equivalent changes and modifications also fall within the scope defined by the claims of the present application.

Claims

1. A method for classifying motor imagery EEG signals based on parallel DAMSCN-LSTM, characterized in that, Includes the following steps: S1 preprocesses the four types of motor imagery EEG signals, including removing electrooculography and electromyography, and performing bandpass filtering. S2 utilizes LSTM to extract the temporal features of EEG signals, including processing three channels (C3, C4, C5). z The EEG signals are transposed and then concatenated into a one-dimensional vector. Each LSTM unit processes information through three inputs: x t The input is the current time step, h. t-1 It is the output of the previous LSTM unit, c t-1 This refers to the cell state of the previous unit. An LSTM unit includes three gate controls: the forget gate, the input gate, and the output gate. The forget gate f... t The input gate i determines which information from the previous cell state to discard using the sigmoid function. t The output gate determines whether the current state is updated by the current input. t Select useful information from the current cell state and display it as output; S3, using DAMSCN to extract time-frequency features of EEG signals at different scales, including: performing continuous wavelet processing on the EEG signals of the three channels, stitching the resulting feature maps together, and then inputting them into DAMSCB for multi-scale feature extraction. The input for DAMSCB is H is the height of the feature map, W is the width of the feature map, and C is the number of channels. The formula for multi-scale feature extraction is as follows: In the formula F l l = 1, 2, 3, representing the features extracted by each branch with different receptive fields, having the same size and depth as the input X. This represents a dilated convolutional layer based on a k×k convolutional kernel; then it is passed into a batch normalized layer BN(·); finally, ReLU(·) is used as the activation function. S4 extracts multi-scale time-frequency features while introducing spatial attention and channel attention mechanisms; By applying the spatial attention mechanism and channel attention mechanism to each branch of the multi-scale feature extraction, the resulting spatial attention map is as follows: ALONE l =σ(C 5×5 ([MaxPool(F l ),AvgPool(F l )])) Where C 5×5 (·) represents a convolutional layer with a kernel size of 5, σ(·) represents the Sigmoid function, MaxPool and AvgPool represent channel-level max pooling and average pooling operations, respectively, and multi-scale feature extraction extracts the output features F of each branch. l Spatial Attention Diagram (SAM) l Element-wise multiplication yields the output of spatial attention. And input it into subsequent network layers, the calculation formula is as follows: S5 concatenates the extracted multi-scale time-frequency features and time features, and then uses a fully connected layer and a SoftMax layer to achieve feature classification.

2. The method for classifying motor imagery EEG signals based on parallel DAMSCN-LSTM according to claim 1, characterized in that: The bandpass filtering process includes applying an 8-30 Hz bandpass filter to the original motor imagery EEG signal to preserve the mu and beta rhythms associated with motor imagery.

3. The method for classifying motor imagery EEG signals based on parallel DAMSCN-LSTM according to claim 1, characterized in that: At the end of the LSTM unit, a dropout layer is added.

4. The method for classifying motor imagery EEG signals based on parallel DAMSCN-LSTM according to claim 1, characterized in that: The dilated convolution is obtained by inserting gaps between kernel elements of the original convolution kernel w based on the dilation coefficient d. The size of the two-dimensional dilated convolution kernel is defined as: k'=(k-1)d+1 Where k is the original convolution kernel size, and k' is the effective kernel size for the expansion rate d.

Images