A motor imagery electroencephalogram signal classification method based on self-attention mechanism and parallel convolution

Abstract

A motor imagery electroencephalogram signal classification method based on multi-head self-attention mechanism and parallel convolution belongs to the field of computer software. In view of the problem that the low signal-to-noise ratio of the electroencephalogram signal leads to difficult feature extraction, an improved network model based on EEGNet is proposed, which is referred to as EEG-MATCNet. First, the original electroencephalogram signal is subjected to preliminary feature extraction by using a parallel convolution layer, and different scale convolution kernels can extract time features of different time steps. At the same time, the attention weights of the electroencephalogram signals between the electrodes are calculated by using a multi-head self-attention mechanism, so that the network can better extract spatial features during training. In addition, the receptive field of the convolution kernel is improved by using a time convolution network, so that the model can extract higher-level time features. Experiments prove that the classification method proposed in the application can more effectively improve the feature extraction and classification performance of the motor imagery electroencephalogram signal.

Description

Technical Field

[0001] This invention discloses a method for classifying motor imagery EEG signals based on self-attention mechanism and parallel convolution, which can be used for decoding motor imagery EEG signals and belongs to the field of computer science. Background Technology

[0002] Brain-computer interface (BCI) technology is a cutting-edge research direction integrating multiple disciplines. It involves acquiring and decoding brainwave signals through devices, converting them into commands, and then forwarding them to output devices to execute the required operations. It has been widely applied in biomedicine, entertainment, education, smart homes, military, and many other fields. Brain activity can be recorded using various neuroimaging methods, which can be invasive or non-invasive. Currently, the most popular non-invasive method for acquiring brainwaves in BCIs is electroencephalography (EEG). The widespread adoption of EEG is due to the low cost of the devices, the reduction of complications compared to invasive surgery, portability, ease of setup and use, and the possibility of directly measuring neural activity. In EEG-based BCIs, motor imagery (MI) is a classic paradigm. Motor imagery EEG signals are electrical signals emitted on the scalp when a person imagines the movement of different parts of their body. When a person performs a hand-motor imagery task, the amplitudes of alpha waves (8-12Hz) and beta waves (13-30Hz) in the sensorimotor EEG signal on the contralateral side of the moving hand decrease, a phenomenon known as event-related desynchronization (ERD), while the amplitudes of alpha and beta waves in the sensorimotor EEG signal on the ipsilateral side of the moving hand increase, a phenomenon known as event-related synchronization (ERS). This pattern can be used to interpret people's intentions. The motor imagery paradigm is considered one of the most promising paradigms for assisting in the rehabilitation of patients with quadriplegia, spinal cord injuries, and amyotrophic lateral sclerosis (ALS). Although brain-computer interface technology based on the motor imagery paradigm has been widely used in rehabilitation medicine and other fields, its decoding performance still cannot fully meet the needs of practical applications. This is because EEG signals are non-Gaussian, non-stationary, and nonlinear, and the acquired EEG signals are easily affected by external noise (such as power frequency interference from electrical equipment) and internal noise (such as physiological sources and electrooculogram signals). Furthermore, due to physiological differences among individuals, the EEG signals of different subjects performing the same motor imagery task can vary significantly; the EEG signals of the same subject performing the same imagery task at different times can also vary considerably. Therefore, extracting effective features from motor imagery EEG signals and decoding them correctly remains a challenging problem.

[0003] The common algorithm flow for classifying motor imagery EEG signals is shown in the figure below, typically including four parts: preprocessing, feature extraction, feature selection, and classification. Traditional feature extraction methods mainly target the time, frequency, or spatial characteristics of EEG signals. For example, temporal features at different time points or time periods can be extracted in the time domain using mean, variance, Hjorth parameter, skewness, etc. Wavelet transform, power spectral density, and fast Fourier transform can be used to extract time-frequency domain features of the original EEG signal. Common Spatial Pattern (CSP) and its variants are used to extract spatial domain features of the EEG signal. The extracted features are then classified using methods such as linear discriminant analysis, support vector machines, neural networks, and Bayesian classifiers. However, traditional methods usually require rich prior knowledge and extensive feature selection processes. With the rise of deep learning, more and more researchers are attempting to apply end-to-end deep models to the classification of motor imagery EEG signals and have achieved good results. Lawhen et al. proposed EEGNet, a compact CNN-based network that incorporates convolutions along both the temporal and spatial dimensions. It demonstrated stable performance across four different paradigms with a significant reduction in training parameters, achieving good experimental results and high accuracy in both cross-subject and single-subject scenarios. Dai et al. proposed a hybrid convolutional neural network (HS-CNN), which effectively addresses the limitation of classification performance caused by using a single convolutional scale in CNNs, further improving classification accuracy. However, there is still room for further optimization in these studies.

[0004] Recently, a new CNN variant called Temporal Convolutional Network (TCN) has emerged, specifically designed for time series modeling and classification. TCN outperforms recurrent networks like LSTM and GRU in many sequence-related tasks. Compared to typical CNNs, TCNs can exponentially expand their receptive field size with a linear increase in the number of parameters, and unlike RNNs, they are not affected by vanishing or exploding gradients. Ingolfsson et al. proposed a TCN model called EEG-TCN, which combines TCN with the well-known EEGNet architecture. Attention mechanisms are an effort to mimic the behavior of the human brain, selectively focusing on some important elements while ignoring others. Integrating attention mechanisms with deep learning models helps to automatically (through learning) focus on the most important parts of the input data. In 2017, Google researchers proposed a pure attention model with multi-head attention, consisting of multiple self-attention layers. Self-attention mechanisms help the model focus on the most effective channel information in the data; multiple heads help focus on multiple locations, resulting in multiple attentional representations. This further enhances the deep learning network's ability to learn features. Compared to convolutional neural networks, self-attention mechanisms can directly compute information at different locations within each sequence and calculate a comprehensive representation of the sequence. Therefore, the effectiveness of integrating self-attention mechanisms into the classification of motor imagery EEG signals warrants further investigation.

[0005] By analyzing the advantages and disadvantages of existing methods, this invention provides inspiration and research ideas. Based on EEGNet, a novel improved network model (EEG-MATCNet) is proposed. This model uses parallel convolutional layers instead of the ordinary convolutional layers in the EEGNet model to encode motor imagery EEG signals into high-level time series. Then, a self-attention layer highlights the most valuable information in the time series. Finally, a TCN is used to extract high-level temporal features from the highlighted information for classification. Compared to EEGNet and HS-CNN models, the classification method proposed in this invention can more effectively improve the decoding performance of motor imagery EEG signals. Summary of the Invention

[0006] This invention proposes a classification method for motor imagery EEG signals based on self-attention mechanisms and parallel convolutions. This method effectively improves the feature extraction and classification performance of motor imagery EEG signals. Addressing the challenge of feature extraction due to the low signal-to-noise ratio of EEG signals, parallel temporal convolutional layers are used instead of ordinary convolutional layers in the EEGNet model to better extract features and thus improve classification accuracy. Simultaneously, a self-attention mechanism is employed to focus on the most effective channel information in the EEG data. Finally, a TCN module extracts high-level temporal features from the highlighted information for classification. Compared to models such as EEGNet and HS-CNN, the proposed method achieves higher classification accuracy.

[0007] To achieve the above-mentioned objectives, after research, discussion, and repeated practice, the final solution determined by this method is as follows:

[0008] First, the original motor imagery EEG dataset is preprocessed. Then, the dataset is divided into three parts: training set, validation set, and test set. These parts are then input into the constructed EEG-MATCNet model for training and testing, respectively. Finally, the model classification results are obtained, and the classification results are evaluated to verify the effectiveness of the method.

[0009] The specific steps of the technical solution of this invention are as follows:

[0010] Step 1. Data Preprocessing: First, the collected motor imagery EEG signals are common-mean referenced, and then bandpass filtered. After that, the filtered signals are standardized by exponential moving mean. The preprocessed EEG signal dataset for training is divided into training and validation sets in a 4:1 ratio for subsequent 5-fold cross-validation.

[0011] Step 2. Construct the EEG-MATCNet model: Use parallel multi-scale temporal convolutional layers to replace the ordinary convolutional layers in the EEGNet model to obtain EEG signal features with mixed temporal scales; add a multi-head self-attention module (MSA) to calculate attention weights for EEG signals from different electrodes to enhance the extraction of spatial features; finally, use the TCN module to extract high-level temporal features from the highlighted information for classification.

[0012] Step 3. Input the training set and validation set from Step 1 into the EEG-MATCNet model for training;

[0013] Step 4. Input the test set from Step 1 into the model trained in Step 3 for classification and evaluate the classification accuracy.

[0014] The present invention has the following advantages:

[0015] 1. Compared to networks with single-scale convolutional layers, networks using parallel multi-scale temporal convolutional layers can simultaneously extract local and long-range temporal features from EEG signals. This achieves better accuracy and efficiency, thereby further improving the accuracy of motor imagery classification tasks.

[0016] 2. Introducing a multi-head self-attention module allows the network to focus more on the most effective channel information in the EEG data during training. Multiple heads help to focus on multiple locations, thus generating multiple attentional representations. At the same time, the multi-head self-attention mechanism can compute attention weights in parallel with minimal impact on model performance.

[0017] 3. Introducing the TCN module to extract advanced temporal features from time series data, and continuously expanding the receptive field by stacking TCN layers, and effectively avoiding gradient vanishing and gradient exploding problems through residual structures. Attached Figure Description

[0018] Figure 1 Overall flowchart of the present invention

[0019] Figure 2 EEG-MATCNet network structure diagram

[0020] Figure 3 Temporal convolutional network diagram Detailed Implementation

[0021] This invention addresses the problem of low signal-to-noise ratio in EEG signals, which leads to difficulties in feature extraction and classification. It proposes a method for classifying motor imagery EEG signals based on self-attention mechanisms and parallel convolution. By replacing the ordinary convolutional layers in the EEGNet model with mixed-scale temporal convolutional layers, the method better extracts temporal features, thereby improving classification accuracy. Simultaneously, the addition of a multi-head self-attention module (MSA) and a temporal convolutional network (TCN) allows the network to focus on global features of the sequence during training, expanding the receptive field and further improving model performance. This provides an efficient and higher-performance deep learning method for classifying motor imagery EEG signals. Figure 1 The present invention can be broken down into the following steps as a whole flowchart.

[0022] Step 1: Data preprocessing, including dividing the dataset.

[0023] Step 2: Construct the EEG-MATCNet model.

[0024] Step 3: Train the model using the training set and validation set.

[0025] Step four: Test the model's performance and evaluate its classification accuracy.

[0026] The specific details of each step are explained below:

[0027] Step 1:

[0028] (1) Perform common-average reference on the raw EEG signals;

[0029] (2) Extracting 4-40Hz EEG signals using a third-order Butterworth bandpass filter;

[0030] (3) The filtered EEG signal was standardized by exponential moving mean, with the attenuation factor set to 0.999;

[0031] (4) Divide the training set into a training set and a validation set in a 4:1 ratio for subsequent 5-fold cross-validation;

[0032] (5) The EEG signal is segmented and selected. Each segment represents a complete motor imagery EEG task. The length of each segment of the EEG signal is 4s.

[0033] Step 2:

[0034] To address the challenges of feature extraction and classification difficulties caused by the nonlinearity, non-stationarity, and low signal-to-noise ratio of electroencephalogram (EEG) signals, this invention proposes a novel model improvement method based on EEGNet, abbreviated as EEG-MATCNet. The network structure of this method is as follows: Figure 2 As shown, the model mainly consists of four parts: parallel convolutional layers, self-attention layers, temporal convolutional network layers, and fully connected layers. PyTorch is used to build the model. Each part is explained in detail below:

[0035] (1) Parallel convolutional layers

[0036] The initial temporal feature extraction of the input EEG signal is performed using convolutional kernels of different sizes in three branches. After multiple experiments, the optimal parallel structure is as follows: Branch 1 uses two convolutional kernels with a kernel size of (1,16) and a stride of 1. Branch 2 uses four convolutional kernels with a kernel size of (1,32) and a stride of 1. Branch 3 uses eight convolutional kernels with a kernel size of (1,64) and a stride of 1 for feature extraction. The kernel padding mode of all three branches is set to "same". Batch normalization layers are then used to prevent gradient vanishing during network training and reduce overfitting. Finally, the ELU activation function is used to help the network converge faster.

[0037] (2) Self-attention layer

[0038] First, the EEG signals from each electrode, after preliminary temporal feature extraction, are transformed through a linear layer to obtain three vectors: a query vector (Q), a key vector (K), and a value vector (V). Then, the query vector from each electrode is multiplied by the key vectors from all electrodes using a scaled dot product attention mechanism. Finally, the product is calculated using the dimension d of the key vector. ｋ Normalization is performed to obtain the attention weights. Calculation is performed for each query vector in the sequence, resulting in a vector with the same length as the input sequence and the same dimensions as the weight matrix. The formula for calculating the output vector obtained from the above process is shown below:

[0039]

[0040] After obtaining the enhanced feature vectors along the spatial axis, the C-dimensional EEG features are flattened into 1-dimensional features by performing a depthwise convolution with a kernel size of (C, 1) and a stride of 1, where C represents the number of electrodes. The resulting features are then passed through an average pooling layer with a kernel size of (1, 4) and a stride of (1, 4) to reduce the sampling rate. To prevent overfitting, a Dropout mechanism is introduced to randomly discard parameters learned in the previous layer, with a dropout ratio of 0.5. The features are then input into a separable convolutional layer, which includes both depthwise convolution and pointwise convolution. The depthwise convolution kernel size is set to (1, 16), the stride is 1, and the padding mode is set to "same." The pointwise convolution kernel size is set to (1, 1), the stride is 1, and the padding mode is set to "same." The features are then processed sequentially through a batch normalization layer, an activation function layer, an average pooling layer, and a random parameter dropout layer. The average pooling layer has a size and stride of (1, 8) to reduce the number of parameters. The proportion of random discards is set to 0.5 to prevent overfitting.

[0041] (3) Temporal Convolutional Network Layer

[0042] To expand the receptive field of the convolutional kernels, two temporal convolutional layers are introduced, and residual connections are used to avoid the gradient vanishing problem that may occur during network training. The first temporal convolutional layer uses a causal dilated convolution with a kernel size of 4 and a dilation factor of 1, ensuring that each point in the network output contains feature information from the previous four points. The second temporal convolutional layer uses a causal dilated convolution with a kernel size of 4 and a dilation factor of 2, further expanding the receptive field of the convolutional kernels. The specific network structure of the temporal convolutional network layers is shown in the appendix of the specification. Figure 3 As shown.

[0043] (4) Fully connected layer

[0044] The obtained high-level temporal features are stacked and then fed into a fully connected layer. Simultaneously, a maximum norm constraint is added to the fully connected layer for regularization, with the maximum norm value set to 0.25 to prevent overfitting and improve the model's generalization ability.

[0045] Finally, it is input into the Softmax classifier for classification to obtain the final judgment of the motion imagery task type.

[0046] Step 3:

[0047] During the training phase, a 5-fold cross-validation method was used. The training set was divided into five equal parts, and five experiments were conducted. In each experiment, four different parts of the data were used as the training set, and the remaining part was used as the validation set. The training set was input into the EEG-MATCNet model for training. Each time, 64 segments of EEG signals were input into the network for training, and the process was iterated for 1000 epochs. During training, the optimal loss value was recorded using the cross-entropy loss function. If the loss value was lower than the optimal loss value for 300 consecutive epochs, the iteration was terminated early, and the average accuracy of the training and validation sets was recorded, and the weights of the optimal model were saved. The Adam optimizer was used to mitigate gradient oscillations during network training, and the learning rate was set to 0.001. The above model was trained and tested on nine subjects, and nine sets of validation set accuracies were obtained. The average of these accuracies was recorded as the final model accuracy.

[0048] Step 4:

[0049] Input the test set from step 1 into the model trained in step 3 for classification and recognition, and evaluate the accuracy of the classification.

[0050] The dataset and experimental results used in the method of this invention are described below:

[0051] 1. Dataset

[0052] This invention uses two publicly available datasets, BCI Competition IV Datasets 2a and 2b, for experiments, with the time-based diagrams of the datasets shown below. Figure 3 As shown, all experimental data have been preprocessed using a bandpass filter in the range of 0.5–100 Hz.

[0053] Dataset 2a contains four categories of motor imagery EEG tasks: left hand, right hand, both feet, and tongue. It was provided by nine healthy participants. Each participant underwent two rounds of testing, each round consisting of 288 motor imagery tasks, with 72 trials per category. The first round served as the training set, and the data from the second round served as the test set. EEG signals were acquired from 22 electrodes at a sampling rate of 250 Hz. Data from 0.5 to 2.5 seconds after the prompt was displayed was extracted as a sample. All samples were labeled (i.e., indicating which body part the sample corresponded to in the motor imagery).

[0054] Dataset 2b contains EEG signals from nine subjects, covering two categories: left and right hands, for motor imagery. These EEG signals were acquired from three electrodes at a sampling rate of 250 Hz. For each subject, the motor imagery task was divided into five sessions. (Compared to 2a...)

[0055] The datasets differ. In dataset 2b, the first two sessions were conducted without feedback, representing EEG imagery data without visual feedback, while the latter three sessions included EEG imagery data with visual feedback. For dataset 2b, data from 0.5 seconds to 4 seconds after the cue was displayed was extracted as a single sample.

[0056] Because the EEG characteristics of different subjects vary greatly, the classification experiment of EEG signals completed in this invention needs to calculate the classification accuracy for each subject individually, and then calculate the average of the classification accuracy of multiple subjects as the performance index of the model.

[0057] 2. Experimental Results and Discussion

[0058] To verify the effectiveness and universality of the method of the present invention, comparative experiments and ablation experiments were conducted on public datasets 2a and 2b, respectively. The experimental results are as follows:

[0059] Cross-dataset comparison experiments were conducted using datasets 2a and 2b to compare the proposed method with the EEGNet method. The experimental results are shown in the table below:

[0060] Table 1. Results of cross-dataset comparison experiments

[0061]

[0062] The experimental results show that EEG-MATCNet outperforms EEGNet in the average classification accuracy of 9 subjects on both datasets 2a and 2b. On dataset 2a, it can improve the accuracy by up to 6.45%, and on dataset 2b, it can improve the accuracy by up to 8%.

[0063] To verify the effect of the self-attention layer, the attention mechanism of the proposed method was replaced on the 2a dataset. The experimental data are shown in the table below:

[0064] Table 2 Ablation Experiment Results for Dataset 2a

[0065]

[0066] As shown in Table 2, the classification accuracy of the model is higher than that of EEGNet but lower than that of the complete method when the self-attention mechanism, parallel convolution and temporal convolution network are removed. This shows that all three schemes are effective.

Claims

1. A method for classifying motor imagery EEG signals based on self-attention mechanism and parallel convolution, characterized in that, Includes the following steps: Step 1, Data Preprocessing: Bandpass filtering is applied to the motor imagery EEG signals, followed by exponential moving mean standardization of the filtered signals; the EEG signal dataset is divided into training, validation, and test sets. Step 2: Construct the EEG-MATCNet model; use parallel convolutional layers to replace the ordinary convolutional layers in the EEGNet model to extract multi-scale temporal features, add a spatial self-attention mechanism to enable the network to extract spatial features better, and add a temporal convolutional network to extract high-level temporal features. Step 3: Input the training set and validation set from Step 1 into EEG-MATCNet for training; Step 4: Input the test set from Step 1 into the model trained in Step 3 for classification, and evaluate the classification accuracy; Step 2 is as follows: The EEG-MATCNet structure can be summarized into four main parts: parallel convolutional layers, self-attention layers, temporal convolutional network layers, and fully connected layers. The model is built using PyTorch. Each part will be described in detail below. (1) Parallel convolutional layers The initial temporal features of the input EEG signal are extracted using convolutional kernels of different sizes in three branches. After multiple experiments, the optimal parallel structure is as follows: Branch 1 uses two convolutional kernels with a kernel size of (1,16) and a stride of 1; Branch 2 uses four convolutional kernels with a kernel size of (1,32) and a stride of 1; and Branch 3 uses eight convolutional kernels with a kernel size of (1,64) and a stride of 1 for feature extraction. The kernel padding mode of all three branches is set to "same". Then, a batch normalization layer is used to prevent gradient vanishing during network training, and finally, the ELU activation function is used to help the network converge faster. (2) Self-attention layer First, the EEG signals from each electrode, after preliminary temporal feature extraction, are transformed through a linear layer to obtain three vectors: a query vector (Q), a key vector (K), and a value vector (V). Then, the query vector from each electrode is multiplied by the key vectors from all electrodes using a scaled dot product attention mechanism. Finally, the product is calculated using the dimension d of the key vector. k Normalization is performed to obtain the attention weights; calculations are performed on each query vector in the sequence to obtain a vector with the same length as the input sequence and the same dimensions as the weight matrix; the formula for calculating the output vector obtained from the above process is shown below: After obtaining the enhanced feature vectors along the spatial axis, the C-dimensional EEG features are flattened into 1-dimensional features by using a depthwise convolution with a kernel size of (C, 1) and a stride of 1, where C represents the number of electrodes. The resulting features are then passed through an average pooling layer with a kernel size of (1, 4) and a stride of (1, 4) to reduce the sampling rate. To prevent overfitting, a Dropout mechanism is introduced to randomly discard parameters learned in the previous layer, with a discard ratio set to 0.

5. The features are then input into a separable convolutional layer. Off-convolution includes two operations: depthwise convolution and pointwise convolution. The kernel size for depthwise convolution is set to (1, 16), the stride is 1, and the padding mode is set to "same". The kernel size for pointwise convolution is set to (1, 1), the stride is 1, and the padding mode is set to "same". After that, the convolution is processed sequentially through a batch normalization layer, an ELU activation function layer, an average pooling layer, and a random parameter dropout layer. The size and stride of the average pooling layer are both (1, 8) to reduce the number of parameters. The random dropout ratio is set to 0.5 to prevent overfitting. (3) Temporal Convolutional Network Layer To expand the receptive field of the convolutional kernel, two temporal convolutional modules were introduced, and the gradient vanishing problem that may occur during network training was avoided by using residual connections. The kernel size of the causal dilation convolution in the first temporal convolutional module was set to 4, and the dilation factor was set to 1, which ensured that each point of the network output contained the feature information of the first four points. The kernel size of the causal dilation convolution in the second temporal convolutional module was set to 4, and the dilation factor was set to 2, which further expanded the receptive field of the convolutional kernel. (4) Fully connected layer The obtained high-level temporal features are superimposed and input into a fully connected layer. At the same time, a maximum norm constraint is added to the fully connected layer for regularization. The maximum norm value is set to 0.25 to prevent overfitting. Finally, it is input into a Softmax classifier for classification to obtain the final judgment of the motion imagery task type.

2. The method for classifying motor imagery EEG signals based on self-attention mechanism and parallel convolution as described in claim 1, characterized in that: Step 1 is as follows: (1) Common averaging reference was performed on the raw EEG signals; (2) Extracting 4-40Hz EEG signals using a third-order Butterworth bandpass filter; (3) The filtered EEG signal was standardized by exponential moving mean, with the attenuation factor set to 0.999 to reduce the impact of numerical differences on the model performance. (4) Divide the training set into a training set and a validation set in a 4:1 ratio for subsequent 5-fold cross-validation; (5) The EEG signal is segmented and each segment represents a complete motor imagery EEG task. The length of each segment of the EEG signal is 4s.

3. The method for classifying motor imagery EEG signals based on self-attention mechanism and parallel convolution as described in claim 1, characterized in that: Step 3 specifically involves: During the training phase, a 5-fold cross-validation method was used. The training set was divided into 5 equal parts, and a total of 5 experiments were conducted. In each experiment, 4 different parts of the data were used as the training set, and the remaining part was used as the validation set. The training set was input into the EEG-MATCNet model for training. 64 EEG signals were input into the network for training each time, and the process was iterated for 1000 rounds. During the training process, the optimal loss value was recorded using the cross-entropy loss function. If the loss value was lower than the optimal loss value for 300 consecutive rounds, the iteration was terminated early, and the average accuracy of the training set and validation set was recorded, and the weights of the optimal model were saved. The Adam optimizer was used to mitigate gradient oscillations during network training, and the learning rate was set to 0.

001. The above model was trained and tested on 9 subjects, and 9 sets of validation set accuracies were obtained. The average of these accuracies was recorded as the final model accuracy.

4. The method for classifying motor imagery EEG signals based on self-attention mechanism and parallel convolution as described in claim 1, characterized in that: Step 4 is as follows: Input the test set from step 1 into the model trained in step 3 for classification and recognition, and evaluate the accuracy of the classification.

Images