Motor imagery electroencephalogram decoding method based on multi-view space-time convolution and attention mechanism

Abstract

The invention discloses a motor imagery electroencephalogram decoding method based on multi-view space-time convolution and an attention mechanism, which comprises the following steps of: firstly, constructing multi-view data by utilizing a filter bank, and enhancing training data by adopting a frequency band sensing fragment splicing strategy so as to expand sample distribution; secondly, respectively and accurately capturing specific time sequence features and spatial features of different frequency bands through a frequency adaptive convolution kernel and depth separable convolution; meanwhile, in combination with a channel attention mechanism and a time sequence statistics module, feature channel weights are effectively utilized, and a mean value and a logarithmic variance in a time window are calculated, so that the modeling capability of the model for key spatio-temporal information is enhanced. According to the method, the problem of insufficient spatio-temporal feature mining in the motor imagery task is effectively solved, and a competitive result is obtained.

Description

Technical Field

[0001] This invention relates to the fields of brain-computer interface and biomedical signal processing technology, and in particular to a method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms. Background Technology

[0002] Brain-computer interface (BCI) is a technology that establishes a communication channel directly between the brain and external devices without relying on peripheral nerves and muscle tissue. It has broad application prospects in medical fields such as stroke rehabilitation and prosthetic control, as well as non-medical fields such as virtual reality and drone control. Among various brain signal acquisition methods, electroencephalography (EEG) has become the mainstream signal source for BCI systems due to its advantages such as non-invasiveness, high temporal resolution, and low cost.

[0003] However, accurate decoding of motor imagery EEG signals still faces many challenges: First, EEG signals are extremely weak (microvolt level), with a low signal-to-noise ratio, and are highly susceptible to interference from physiological artifacts such as electrooculography (EOG) and electromyography (EMG). Second, traditional machine learning decoding methods (such as Common Spatial Patterns (CSP) and its variants) typically rely on manually designed feature extraction, and feature extraction and classification are two independent stages, making global optimization difficult. Third, while existing deep learning methods (such as EEGNet and FBCNet) have improved performance to some extent, they often use fixed-size convolutional kernels to extract temporal information. Since different subjects or different frequency bands of EEG rhythms have different physiological characteristics and optimal response windows, single-scale convolutional kernels struggle to capture fine temporal dynamic features at specific frequencies. Furthermore, some methods based on self-attention mechanisms tend to focus on global features, easily neglecting key local dynamic changes, leading to homogenization of contributions from different EEG rhythms and limiting further improvements in classification accuracy.

[0004] Therefore, this invention proposes a decoding method that can adaptively extract multi-scale spatiotemporal features for different frequency band characteristics and effectively enhance the model's generalization ability under small sample data. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention proposes a motor imagery EEG decoding method based on multi-view spatiotemporal convolution and attention mechanism, which has achieved good results in recognizing motor imagery EEG signals.

[0006] The technical solution for realizing the present invention is as follows:

[0007] A method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms includes the following steps:

[0008] S1. Acquire raw EEG signals, and use a Chebyshev Type II filter bank to decompose the signals into multiple sub-signals of different frequency bands to construct multi-view EEG data X. f The dataset was then divided into a training set, a validation set, and a test set.

[0009] S2, the multi-view EEG data X in the training set f Data augmentation is performed by generating enhanced multi-view data X using a frequency band sensing segment stitching method. aug ;

[0010] S3. Construct the MSTCNet model, which includes a multi-view temporal convolution (MTC) module, a spatial convolution and attention (SCA) module, a temporal mean-log variance (TML) module, and a classification module.

[0011] S4. Transfer the enhanced multi-view data X aug The input MTC module extracts time-series dynamic features from sub-signals in different frequency bands and performs multi-view feature fusion to obtain the time-series fused feature Z. MTC ;

[0012] S5. The time-series fusion feature Z MTC The input SCA module extracts spatial features using depthwise separable convolution, and adaptively weights the feature channels using a channel attention mechanism to obtain the spatially enhanced feature Z. SCA ;

[0013] S6. The spatial enhancement feature Z SCA Input the TML module to calculate the mean and logarithmic variance within the time window, and concatenate them to obtain the joint statistical feature Z. TML ;

[0014] S7. The joint statistical feature Z TML The input classification module calculates the class probability distribution through a fully connected layer and a Softmax function to obtain the predicted probability vector;

[0015] S8. Calculate the cross-entropy loss based on the predicted probability vector and the true label, and train the MSTCNet model to obtain the trained MSTCNet model.

[0016] S9. Transfer the multi-view EEG data X from the test set. f The trained MSTCNet model is input, and the final motor imagery classification result of the EEG signal is output through forward propagation of the model.

[0017] Furthermore, in step 2, the multi-view EEG data X in the training set... f Data augmentation will be performed as follows:

[0018] For each frequency band b, it is uniformly divided into n parts on the time axis. seg Each segment, let's say This represents the j-th sub-segment after segmentation;

[0019] Randomly select n from the training dataset seg Data from a sample, and extracting sub-segments at the corresponding time positions for each sample. Where k j The index of the randomly selected sample;

[0020] These randomly sampled sub-segments are spliced ​​together according to the original time sequence to generate enhanced samples for that frequency band. It can be represented as:

[0021]

[0022] The above operations are performed in parallel on all frequency bands, and the enhanced samples from each frequency band are stitched together to obtain the enhanced multi-view data X. aug .

[0023] Furthermore, in step 4, temporal dynamic features are extracted for sub-signals of different frequency bands, using a temporal convolution kernel size L. i The calculation formula is:

[0024] Among them, F s Indicates the sampling rate. Let L represent the center frequency of the i-th frequency band, and k be the periodicity coverage coefficient; for low-frequency rhythms, k is set to 1.5; for high-frequency rhythms, k is set to 0.8; and L i Always choose odd numbers;

[0025] Subsequently, 1×1 convolutions are used to expand the channel dimension, and the outputs of all views are concatenated along the channel dimension. After batch normalization, the temporal fusion feature Z is obtained. MTC .

[0026] Furthermore, in step 5, spatial features are extracted using depthwise separable convolution, and the feature channels are adaptively weighted using a channel attention mechanism, as follows:

[0027] First, Z MTC Spatial filtering is performed by inputting a deep convolutional layer, and then cross-channel integration is performed by inputting a pointwise convolutional layer to obtain the spatial feature map X1.

[0028] Next, global spatial information of X1 is compressed using global average pooling, and channel importance weights α are generated through two fully connected layers. These weights are then used to recalibrate X1 to obtain X. SE ;

[0029] At the same time, residual paths are introduced to Z MTC X is obtained by adjusting the dimensions through convolution. res X SE With X res The addition yields the spatially enhanced feature Z. SCA Z SCA =X main +X res , where X main For X SE The output after being processed by Dropout.

[0030] Furthermore, in step 6, the mean and logarithmic variance within the time window are calculated, as follows:

[0031] The features are divided into n non-overlapping time windows along the time dimension, with a window size of ω;

[0032] Calculate the mean feature μ of the i-th window on the c-th channel. c,i :

[0033]

[0034] Calculate the log-variance feature of the i-th window on the c-th channel.

[0035]

[0036] The mean feature and the log-variance feature are concatenated to form the joint statistical feature Z. TML .

[0037] Furthermore, in step 1, the filter bank employs a frequency band division strategy tailored to the characteristics of motor imagery EEG rhythms, specifically configured as follows:

[0038] The frequency range of 4-40Hz is divided into 9 non-overlapping sub-bands, each with a bandwidth of 4Hz;

[0039] The frequency band division strategy is the time convolution kernel size L of the MTC module in step 4. i The calculation provides the center frequency parameter.

[0040] The beneficial effects of this invention are as follows:

[0041] (1) This invention designs a multi-view temporal convolution module (MTC) for decoding EEG signals of motor imagery. For the first time, it uses the sampling theorem to customize the size of the convolution kernel for different frequency bands, effectively capturing the specific temporal dynamic features of EEG signals of different frequencies, and solving the problem that the traditional method of using a fixed convolution kernel causes the loss of key local dynamic information in some frequency bands.

[0042] (2) This invention combines depthwise separable convolution with channel attention mechanism (SE Block) to construct SCA module and introduces residual path. While greatly reducing the number of model parameters, it enhances the ability to express spatial features of task-related brain regions (such as motor cortex) by adaptively learning feature channel weights.

[0043] (3) This invention proposes a frequency band-aware segment splicing data augmentation method. By asynchronously and randomly recombining different frequency band segments while maintaining the temporal continuity within the frequency band, the distribution of training samples is effectively expanded, and the generalization ability and anti-interference ability of the model under small sample data are significantly improved.

[0044] (4) The present invention designs a time mean-log variance module (TML), which, compared with the traditional global average pooling, uses log variance to capture the fluctuation characteristics of the signal within the time window and can more effectively extract discriminative information in the time-series dependencies at different stages. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a diagram of the network architecture of the model based on multi-view spatiotemporal convolution and attention mechanism in this invention;

[0047] Figure 2 This is a schematic diagram of the mid-frequency band sensing segment splicing data enhancement method of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of the multi-view temporal convolution module (MTC) in this invention;

[0049] Figure 4 This is a schematic diagram of the spatial convolution and attention module (SCA) in this invention;

[0050] Figure 5 This is a schematic diagram of the Time Series Mean-Logarithmic Variance (TML) module in this invention; This represents the splicing operation in the last dimension. (Represents element-wise addition);

[0051] Figure 6 This is an overall flowchart of the EEG decoding method for motor imagery based on multi-view spatiotemporal convolution and attention mechanism of the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0053] Combination Figure 1 (Model architecture diagram) Figure 2 (Data augmentation diagram) Figure 3 (Diagram of multi-view temporal convolution module) Figure 4 (Schematic diagram of spatial convolution and attention module) Figure 5 (Diagram of the time series mean-log variance module) and Figure 6 (Overall flowchart), a method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms, includes the following steps:

[0054] Step 1: First, acquire the raw electroencephalogram (EEG) signal data. To fully utilize frequency domain information, this embodiment uses a Chebyshev Type II filter bank to filter the raw signal. The stopband attenuation is set to 30dB, and the frequency range of 4-40Hz is divided into 9 non-overlapping sub-bands (e.g., 4-8Hz, 8-12Hz, ..., 36-40Hz) with a bandwidth increment of 4Hz. The filtered data is then used to construct multi-view EEG data. Where N b =9 represents the number of frequency bands, C represents the number of channels, and T represents the number of sampling points. Finally, the dataset is divided into training, validation, and test sets. Proceed to step 2.

[0055] Step 2: Perform data augmentation on the multi-view EEG data in the training set. To address the scarcity of EEG data, this invention proposes a frequency band sensing segment stitching method, as follows:

[0056] For each frequency band b, it is uniformly divided into n parts on the time axis. seg A single segment. Let... This represents the j-th sub-segment after segmentation;

[0057] Randomly select n from the training dataset seg Data from a sample, and extracting sub-segments at the corresponding time positions for each sample. Where k j The index of the randomly selected sample;

[0058] These randomly sampled sub-segments are spliced ​​together according to the original time sequence to generate enhanced samples for that frequency band. The formula is as follows: The above operations are performed in parallel on all frequency bands, and the enhanced samples from each frequency band are stitched together to obtain the enhanced multi-view data X. aug Proceed to step 3.

[0059] Step 3: Construct the MSTCNet model. The model includes a multi-view temporal convolution module (MTC module), a spatial convolution and attention module (SCA module), a temporal mean-log variance module (TML module), and a classification module. Proceed to Step 4.

[0060] Step 4: Transfer the enhanced multi-view data X aug The input MTC module extracts time-series dynamic features from sub-signals in different frequency bands and performs multi-view feature fusion, as detailed below:

[0061] Step 4.1: For the i-th frequency band branch, design a specific temporal convolution kernel size L based on the sampling theorem. i The calculation formula is:

[0062] Among them, F s This indicates the sampling rate (250Hz in this example). Let L represent the center frequency of the i-th frequency band, and k be the periodicity coverage coefficient. For low-frequency rhythms, k is set to 1.5; for high-frequency rhythms, k is set to 0.8. i Always choose odd numbers.

[0063] Step 4.2: Each branch undergoes a convolution with a kernel size of (1, L). i After temporal convolution, the channel dimension is expanded using 1×1 convolution, and the outputs of all views are concatenated along the channel dimension. After batch normalization (BN), the temporal fusion feature Z is obtained. MTC :

[0064]

[0065] in, This represents the output feature of the i-th branch. Proceed to step 5.

[0066] Step 5: Combine the temporal fusion features Z MTC The input SCA module extracts spatial features using depthwise separable convolutions and adaptively weights the feature channels using a channel attention mechanism, as detailed below:

[0067] Step 5.1: Spatial filtering is performed using depthwise convolution (DEP), followed by cross-channel integration using pointwise convolution (Pointwise Convolution), and the spatial feature map X1 is obtained by combining it with the Swish activation function.

[0068] X1 = Swish(BN(Conv) point (Conv depth (Z MTC ))))

[0069] Step 5.2: Calculate channel importance weights using SE Block. First, obtain the channel descriptor z by compressing global information through global average pooling. s Then, weights α are generated through two fully connected layers:

[0070]

[0071] α=σ(W2,δ(W1z s ))

[0072] Where W1 and W2 are weight matrices, δ is the ReLU activation function, and σ is the Sigmoid function. X1 is recalibrated using these weights to obtain X. SE =α·X1. Step 5.3: Introduce the residual path, and set Z... MTC X is obtained by adjusting the dimensions through convolution. res , and the main path output X after Dropout main Adding them together yields the spatially enhanced feature Z. SCA Z SCA =X main +X res Proceed to step 6.

[0073] Step 6: Augment the spatial feature Z SCA Input the TML module to calculate the mean and log-variance within the time window, as follows:

[0074] The features are divided into n non-overlapping time windows in the time dimension (n=4 in this embodiment), and the window size is ω.

[0075] Calculate the mean feature μ of the i-th window on the c-th channel. c,i :

[0076] Calculate the log-variance feature of the i-th window on the c-th channel.

[0077]

[0078] The mean feature and the log-variance feature are concatenated to form the joint statistical feature Z. TML Proceed to step 7.

[0079] Step 7: Combine the joint statistical features Z TML The input classification module is mapped through a fully connected layer, and the class probability is calculated using the LogSoftmax function, which serves as the model's final predicted classification result. Proceed to step 8.

[0080] Step 8: Train the model using a loss function to obtain the trained MSTCNet model, specifically as follows: Use the cross-entropy loss function and optimize it using the Adam optimizer. In this embodiment, the initial learning rate is set to 0.0002, the batch size is set to 12, and L2 norm constraints are applied to the weights of the fully connected layers during training. Proceed to Step 9.

[0081] Step 9: Input the multi-view EEG data of the test set into the trained MSTCNet model, and obtain the predicted classification results of all EEG samples through forward propagation of the model.

[0082] This invention uses accuracy, standard deviation (SD), and Kappa coefficient as evaluation metrics, and the calculation methods are as follows:

[0083]

[0084] In this context, TP represents true positives, TN represents true negatives, FP represents false positives, and FN represents false negatives. In multi-class classification tasks, accuracy is defined as the ratio of correctly classified samples to the total number of samples.

[0085]

[0086] Where N represents the sample size (e.g., the number of subjects), x i Let μ represent the accuracy of the i-th sample, and μ represent the average accuracy of all samples.

[0087]

[0088] Where, p o For observed consistency (i.e. accuracy), p e This represents the expected coincidental consistency. e The calculation method is as follows:

[0089] Where M is the total number of categories, a k Let b be the number of true samples in the k-th class. k T represents the number of samples predicted to be in the k-th class, where T is the total number of samples.

[0090] The model of this invention was tested on a motor imagery EEG decoding task. This invention was validated on three publicly available benchmark datasets: BCI Competition IV-2a (BCIC-IV-2a) and BCI Competition IV-2b (BCIC-IV-2b), and the results are shown in Tables 1 and 2 below.

[0091] Table 1. Experimental Results of BCI Competition IV-2a

[0092] Model Accuracy (%) SD Kappa EEGNet 74.61 13.68 0.6610 FBCNet 76.62 12.45 0.6883 EEG Conformer 78.66 15.30 0.7155 MI-DAGSC 79.63 12.27 0.7280 Our Method 84.34 9.28 0.7912

[0093] Table 2. Experimental Results of BCI Competition IV-2b

[0094] Model Accuracy (%) SD Kappa EEGNet 76.12 12.21 0.6820 FBCNet 81.94 14.59 0.6389 EEG Conformer 84.63 12.18 0.6926 MI-DAGSC 85.76 12.62 0.7100 Our Method 88.33 10.34 0.7666

[0095] Three evaluation metrics were calculated: accuracy, standard deviation, and Kappa coefficient. Higher accuracy and Kappa coefficient values ​​indicate better model classification performance and higher consistency, while lower standard deviation values ​​indicate stronger model stability. Tables 1 and 2 show that the MSTCNet model proposed in this invention achieves competitive results compared to existing technologies, significantly improving the accuracy of decoding motor imagery EEG signals while maintaining high stability.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms, characterized in that, Includes the following steps: S1. Acquire raw EEG signals, and use a Chebyshev Type II filter bank to decompose the signals into multiple sub-signals of different frequency bands to construct multi-view EEG data X. f The dataset was then divided into a training set, a validation set, and a test set. S2, the multi-view EEG data X in the training set f Data augmentation is performed by generating enhanced multi-view data X using a frequency band sensing segment stitching method. aug ; S3. Construct the MSTCNet model, which includes a multi-view temporal convolution (MTC) module, a spatial convolution and attention (SCA) module, a temporal mean-log variance (TML) module, and a classification module. S4. Transfer the enhanced multi-view data X aug The input MTC module extracts time-series dynamic features from sub-signals in different frequency bands and performs multi-view feature fusion to obtain the time-series fused feature Z. MTC ; S5. The time-series fusion feature Z MTC The input SCA module extracts spatial features using depthwise separable convolution, and adaptively weights the feature channels using a channel attention mechanism to obtain the spatially enhanced feature Z. SCA ; S6. The spatial enhancement feature Z SCA Input the TML module to calculate the mean and logarithmic variance within the time window, and concatenate them to obtain the joint statistical feature Z. TML ; S7. The joint statistical feature Z TML The input classification module calculates the class probability distribution through a fully connected layer and a Softmax function to obtain the predicted probability vector; S8. Calculate the cross-entropy loss based on the predicted probability vector and the true label, and train the MSTCNet model to obtain the trained MSTCNet model. S9. Transfer the multi-view EEG data X from the test set. f The trained MSTCNet model is input, and the final motor imagery classification result of the EEG signal is output through forward propagation of the model.

2. The method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms according to claim 1, characterized in that: In step 2, the multi-view EEG data X in the training set... f Data augmentation will be performed as follows: For each frequency band b, it is uniformly divided into n parts on the time axis. seg Each segment, let's say This represents the j-th sub-segment after segmentation; Randomly select n from the training dataset seg Data from a sample, and extracting sub-segments at the corresponding time positions for each sample. Where k j The index of the randomly selected sample; These randomly sampled sub-segments are spliced ​​together according to the original time sequence to generate enhanced samples for that frequency band. It can be represented as: The above operations are performed in parallel on all frequency bands, and the enhanced samples from each frequency band are stitched together to obtain the enhanced multi-view data X. aug .

3. The method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms according to claim 1, characterized in that: In step 4, temporal dynamic features are extracted for sub-signals of different frequency bands, and the temporal convolution kernel size L is used. i The calculation formula is: Among them, F s Indicates the sampling rate. Let L represent the center frequency of the i-th frequency band, and k be the periodicity coverage coefficient; for low-frequency rhythms, k is set to 1.5; for high-frequency rhythms, k is set to 0.8; and L i Always choose odd numbers; Subsequently, 1×1 convolutions are used to expand the channel dimension, and the outputs of all views are concatenated along the channel dimension. After batch normalization, the temporal fusion feature Z is obtained. MTC .

4. The method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms according to claim 1, characterized in that: In step 5, spatial features are extracted using depthwise separable convolution, and the feature channels are adaptively weighted using a channel attention mechanism, as follows: First, Z MTC Spatial filtering is performed by inputting a deep convolutional layer, and then cross-channel integration is performed by inputting a pointwise convolutional layer to obtain the spatial feature map X1. Next, global spatial information of X1 is compressed using global average pooling, and channel importance weights α are generated through two fully connected layers. These weights are then used to recalibrate X1 to obtain X. SE ; At the same time, residual paths are introduced to Z MTC X is obtained by adjusting the dimensions through convolution. res X SE With X res The addition yields the spatially enhanced feature Z. SCA Z SCA =X main +X res , where X main For X SE The output after being processed by Dropout.

5. The method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms according to claim 1, characterized in that: In step 6, the mean and logarithmic variance within the time window are calculated as follows: The features are divided into n non-overlapping time windows along the time dimension, with a window size of ω; Calculate the mean feature μ of the i-th window on the c-th channel. c,i : Calculate the log-variance feature of the i-th window on the c-th channel. The mean feature and the log-variance feature are concatenated to form the joint statistical feature Z. TML .

6. A method for decoding motor imagery EEG based on multi-view spatiotemporal convolution and attention mechanisms according to claim 1 or 3, characterized in that: In step 1, the filter bank employs a frequency band division strategy tailored to the characteristics of motor imagery EEG rhythms, specifically configured as follows: The frequency range of 4-40Hz is divided into 9 non-overlapping sub-bands, each with a bandwidth of 4Hz; The frequency band division strategy is the time convolution kernel size L of the MTC module in step 4. i The calculation provides the center frequency parameter.

Images