Motor imagery electroencephalogram decoding method based on multi-attention bridge structure network

Abstract

The invention discloses a motor imagery electroencephalogram decoding method based on a multi-attention bridge structure network, and relates to the field of biological signal processing and motor imagery brain-computer interfaces. The method comprises three modules: 1) a depth separable convolution attention module; 2) bridge structure modules; and 3) a multi-scale parallel space-time convolution module. In order to solve the problems of poor robustness and weak generalization ability in a motor imagery electroencephalogram signal decoding process, a deep separable convolution attention module is utilized to focus important channel space topological characteristics of electroencephalogram signals, and a bridge structure module is utilized to effectively capture local and global time domain characteristics; and multi-dimensional attention on spatial-temporal characteristics of the electroencephalogram signals is realized. And finally, spatial-temporal features of different scales are extracted through a multi-scale parallel spatial-temporal convolution module, so that the decoding recognition rate, robustness and generalization ability of the network are improved, and an effective solution is provided for improving the stability and applicability of a motor imagery brain-computer interface.

Description

Technical Field

[0001] The present invention relates to the field of biological signal recognition and brain-computer interface technology, and in particular to a motor imagery electroencephalogram decoding method based on a multi-attention bridge structure network. Background Art

[0002] Brain-Computer Interface (BCI) is an emerging system that promotes human-computer interaction and control by analyzing EEG signals. In recent years, more and more researchers have devoted themselves to the development of BCI systems, especially the research of spontaneous BCI systems, which have shown significant advantages in the fields of rehabilitation assistance and robot control. The BCI system based on motor imagery (MI) EEG decoding is an important part of the research on spontaneous BCI systems and is gradually becoming the focus of current BCI system research.

[0003] Motor imagery EEG signals are weak biological signals that are easily affected by non-EEG signals such as electrocardiogram, electromyography, and equipment noise, resulting in low signal-to-noise ratio and poor stability. In addition, EEG signals vary significantly between individuals, and these differences are affected by multiple factors such as genetics, age, and health status, making it extremely challenging to build a universal recognition model. The above problems not only affect the recognition rate of decoding motor imagery EEG signals, but also limit the practical application of BCI systems. Therefore, it is urgent to develop and apply new methods and strategies to improve the robustness and generalization ability of EEG signal decoding networks, thereby improving the decoding recognition rate of EEG signals and promoting further development in the field of BCI.

[0004] Traditional machine learning methods perform well when dealing with linear features, but are not effective when dealing with complex nonlinear motor imagery EEG signals. In addition, manual feature selection and extraction have high requirements for operators, making the analysis process more complicated. Deep learning can automatically extract features and classify them without manual intervention. Deep learning models also show strong nonlinear modeling capabilities when dealing with complex data and perform well in motor imagery EEG signal decoding. However, existing deep learning models have not yet achieved ideal results in terms of generalization ability. With the advent of GPT, the Transformer structure has attracted widespread attention due to its excellent global attention ability and the advantages of improving model generalization performance. Combining the Transformer structure with the EEG signal decoding model has become an important idea to improve the generalization ability of the decoding model. At the same time, in the current motor imagery EEG signal decoding task, the use of attention mechanism for signal analysis is increasingly valued. However, current research has not fully explored the multi-dimensional attention to local and global time domain features and channel space topology information. The spatiotemporal feature information of EEG signals has not been fully utilized, which limits the accuracy and effectiveness of decoding. Therefore, exploring how to effectively decode from the perspective of multi-dimensional features will help improve the effectiveness of the motor imagery EEG signal decoding framework. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a motor imagery EEG decoding method based on a multi-attention bridge structure network, which aims to improve the robustness and generalization ability of the motor imagery EEG signal decoding model and provide an effective solution for the stability and applicability of the brain-computer interface.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0007] A motor imagery EEG decoding system based on a multi-attention bridge structure network, comprising:

[0008] A deep separable convolutional attention module is used to extract multi-dimensional feature information from raw EEG signals and perform weighted processing on it;

[0009] The bridge structure module is used to integrate the weighted local features and global features;

[0010] The multi-scale parallel spatiotemporal convolution module is used to perform multi-scale analysis on the integrated feature information to obtain the final EEG signal classification result.

[0011] The depthwise separable convolutional attention module includes a temporal convolutional layer, a depthwise separable convolutional layer, a spatial convolutional layer and an efficient channel attention structure connected sequentially, wherein the output of the temporal convolutional layer is processed by batch normalization, and the outputs of the depthwise separable convolutional layer and the spatial convolutional layer are processed by batch normalization, activation function and average pooling layer.

[0012] The calculation process of the channel attention convolution kernel in the efficient channel attention structure is:

[0013] log value =log2(channel)

[0014]

[0015] k=int(k raw |)

[0016]

[0017] Among them, channel is the number of input channels, b and γ are specified constants, k is the final calculated convolution kernel size, k is an odd number, int is the rounding function, log value and k raw All are process variables.

[0018] The bridge structure module is a bridge structure composed of a multi-head self-attention sub-module and a bidirectional long short-term memory network, and the number is greater than or equal to 2, wherein the multi-head self-attention sub-module is composed of a convolutional neural network and a multi-head self-attention structure.

[0019] The multi-scale parallel spatiotemporal convolution module is composed of three parallel spatiotemporal convolution modules with different convolution kernel sizes, whose convolution kernel sizes are (1, 4), (1, 8) and (1, 12) respectively, and the output results of the three spatiotemporal convolution modules are averaged to generate a vector, wherein the spatiotemporal convolution module is a double-layer residual network serial dilated convolution structure.

[0020] A motor imagery EEG decoding method based on a multi-attention bridge structure network comprises the following steps:

[0021] The deep separable convolutional attention module extracts multi-dimensional feature information from the original EEG signal and performs weighted processing on it;

[0022] The bridge structure module integrates weighted local and global features;

[0023] The multi-scale parallel spatiotemporal convolution module performs multiple scale analyses on the integrated feature information to obtain the final EEG signal classification results.

[0024] The bridge structure module performs the following steps:

[0025] In the first bridge structure module, the feature information output by the depthwise separable convolutional attention module is used as data 1 and input into the multi-head self-attention sub-module to output data 2. Data 2 and data 1 are concatenated and used together as the input of the bidirectional long short-term memory network. Data 3 is output and data 3 and data 2 are concatenated to generate data 4. Data 3 and data 4 are used as the input of the next bridge structure module, where data 4 is input into the multi-head self-attention sub-module, and its output is concatenated with data 3 and input into the bidirectional long short-term memory network. The two outputs of the last bridge structure module are concatenated and used as the input of the multi-scale parallel spatiotemporal convolution module.

[0026] The multi-scale parallel spatiotemporal convolution module performs the following steps:

[0027] The feature vector output by the bridge structure module passes through three spatiotemporal convolution modules with different convolution kernel sizes respectively. The outputs of the three spatiotemporal convolution modules are averaged to generate an output vector, which is input into the fully connected layer and uses softmax as the activation function for decoding and classification.

[0028] The present invention has the following beneficial effects and advantages:

[0029] 1. The present invention combines the deep separable convolution method and the efficient channel attention structure to convolve the channel space topology information of the EEG signal and perform weighted processing on important channels, thereby increasing the attention paid to important features in the spatial domain; using convolutional neural networks and multi-head self-attention to form a multi-head self-attention submodule, and a bidirectional long short-term memory network to build a bridge structure to process feature information, comprehensively focusing on time domain features from both local and global perspectives, and effectively integrating local and global features of the EEG signal time domain. Through the processing methods of multi-attention and bridge structure, the robustness of the model and the decoding recognition rate are improved.

[0030] 2. The present invention uses a multi-head self-attention sub-module to build a bridge structure with Transformer-like characteristics, performs independent nonlinear transformation on the feature expression of each position, and improves the generalization ability of the model.

[0031] 3. The present invention adopts a multi-scale parallel spatiotemporal convolution module to analyze the spatiotemporal characteristics of signals from multiple scales, and alleviates the problem of vanishing gradients of the model through a residual network structure, further improving the robustness of the network and providing an effective solution for the stability and applicability of the brain-computer interface system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the method of the present invention;

[0033] Figure 2 It is a schematic diagram of the efficient channel attention (ECA) structure;

[0034] Figure 3 It is a schematic diagram of the structure of the multi-head self-attention submodule;

[0035] FIG. 4 is a graph of experimental classification performance of the present invention. DETAILED DESCRIPTION

[0036] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0037] A motor imagery EEG decoding method based on a multi-attention bridge structure network, including the following modules:

[0038] 1) Deep Separable Convolutional Attention Module: It uses temporal and spatial convolutional layers to extract multi-dimensional feature information from the original EEG signals, and introduces the efficient channel attention (ECA) structure to perform weighted attention on the spatial topological features of important channels in the EEG signals, thereby increasing the attention to important features in the spatial domain;

[0039] 2) Bridge structure module: The multi-head self-attention sub-module is combined with the neural network and multi-head self-attention, and the feature information is processed in a bridge structure with the bidirectional long short-term memory network. The bidirectional long short-term memory network is combined with the ability to capture the contextual time domain information and the multi-head self-attention pays attention to the global time domain features, which effectively integrates the local and global features of the EEG signal time domain information. The bridge structure is designed based on the design idea of ​​Transformer, using the next multi-head self-attention sub-module or bidirectional long short-term memory network as the feedforward neural network in the structure to perform independent nonlinear transformations on the feature expression of each position, thereby improving the generalization ability of the model;

[0040] 3) Multi-scale parallel spatiotemporal convolution module: Spatiotemporal convolution modules with different receptive fields are used to analyze the spatiotemporal characteristics of signals from multiple scales. The spatiotemporal convolution module adopts a double-layer residual network serial dilated convolution structure to alleviate the problem of vanishing gradients of the model, enhance the model's adaptability to input data, and thus improve the robustness of the network.

[0041] Figure 1 The flowchart of the method of the present invention is shown. The depth-separable convolutional attention module adopts a three-layer convolution structure, namely, a temporal convolution layer, a depth-separable convolution layer and a spatial convolution layer, and the convolution kernel sizes are (64, 1), (1, 22) and (16, 1) respectively. Batch normalization is used after the first convolution layer, and batch normalization, activation function (exponential linear unit) and average pooling layer are used after the second and third convolution layers. The parameters poolsize of the two average pooling layers are (8, 1) and (7, 1) respectively.

[0042] The efficient channel attention (ECA) structure is used to focus on the channel characteristics of EEG signals. Figure 2 As shown in the figure, important channels are paid attention to in a weighted manner to improve the network's ability to extract important spatial features. The calculation process of the channel attention convolution kernel is as follows:

[0043] log value =log2(channel)

[0044]

[0045] k=int(k raw |)

[0046]

[0047] Where channel is the number of input channels, b and γ are specified constants, 1 and 2 respectively, and k is the final calculated convolution kernel size. In order to keep the center of the convolution operation aligned, k must be an odd number, and the final value of k is 3. One-dimensional convolution calculates channel attention:

[0048] s=σ(W*z+b)

[0049] Where W is the convolution kernel, size is (1, k), b is the bias term, and σ is the activation function. Finally, the attention weight is assigned to the channel vector:

[0050]

[0051] is the feature vector recalibrated by attention.

[0052] The multi-head self-attention submodule is composed of a convolutional neural network and a multi-head self-attention structure. The bridge structure is composed of two parts: a multi-head self-attention submodule and a bidirectional long short-term memory network. The two parts are composed as follows: Figure 1 The bridge structure is a Transformer-like structure. The bidirectional long short-term memory network and the next multi-head self-attention submodule serve as feedforward neural networks in the Transformer structure, performing independent nonlinear transformations on the feature expression of each position, thereby improving the generalization ability of the model.

[0053] The order in which data enters the bridge structure is as follows: after data (1) enters the multi-head self-attention submodule, it outputs data (2), data (2) and data (1) are concatenated and enter the bidirectional long short-term memory network, outputting data (3), data (3) and data (2) are concatenated to form data (4), and data (3) and data (4) enter the next bridge structure as new data input. In the network structure, the number of bridge structures is greater than or equal to 2.

[0054] The multi-head attention submodule consists of a convolutional neural network and a multi-head attention. The convolutional neural network can effectively control the model scale, and its convolution kernel size is 64. The multi-head attention structure consists of three components: query Q, key K and value V, such as Figure 3 As shown. Regarding the choice of the number of attention heads, we set N = Numheads = 8, Dimhead = 8. The query is processed by the dot product with the key, integrated in a multiplicative attention manner, and then passed through the softmax layer to generate a representative score for each value. The calculation process of the representative score matrix of all channels is as follows:

[0055]

[0056] in represents the parameters related to layer normalization, λ(·) represents the convolution change performed by the convolutional neural network, and in the typical score matrix Atten(q t ,k t ), each value is refined by aggregating relevant information to enhance the expressiveness of the matrix. Multiplicative attention is used to improve the effectiveness of the attention mechanism. The output of the dot product attention is then processed through a linear transformation and the result is added to the vector through a residual connection. Its process definition is as follows:

[0057]

[0058] The multi-scale parallel spatiotemporal convolution module uses spatiotemporal convolution modules with different receptive fields to analyze multi-scale features. It consists of three parallel spatiotemporal convolution modules with the following structure: Figure 1 As shown in the figure, the scale of each spatiotemporal convolution module is determined by different convolution kernels. The sizes of different convolution kernels corresponding to spatiotemporal convolution modules 1, 2, and 3 are (1, 4), (1, 8), and (1, 12), respectively. Each spatiotemporal convolution module adopts a double-layer residual network serial dilated convolution structure. Finally, the outputs of the three parallel spatiotemporal convolution modules are averaged to generate an output vector, which is input into the fully connected layer (FC) layer and softmax is used as the activation function for classification tasks.

[0059] The performance of the network of the present invention is tested using a four-category motor imagery dataset dataset 2a provided by the 4th International BCI Competition. The dataset includes motor imagery EEG signal data of 9 subjects, including four types of tasks: left hand, right hand, feet and tongue. A total of 288 experiments were conducted, with 72 experiments for each type of task. Figure 4a The figure shows the recognition rate confusion matrix of the network framework decoding of the present invention, and the average decoding recognition rate is 84.10%. Figure 4b Shown is the decoding recognition rate confusion matrix of the present invention when the subject is irrelevant, and the obtained decoding recognition rate is 68.04%. The results fully demonstrate that the motor imagery EEG decoding method based on the multi-attention bridge structure network proposed in the present invention improves the decoding recognition rate, robustness and generalization ability of the network, and provides an effective solution for the stability and applicability of the brain-computer interface.

Claims

1. A motor imagery EEG decoding system based on a multi-attention bridge structure network, characterized in that: include: A deep separable convolutional attention module is used to extract multi-dimensional feature information from raw EEG signals and perform weighted processing on it; The bridge structure module is used to integrate the weighted local features and global features; The multi-scale parallel spatiotemporal convolution module is used to perform multi-scale analysis on the integrated feature information to obtain the final EEG signal classification result.

2. According to claim 1, a motor imagery EEG decoding system based on a multi-attention bridge structure network is characterized in that: The depthwise separable convolutional attention module includes a temporal convolutional layer, a depthwise separable convolutional layer, a spatial convolutional layer and an efficient channel attention structure connected sequentially, wherein the output of the temporal convolutional layer is processed by batch normalization, and the outputs of the depthwise separable convolutional layer and the spatial convolutional layer are processed by batch normalization, activation function and average pooling layer.

3. The motor imagery EEG decoding system based on a multi-attention bridge structure network according to claim 2, characterized in that: The calculation process of the channel attention convolution kernel in the efficient channel attention structure is: log value =log2(channel) k=int(k raw |) Among them, channel is the number of input channels, b and γ are specified constants, k is the final calculated convolution kernel size, k is an odd number, int is the rounding function, log value and k raw All are process variables.

4. The motor imagery EEG decoding system based on a multi-attention bridge structure network according to claim 1, characterized in that: The bridge structure module is a bridge structure composed of a multi-head self-attention sub-module and a bidirectional long short-term memory network, and the number is greater than or equal to 2, wherein the multi-head self-attention sub-module is composed of a convolutional neural network and a multi-head self-attention structure.

5. The motor imagery EEG decoding system based on a multi-attention bridge structure network according to claim 1, characterized in that: The multi-scale parallel spatiotemporal convolution module is composed of three parallel spatiotemporal convolution modules with different convolution kernel sizes, whose convolution kernel sizes are (1, 4), (1, 8) and (1, 12) respectively, and the output results of the three spatiotemporal convolution modules are averaged to generate a vector, wherein the spatiotemporal convolution module is a double-layer residual network serial dilated convolution structure.

6. A motor imagery EEG decoding method based on a multi-attention bridge structure network, characterized in that: The following steps are involved: The deep separable convolutional attention module extracts multi-dimensional feature information from the original EEG signal and performs weighted processing on it; The bridge structure module integrates weighted local and global features; The multi-scale parallel spatiotemporal convolution module performs multiple scale analyses on the integrated feature information to obtain the final EEG signal classification results.

7. The method for motor imagery EEG decoding based on a multi-attention bridge structure network according to claim 6, characterized in that: The bridge structure module performs the following steps: In the first bridge structure module, the feature information output by the depthwise separable convolutional attention module is used as data 1 and input into the multi-head self-attention sub-module to output data 2. Data 2 and data 1 are concatenated and used together as the input of the bidirectional long short-term memory network. Data 3 is output and data 3 and data 2 are concatenated to generate data 4. Data 3 and data 4 are used as the input of the next bridge structure module, where data 4 is input into the multi-head self-attention sub-module, and its output is concatenated with data 3 and input into the bidirectional long short-term memory network. The two outputs of the last bridge structure module are concatenated and used as the input of the multi-scale parallel spatiotemporal convolution module.

8. The method for motor imagery EEG decoding based on a multi-attention bridge structure network according to claim 6, characterized in that: The multi-scale parallel spatiotemporal convolution module performs the following steps: The feature vector output by the bridge structure module passes through three spatiotemporal convolution modules with different convolution kernel sizes respectively. The outputs of the three spatiotemporal convolution modules are averaged to generate an output vector, which is input into the fully connected layer and uses softmax as the activation function for decoding and classification.

Images