A spatiotemporal convolutional EEG decoding method based on multi-head self-attention mechanism fusion

Abstract

The present invention discloses a spatiotemporal convolutional EEG decoding method incorporating a multi-head self-attention mechanism, comprising: acquiring EEG signals; inputting the EEG signals into an EEG decoding network, whereby a first spatiotemporal feature extraction module and a second spatiotemporal feature extraction module in the EEG decoding network sequentially process the EEG signals; the first and second spatiotemporal feature extraction modules extract spectral features of the input data, extract temporal features based on the spectral features, and then utilize a multi-head self-attention mechanism to extract spatial-temporal features; and classify the spatial-temporal features output by the second spatiotemporal feature extraction module to obtain EEG signal classification results. The present invention accurately characterizes EEG signals. Separable convolutional layers effectively decouple the temporal correlation between EEG tasks and EEG signals. Furthermore, a multi-head self-attention mechanism is utilized to focus on the spatial activation patterns of the brain, extracting complementary spatial representation information, which facilitates accurate EEG signal classification.

Description

Technical Field

[0001] The present invention belongs to the field of neural information decoding technology, and specifically relates to a spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism. Background Art

[0002] BCI (brain-computer interface) is a key technology for enabling information exchange between the human brain and machines. It utilizes EEG (electroencephalogram) signals to open up a new channel for communication and control that doesn't rely on peripheral nerves or muscles. EEG signals can be decoded into different commands and input into machines for corresponding operations. Due to their practicality, enormous potential, and value, non-invasive BCIs have garnered widespread attention.

[0003] Traditionally, brain-computer interfaces (BCIs) can be divided into two types based on the signal source: induced BCIs and spontaneous BCIs. Both types have excellent performance in BCI applications. Induced BCIs use external stimuli to evoke EEG signals, which do not require much training but require a specific environment, such as rapid serial visual presentation. Spontaneous BCIs use EEG signals that are voluntarily controlled by the subject as input, such as MI (Motor Imagery) and IS (Imagined-Speech).

[0004] The EEG decoding process typically consists of five stages: acquiring EEG data from different paradigms, preprocessing, feature extraction, classifier training, and feedback. Most machine learning-based BCI approaches follow this process, providing many effective solutions based on handcrafted features. However, despite the importance of handcrafted features across different EEG paradigms, they still require specific modifications based on prior knowledge.

[0005] With the development of deep learning methods, the need for handcrafted features in EEG decoding has been greatly reduced. Deep learning-based BCI methods can maintain the structure and configuration information of the original data, combining the steps of feature extraction and classification learning for joint optimization, thereby improving model performance. The HS-STDCN proposed by Li et al. integrates feature learning of spatiotemporal information into a unified end-to-end model, achieving an accuracy of 54.31% in imagined speech. Furthermore, the EEGNet proposed by Lawhern et al. uses an end-to-end approach to process EEG signals and has been applied to four BCI paradigms: P300 visual evoked potential, error-related negativity, movement-related cortical potentials, and sensorimotor rhythms.

[0006] The EEG signal decoding method based on deep learning has made some progress, but there are still many difficulties: (1) The EEG signal itself is relatively weak, the electrode distribution is irregular, and the characteristics are not obvious, which makes it difficult to model the fine features of the signal; (2) Without changing the network structure, it is difficult to process a variety of different types of EEG signals, and the performance is monotonous. Summary of the Invention

[0007] In order to solve the above problems existing in the prior art, the present invention provides a spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0008] The present invention provides a spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism, comprising:

[0009] Collect EEG signals;

[0010] Inputting the EEG signal into an EEG decoding network, so that a first spatiotemporal feature extraction module and a second spatiotemporal feature extraction module in the EEG decoding network sequentially process the EEG signal, wherein the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module are used to extract spectral features of the input data at different bandpass frequencies, and after extracting temporal features based on the spectral features, use a multi-head self-attention mechanism to extract spatial-temporal features;

[0011] The space-time features output by the second space-time feature extraction module are classified to obtain a classification result of the EEG signal.

[0012] In one embodiment of the present invention, the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module both include a time feature extraction module and a space feature extraction module;

[0013] After the EEG signal is input into the EEG decoding network, the first spatiotemporal feature extraction module processes it according to the following steps:

[0014] The time feature extraction module in the first spatiotemporal feature extraction module uses a first convolution layer to extract spectral features of the EEG signal at different bandpass frequencies, and uses a separable convolution layer to perform depth convolution and point convolution on the spectral features to obtain a first time feature;

[0015] The spatial feature extraction module in the first spatiotemporal feature extraction module extracts the first spatial-temporal feature based on the first temporal feature X using a multi-head self-attention mechanism.

[0016] In one embodiment of the present invention, the convolution kernel size of the first convolutional layer is (fs / 2, 1), where fs is the sampling frequency of the EEG signal.

[0017] In one embodiment of the present invention, the step of extracting the first space-time feature by the spatial feature extraction module in the first space-time feature extraction module using a multi-head self-attention mechanism based on the first time feature X includes:

[0018] The spatial feature extraction modules in the first spatiotemporal feature extraction module use their respective attention heads to perform linear transformation on the first temporal feature X to project it into different low-dimensional feature subspaces, obtaining vectors Q, K, and V:

[0019] (Q,K,V)=Matmul(X,(W Q ,W K ,W V ));

[0020] Where, Matmul represents matrix multiplication, W Q 、W K and W V represents a preset weight matrix, the dimension of the low-dimensional feature subspace is lower than the dimension of the first time feature X;

[0021] Calculate the output matrix of each self-attention head based on vector Q, vector K, and vector V:

[0022]

[0023] Where, d K represents the dimension of vector K;

[0024] Concatenate the output matrices of all self-attention heads to get the result matrix:

[0025] X multi-head =Concat(Z1,Z2,...,Z H );

[0026] Where H represents the number of self-attention heads;

[0027] For the result matrix X multi-head Perform a residual connection with the first temporal feature X and normalize it using BatchNorm:

[0028] X′=BatchNorm(X+X multi-head );

[0029] Perform feature mapping on the normalized feature map X′ to obtain a mapping feature map;

[0030] The mapping feature map and the normalized feature map X′ are residually connected and normalized using BatchNorm to obtain a first space-time feature.

[0031] In one embodiment of the present invention, the step of performing feature mapping on the normalized feature map X′ to obtain a mapped feature map includes:

[0032] After performing a convolution operation on the normalized feature map X′ using a second convolution layer with a convolution kernel size of 1*1, normalization and nonlinear activation are performed, and a convolution operation is performed again using a third convolution layer with a convolution kernel size of 1*1 to obtain a mapping feature map.

[0033] In a second aspect, the present invention provides an electronic device comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0034] Memory for storing computer programs;

[0035] The processor is configured to implement the method steps described in the first aspect when executing the program stored in the memory.

[0036] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in the first aspect are implemented.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention provides a spatiotemporal convolutional EEG decoding method and device fused with a multi-head self-attention mechanism. After the EEG signal is input into the EEG decoding network, the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module process the EEG signal in sequence, that is, the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module are used to extract the spectral features of the input data at different bandpass frequencies, and after extracting the time features based on the spectral features, the multi-head self-attention mechanism is used to extract the space-time features, so that the EEG signal can be accurately characterized.

[0039] In addition, the separable convolutional layer with multi-time scale factors can effectively decouple the temporal correlation between EEG tasks and EEG signals. On this basis, the multi-head self-attention mechanism is used to focus on the spatial activation patterns of the brain and extract complementary spatial representation information from multiple subspaces, which is conducive to the accurate classification of EEG signals.

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism provided by an embodiment of the present invention;

[0042] Figure 2This is a schematic diagram of a spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism provided by an embodiment of the present invention;

[0043] Figure 3 This is another schematic diagram of the spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism provided by an embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of a spatial feature extraction module provided by an embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of collecting EEG signals provided by an embodiment of the present invention;

[0046] Figure 6a This is a visualization diagram of the RSVP spatial-temporal features of an EEG task provided by an embodiment of the present invention;

[0047] Figure 6b Schematic diagram of the RSVP confusion matrix provided by an embodiment of the present invention;

[0048] Figure 6c This is a visualization diagram of the MI space-time characteristics of the EEG task provided by an embodiment of the present invention;

[0049] Figure 6d 1 is a schematic diagram of the MI confusion matrix provided by an embodiment of the present invention;

[0050] Figure 6e This is a visualization diagram of the IS spatial-temporal characteristics of the EEG task provided by an embodiment of the present invention;

[0051] Figure 6f Schematic diagram of the IS confusion matrix provided by an embodiment of the present invention;

[0052] Figure 7 This is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0054] Figure 1 This is a flow chart of a spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of a spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism provided by an embodiment of the present invention. Figure 1-2 As shown, an embodiment of the present invention provides a spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism, including:

[0055] S1, collect EEG signals;

[0056] S2. Inputting the EEG signal into the EEG decoding network, so that the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module in the EEG decoding network process the EEG signal in sequence, wherein the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module are used to extract spectral features of the input data at different bandpass frequencies, and after extracting temporal features based on the spectral features, use a multi-head self-attention mechanism to extract spatial-temporal features;

[0057] S3. Classify the space-time features output by the second space-time feature extraction module to obtain a classification result of the EEG signal.

[0058] It should be understood that EEG signals, as a direct reflection of the brain's subjective experience and feelings, can intuitively reflect the neurophysiological activity of different BCI task intentions. At the same time, subtle changes in various BCI tasks can be reflected in the temporal dimension of EEG signals and in varying degrees of spatial activation patterns. However, EEG signal features are weak, spatial activation patterns are similar, and time sensitivity is high. Therefore, their classification results are highly dependent on the ability to extract features in both spatial and temporal dimensions. Figure 3 This is another schematic diagram of the spatiotemporal convolutional EEG decoding method fused with the multi-head self-attention mechanism provided by the embodiment of the present invention. Figure 3 As shown, taking this into consideration, the present invention divides the decoding process of EEG signals into two parts: time feature extraction and spatial feature extraction, and then extracts feature information of high-precision recognition task categories from the original EEG signals to improve the feature quality of EEG signals.

[0059] Alternatively, see Figure 2-3 , the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module both include a time feature extraction module and a space feature extraction module;

[0060] After the EEG signal is input into the EEG decoding network, the first spatiotemporal feature extraction module processes it according to the following steps:

[0061] The time feature extraction module in the first spatiotemporal feature extraction module uses the first convolution layer to extract the spectrum features of the EEG signal at different bandpass frequencies, and uses the separable convolution layer to perform depth convolution and point convolution on the spectrum features to obtain the first time feature X;

[0062] The spatial feature extraction module in the first spatiotemporal feature extraction module extracts the first spatial-temporal feature based on the first temporal feature X using a multi-head self-attention mechanism.

[0063] Exemplarily, the convolution kernel size of the first convolutional layer is (fs / 2, 1), where fs is the sampling frequency of the EEG signal.

[0064] Specifically, for the input EEG signal, a convolution layer with a convolution kernel size of (fs / 2, 1) is first used to extract the spectral features of the EEG signal at different bandpass frequencies. This convolution kernel size setting is based on the Nyquist-Shannon sampling theorem, which allows the temporal feature extraction module to obtain spectral information above 2Hz. Based on the above spectral features, separable convolution layers with convolution kernels of different sizes are used to further explore temporal features with classification capabilities. Using separable convolution layers to extract temporal features from EEG signals not only greatly reduces the network's adjustable weight matrix, but also enables the temporal feature extraction module to separately learn the temporal kernel from the feature map (depth convolution), and then optimally integrate the feature map (point convolution), effectively decoupling information in the temporal dimension.

[0065] Compared with traditional convolution, separable convolution layer has fewer parameters and lower computational complexity, which is suitable for small sample data such as EEG signals. For example, for the input matrix X of the same dimension and the dimension (c in ,h,w), the output matrix Y, dimension is (c out ,h,w), calculation of convolution kernel size (1,k), traditional convolution layer has k*c out *c in parameters, k*c out *h*w*c in operations, while the separable convolutional layer only has (k+c out )*c in parameters, 2*(k+c out )*h*w*c in operations. It can be seen that the computational complexity of the separable convolution layer is much lower than that of the traditional convolution layer. In the subsequent temporal correlation information mining, the convolution kernel size of the separable convolution layer is (T j ,1),T j Represents the kernel size of the j-th convolutional layer. Convolution kernels of different scale factors can process features of different time points and frequency ranges, that is, larger convolution kernels can learn longer time features and represent spectral features of a lower range; and vice versa. In addition, this embodiment also adds a batch standard layer and an activation function after the separable convolution layer. The former provides regularization for the features and reduces the risk of model overfitting, and the latter provides nonlinear transformation of the features and improves the learning and expression ability of the model. The deep separable convolution layer with multiple scale factors can effectively decouple the EEG correlation information of different EEG tasks in the time dimension and improve the ability to represent time features.

[0066] It should be noted that, since the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module in this embodiment have the same structure, both include a time feature extraction module and a spatial feature extraction module, and their processing procedures for input data are also the same, this embodiment only describes the processing procedure of the first spatiotemporal feature extraction module in detail.

[0067] Figure 4 is a schematic diagram of the spatial feature extraction module provided by an embodiment of the present invention. Figure 3 As shown, the spatial feature extraction module in the first spatiotemporal feature extraction module extracts the first spatial-temporal feature based on the first temporal feature X using a multi-head self-attention mechanism, including:

[0068] The spatial feature extraction modules in the first spatiotemporal feature extraction module use their respective attention heads to perform linear transformation on the first temporal feature X to project it into different low-dimensional feature subspaces, obtaining vectors Q, K, and V:

[0069] (Q,K,V)=Matmul(X,(W Q ,W K ,W V ));

[0070] Where, Matmul represents matrix multiplication, W Q 、W K and W V Represents the preset weight matrix, the dimension of the low-dimensional feature subspace is lower than the dimension of the first time feature X;

[0071] Calculate the output matrix of each self-attention head based on vector Q, vector K and vector V;

[0072] Concatenate the output matrices of all self-attention heads to get the result matrix:

[0073] X multi-head =Concat(Z1,Z2,...,Z H );

[0074] Where H represents the number of self-attention heads;

[0075] For the result matrix X multi-head Perform residual connection with the first time feature X and normalize using BatchNorm:

[0076] X′=BatchNorm(X+X multi-head );

[0077] Perform feature mapping on the normalized feature map X′ to obtain a mapping feature map;

[0078] The mapped feature map is residually connected to the normalized feature map X′ and normalized using BatchNorm to obtain the first spatial-temporal feature.

[0079] The step of performing feature mapping on the normalized feature map X′ to obtain a mapped feature map includes:

[0080] After performing a convolution operation on the normalized feature map X′ using a second convolution layer with a convolution kernel size of 1*1, normalization and nonlinear activation are performed, and a convolution operation is performed again using a third convolution layer with a convolution kernel size of 1*1 to obtain a mapping feature map.

[0081] Figure 5 Schematic diagram of collecting EEG signals provided by an embodiment of the present invention. Currently, the widely used non-invasive EEG recording methods are as follows: Figure 5 As shown in the figure, the number of leads is generally 32-256. However, volume conduction occurs during the acquisition of EEG signals, resulting in a spatial blurring effect of the signal, which makes it difficult to extract the spatial pattern of each BCI task in the brain. For the EEG signals of BCI tasks, the spatial activation patterns of different intensities between the lead distributions are crucial classification features. This embodiment uses a multi-head self-attention mechanism to extract the spatial activation pattern of EEG signals, focusing on the lead distribution information at different spatial levels in the spatial domain, and learning the representation similarity of different activation patterns, so that the model can learn richer spatial feature information.

[0082] Specifically, the spatial feature extraction module in the first spatiotemporal feature extraction module is still used as an example for explanation. The spatial feature extraction module uses each self-attention head to obtain spatial representation information from the first temporal feature X. Its feature acquisition method depends on the three vectors Q (Query), K (Key), and V (Value) projected by the input feature X through the linear transformation matrix, namely:

[0083] (Q,K,V)=Matmul(X,(W Q ,W K ,W V )) (1)

[0084] The difference in the weight matrices in formula (1) can ensure that the vectors Q, K, and V are projected into multiple different low-dimensional feature subspaces, thereby increasing the network's ability to express spatial features and improving the generalization ability of the model. Here, "low dimensionality" refers to a dimension lower than the dimension of the first time feature X.

[0085] Next, we perform a dot product of the Q and K vectors in the same low-dimensional projection subspace to calculate the attention score, transferring the spatial representation information recorded in the subspace and performing a scaling operation to avoid the vanishing gradient problem. The Softmax function is then applied to the above attention score and dot-producted with the vector V to complete the information extraction process. The mathematical representation is as follows:

[0086]

[0087] Here, Z represents the output matrix of the self-attention head, d KRepresents the dimension of vector K.

[0088] The multi-head self-attention mechanism maps the spatial features of EEG signals into different low-dimensional feature subspaces. Each self-attention head focuses on different levels of spatial information in different low-dimensional feature subspaces. Finally, the output matrices of each attention head are concatenated to achieve information complementarity, allowing the model to learn richer spatial feature information:

[0089] X multi-head =Concat(Z1,Z2,...,Z H ) (3)

[0090] In formula (3), X multi-head The resulting matrix is concatenated together by extracting features in the subspace for all attention heads.

[0091] Perform residual connections on the self-attention head and use BatchNorm (BN) for data normalization:

[0092] X′=BatchNorm(X+X multi-head ) (4)

[0093] The residual connection helps in back-propagation optimization of model parameters and effectively prevents gradient disappearance. In addition, this embodiment uses BN instead of LayerNorm because the former is more suitable for EEG data.

[0094] Furthermore, the normalized feature map X′ is subjected to feature mapping twice to learn more abstract and advanced spatial features. This embodiment uses the second and third convolutional layers with a convolution kernel size of 1×1 to complete this mapping process to reduce the network's learnable parameters:

[0095] FFN(X′)=f2 1*1 (Relu(f1 1*1 (X′))) (5)

[0096] In formula (5), f1 1*1 (·), f2 1*1 (·) represents a convolution operation with a convolution kernel size of 1*1, and FFN is a feedforward layer for abstract feature learning. In addition, this embodiment adds a batch normalization layer and activation function between the two mappings to increase the nonlinear expression capability of the network and enable better network convergence. At the same time, the mapped feature map and the normalized feature map X′ are residually connected and normalized using batch normalization to better train the network. The multi-head self-attention mechanism allows the model to jointly focus on the complex connections between signal leads, can extract multiple spatial activation information, and learn complementary information, which is beneficial to improving the feature quality of EEG signals.

[0097] It should be understood that EEG signals mostly exist in the time domain, and their subtle changes are somewhat correlated with BCI tasks in the temporal dimension. However, the human brain is a complex time-varying system, and traditional analysis methods have difficulty in obtaining sufficiently accurate temporal features, making effective temporal features difficult to extract. This paper designs a multi-scale factor convolutional layer and extracts temporal correlation information between EEG and BCI tasks based on depthwise separable convolution. It then fuses spatial activation information to form spatiotemporal features, thereby refining the characterization of EEG signals.

[0098] In the above step S3, considering that the fully connected layer (FC) will greatly increase the trainable weights of the network and there is a risk of overfitting, this embodiment uses global average pooling (GAP) instead of the fully connected layer commonly used in existing EEG decoding methods to integrate the spatiotemporal information of EEG signals. Global average pooling can complete the function of FC without increasing parameters. Specifically, GAP is a pooling layer that calculates the feature map of each channel as an average value and can accept inputs of any size. This embodiment performs two mapping convolutions with a convolution kernel size of (1,1) on the input X_classifier of the classifier. The purpose is to first map the input X_classifier to a high-dimensional space and then map it to a low-dimensional space, so that the network has the ability to extract deeper and more abstract spatiotemporal features. At the same time, the output channel of the feature map is adjusted in the second mapping to be consistent with the number of categories. In addition, this embodiment also adds an activation function Relu after each mapping to improve the nonlinear fitting ability.

[0099] Next, GAP is used to integrate the global spatiotemporal information of each channel and explicitly map the feature map to the class confidence p. Using GAP, the proposed HCANN can effectively organize the temporal and spatial features of multi-scale factors, emphasize the key discriminative spatiotemporal information, and provide concise information for the final decision. Then, Softmax performs the class confidence p on the input signal to complete the class prediction of the input signal, which is expressed as follows:

[0100]

[0101] Finally, the cross entropy loss function is used to help the network train, and the L2 norm is used to constrain the network parameters and reduce the overfitting problem, as shown in formula (7):

[0102]

[0103] Among them, y and represent the true label and predicted label respectively, B is the number of samples for each training, and θ represents the trainable parameters of the network.

[0104] The following simulation experiments further illustrate the spatiotemporal convolutional EEG decoding method fused with the multi-head self-attention mechanism provided by the present invention.

[0105] The EEG signal data used in this embodiment is shown in Table 1:

[0106] Table 1 EEG signal dataset

[0107]

[0108] (1) Rapid Serial Visual Presentation (RSVP)

[0109] This example uses the RSVP dataset as evoked EEG signals. This dataset contains EEG signals from one man and seven women (aged 19-29). During the experiment, participants were placed in a dimly lit room and maintained a comfortable sitting position. They viewed a 4.1-second image sequence on a computer screen at a frequency of 12 Hz. During viewing, participants were asked to press one of two finger buttons based on the image content (target aircraft / non-target). During training, correct / incorrect visual feedback, informing participants whether their judgment was correct, was provided along with the button press. 256-channel EEG data was recorded at a frequency of 256 Hz using a BioSemi ActiveTwo system. For further analysis, the signal was resampled to 128 Hz and then divided into 1-second segments, resulting in a 256×128 matrix for each sample. Subsequently, all samples were passed through a 6th-order bandpass Butterworth filter with a cutoff frequency between 0.1 and 48 Hz. This dataset was divided into a training set and a test set, which will be used directly for model evaluation.

[0110] (2) Motor imagery (MI)

[0111] The 2008 BCI Competition IV 2a dataset, provided by the Graz University of Technology, was used as experimental data for MI. In this dataset, nine participants performed MI experiments in four categories: left hand, right hand, both feet, and tongue. EEG signals were recorded for 576 trials from each participant. The MI EEG signals for each trial were recorded using 22 channels at a sampling rate of 250 Hz. In the following experiments, the time interval [1s, 4s] after the onset cue in each trial was considered, resulting in a 22×750 matrix for each sample. This example uses a causal third-order Butterworth filter with a cutoff frequency ranging from 4 to 38 Hz to filter the samples. For this competition, the 2a dataset was split into training and test sets. To ensure fair and unbiased comparison with other decoding works, the same partitioning was used in the following experiments.

[0112] (3) Imagery Speech (IS)

[0113] Imagined speech refers to speech activity without any clear articulatory movements, despite normal cognitive and language activity. The dataset contains EEG signals from nine subjects, classified into eight categories: baboon, lion, rhino, zebra, apple, banana, grape, and watermelon. The experiment used pictures as cues for the content of imagined speech. Participants were first asked to evaluate the pictures and then silently read the corresponding words in their heads without overt vocal cord or muscle movement. This process lasted one second and constituted a single imagined speech. The dataset consisted of 10 experimental blocks, each with 20 units; each unit contained eight imagined speech trials, corresponding to the eight categories of words. 64-channel EEG data was recorded at 1024 Hz using a BioSemi ActiveTwo system. A 50 Hz FIR notch filter was then applied to eliminate power line interference; a 1-90 Hz zero-phase FIR bandpass filter was then applied to the raw EEG signals. The EEG was resampled to 256 Hz and normalized using the z-score. The final sample size of the dataset was 64 x 256. The EEG signals are divided into training and testing samples in a ratio of 8:2.

[0114] Table 2 lists the hyperparameter settings of the EEG decoding network for each dataset, including the convolution kernel size T of the temporal feature extraction module. i And N of the spatial feature extraction module i and H i , where HCANN represents the above-mentioned EEG decoding network that integrates separable convolution and multi-head attention mechanism.

[0115] The Adan optimizer was used to train and optimize the model parameters. A multi-step scheduler was used, and the learning rate was decreased by a factor of 0.5 every 200 epochs, starting from the 500th epoch. Stratified 5-fold cross-validation was performed to evaluate the performance of HCANN.

[0116] Table 2 Hyperparameter settings of EEG decomposition network on each dataset

[0117]

[0118] Table 3 Experimental results on three datasets

[0119]

[0120] Then the experimental results on three datasets are analyzed.

[0121] (1) RSVP

[0122] The proposed EEG decoding network demonstrated slightly better performance than the baseline. Compared to the baseline method, the EEG decoding network achieved a 19.14% improvement in accuracy, and compared to a model specifically designed for RSVP (PPN), the accuracy improved by 1.91%. Its performance was also more stable than other methods in terms of standard deviation. Therefore, multi-head self-attention can effectively work on RSVP tasks.

[0123] (2)MI

[0124] From the results in Table 3, we can see that the standard deviation of the EEG decoding network is slightly smaller than that of other methods, but its performance is reliable and accurate. The experiments show that the hybrid model of CNN and multi-head self-care mechanism can effectively decode MI EEG signals.

[0125] (3)IS

[0126] In the classification of IS, the EEG decoding network proposed in this paper performs better than HS-STDCN. Compared with MI, the working mechanism of IS is more conducive to the practical application of BCI.

[0127] To quantify the performance improvements of the EEG decoding method provided by the present invention, further statistical significance tests were performed. Specifically, a paired sample t-test was performed on the results of HCANN and the other comparison methods. When the P value of the statistical result was less than 0.05, it was considered that there was a significant difference between the two methods. Table 4 shows all P values. It can be seen that HCANN performed better than the other comparison methods on the three datasets, and its performance on the IS dataset was particularly outstanding.

[0128] Table 4 Paired t-test results

[0129]

[0130]

[0131] Figure 6a 、 6c 6e are respectively visualization diagrams of RSVP, MI, and IS spatial-temporal features of three EEG tasks provided by the embodiment of the present invention, Figure 6b 、 6d, 6f are schematic diagrams of RSVP, MI, and IS confusion matrices provided by embodiments of the present invention, respectively. The method proposed in the present invention uses a multi-head self-attention mechanism and CNN to decode spatiotemporal information and classify the EEG signals of the three BCI tasks. In order to have a more intuitive understanding of the signal decoding process of HCANN, the t-distributed stochastic neighbor embedding (t-SNE) method is used to visualize the spatial-temporal features of the two stages, which come from the two spatial blocks of HCANN. In addition, the present invention also proposes a confusion matrix of HCANN under each classification task. The results are shown in Figures 6a-6f In "1 st ", represents the first space-time feature output by the spatial feature extraction module in the first space-time feature extraction module, "2 st " represents the second space-time feature output by the spatial feature extraction module in the second space-time feature extraction module. As the EEG signal is decoded, the space-time feature becomes more and more separable. These visualization results show that the EEG decoding network provided by the present invention can effectively extract discriminative features from EEG signals.

[0132] In the space module, the number of stacks N i and the number of self-attention heads H i It has a great impact on the feature extraction ability of the model. Appropriate N i and H i The number of self-attention heads is conducive to balancing the complexity and expressiveness of the model. Too many self-attention heads will produce redundant model parameters, which is not conducive to model reasoning and feature extraction. i and H i , to examine their impact on HCANN performance across all three tasks. Table 5 lists the relevant results; values not listed are the same as those for the base model in Table 2. The results across all three tasks demonstrate that too many self-attention heads do not improve the model's decoding ability. Furthermore, larger models exhibited better decoding performance but were prone to overfitting.

[0133] Table 5 Spatial block parameter analysis

[0134]

[0135]

[0136] To further analyze the similarity of features extracted by the multi-head self-attention mechanism within a spatial block, we used center kernel alignment (CKA) to calculate the inter-head similarity for each pair of self-attention heads. CKA values range between 0 and 1, with larger values representing higher similarity. CKA is robust even with a small number of samples, making it well-suited for model evaluation on EEG data. Table 6 shows the average CKA values for the two decoding stages across the three tasks. Stages 1 and 2 represent the representation features extracted by the first and second spatial modules, respectively. All CKA values are less than 0.4, demonstrating the diversity of the heads.

[0137] Table 6 Average CKA in two decoding stages in three tasks

[0138]

[0139] In the transformer, the multi-head self-attention mechanism uses the LN layer to normalize the data. According to previous studies, BN is more suitable for EEG signals. Therefore, the EEG decoding signal provided by the present invention uses BN to normalize the EEG features. In order to verify whether BN is superior to LN in EEG data processing, variants of the EEG decoding network are used for three tasks and the results are observed. As shown in Table 7, the performance of the EEG decoding network based on LN is lower than that of the EEG decoding network based on BN, and the stability is also poor. The data characteristics of EEG signals are different from those of NLP. BN is conducive to stabilizing the distribution of EEG features, while LN is more suitable for processing data of variable length.

[0140] Table 7BN-based HCANN vs LN-based HCANN

[0141]

[0142] In summary, the EEG decoding network proposed in this paper is a new deep learning network that can accurately describe EEG features. It combines the advantages of CNN and multi-head self-attention mechanism. The deep separable convolution with multi-scale factors effectively disentangles the temporal correlation between BCI tasks and EEG signals, which improves the model's ability to describe temporal features. The multi-head self-attention mechanism adaptively modified for EEG data allows the model to focus on the complex spatial activation patterns between EEG channels, extract multiple spatial representations, and learn complementary features to improve the quality of discriminative information. Finally, the spatial-temporal features are integrated using a global average pooling layer to obtain discriminative features. Experimental results show that HCANN can effectively improve the classification performance of all three EEG tasks.

[0143] The embodiment of the present invention further provides an electronic device, such as Figure 7As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0144] Memory 703, for storing computer programs;

[0145] The processor 701 is configured to execute the program stored in the memory 703, and implement the following steps:

[0146] Collect EEG signals;

[0147] Inputting the EEG signal into the EEG decoding network, so that the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module in the EEG decoding network process the EEG signal in sequence, wherein the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module are used to extract the spectral features of the input data at different bandpass frequencies, and after extracting the temporal features based on the spectral features, use the multi-head self-attention mechanism to extract the spatial-temporal features;

[0148] The spatial-temporal features output by the second spatial-temporal feature extraction module are classified to obtain the classification results of the EEG signal.

[0149] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0150] The communication interface is used for communication between the above electronic device and other devices.

[0151] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0152] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0153] The method provided in the embodiments of the present invention can be applied to electronic devices. Specifically, the electronic devices can be desktop computers, portable computers, smart mobile terminals, servers, etc. This is not limited here; any electronic device that can implement the present invention falls within the scope of protection of the present invention.

[0154] As for the device / electronic device / storage medium embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] It should be noted that the device, electronic device and storage medium of the embodiments of the present invention are respectively the device, electronic device and storage medium of the spatiotemporal convolutional EEG decoding method fused with the above-mentioned multi-head self-attention mechanism. All embodiments of the spatiotemporal convolutional EEG decoding method fused with the above-mentioned multi-head self-attention mechanism are applicable to the device, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0156] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0157] Descriptions with reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0158] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims.

[0159] The above is a further detailed description of the present invention in conjunction with specific preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be considered to fall within the scope of protection of the present invention.

Claims

1. A spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism, characterized by: include: Collect EEG signals; Inputting the EEG signal into an EEG decoding network, so that a first spatiotemporal feature extraction module and a second spatiotemporal feature extraction module in the EEG decoding network sequentially process the EEG signal, wherein the first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module are used to extract spectral features of the input data at different bandpass frequencies, and after extracting temporal features based on the spectral features, use a multi-head self-attention mechanism to extract spatial-temporal features; Classifying the space-time features output by the second space-time feature extraction module to obtain a classification result of the EEG signal; The first spatiotemporal feature extraction module and the second spatiotemporal feature extraction module both include a time feature extraction module and a space feature extraction module; After the EEG signal is input into the EEG decoding network, the first spatiotemporal feature extraction module processes it according to the following steps: The time feature extraction module in the first spatiotemporal feature extraction module uses the first convolution layer to extract the spectrum features of the EEG signal under different bandpass frequencies, and uses the separable convolution layer to perform depth convolution and point convolution on the spectrum features to obtain the first time feature. X ; The spatial feature extraction module in the first spatiotemporal feature extraction module is based on the first temporal feature X ,using the multi-head self-attention mechanism to extract the first spatial-temporal features; The spatial feature extraction module in the first spatiotemporal feature extraction module is based on the first temporal feature X ,The steps of extracting the first spatial-temporal feature using the multi-head self-attention mechanism include: The spatial feature extraction modules in the first spatiotemporal feature extraction module use their respective attention heads to extract the first temporal feature. X Perform linear transformation to project to different low-dimensional feature subspaces to obtain vectors Q ,vector K and vector V : ； Where, represents matrix multiplication, 、 and Represents a preset weight matrix, the dimension of the low-dimensional feature subspace is lower than the first time feature X Dimensions; Vector-based Q ,vector K and vector V Calculate the output matrix of each self-attention head: ； Where, Represents a vector K Dimensions; Concatenate the output matrices of all self-attention heads to get the result matrix: ； Where, represents the number of self-attention heads; The result matrix With the first time feature X Make residual connections and use BatchNorm Perform normalization: ； Normalized feature map Perform feature mapping to obtain a mapping feature map; The mapping feature map and the normalized feature map Make residual connections and use BatchNorm Normalize and obtain the first space-time feature.

2. The spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism according to claim 1 is characterized in that: The convolution kernel size of the first convolution layer is ( fs / 2,1), where fs is the sampling frequency of the EEG signal.

3. The spatiotemporal convolutional EEG decoding method fused with a multi-head self-attention mechanism according to claim 1 is characterized in that: Normalized feature map The steps of performing feature mapping and obtaining a mapping feature map include: The normalized feature map is processed using the second convolution layer with a convolution kernel size of 1*1. After the convolution operation, normalization and nonlinear activation are performed, and the convolution operation is performed again using the third convolution layer with a convolution kernel size of 1*1 to obtain the mapping feature map.

4. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 3 when executing a program stored in a memory.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 3 are implemented.

Images