Electroencephalogram attention classification model and method based on spatio-temporal feature fusion

Abstract

The application relates to an electroencephalogram attention classification model based on space-time feature fusion and a method thereof, which comprises a convolution block, a graph convolution network (GCN), a time series convolution network (TCN) and an attention feature fusion network. First, the convolution block processes the electroencephalogram signal in the time and space dimensions to extract initial time and space feature maps; then, the GCN captures the spatial dependence between electrodes, and the TCN captures the dynamic characteristics of the signal change over time; subsequently, the feature maps of the two are fused and weighted summed through the attention mechanism to form the final fusion features. Finally, the full connection layer is used in combination with the Softmax function to map the final fusion features to the category probability, and the classification result is output. The application can effectively improve the accuracy and efficiency of electroencephalogram signal classification.

Description

Technical Field

[0001] This invention relates to the field of electroencephalogram (EEG) signal processing technology, specifically to an EEG attention classification model and method based on spatiotemporal feature fusion. Background Technology

[0002] Electroencephalography (EEG) is a key technique for studying brain function and state by recording the electrical activity of the cerebral cortex. Due to its non-invasiveness, high temporal resolution, and relatively low cost, EEG technology has been widely used in neuroscience, clinical diagnostics, brain-computer interfaces (BCI), and cognitive science. In attention detection and classification studies, EEG signals have become an important research tool because they can reflect the dynamics of brain activity in real time.

[0003] In recent years, the development of deep learning technology has brought new opportunities for the automatic classification of electroencephalogram (EEG) signals. Traditional EEG classification methods typically rely on manual feature extraction and conventional machine learning models, but these methods face several challenges when processing complex EEG signals. For example, traditional methods are often sensitive to noise, cannot fully utilize the potential information in multi-channel signals, and have low feature extraction efficiency. Furthermore, these methods struggle to capture the complex interactions between temporal dynamics and spatial features when dealing with complex EEG signal data, resulting in low recognition accuracy. With the rapid development of deep learning technology, models based on convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have shown strong advantages in automatic feature extraction and multi-level information capture, greatly improving the performance of EEG signal classification. However, since EEG signals are essentially multi-channel spatiotemporal sequence data, relying solely on a single feature extraction method is insufficient to comprehensively represent the brain's attentional state. In existing research, how to further enhance the fusion capability of spatiotemporal features while ensuring model efficiency, and how to effectively combine attention mechanisms to improve attention classification results, remain unsolved problems and research challenges. Summary of the Invention

[0004] The present invention aims to provide an EEG attention classification model and method based on spatiotemporal feature fusion, so as to effectively solve the problems of low classification accuracy, incomplete feature extraction and limited ability to process complex EEG signals in the prior art.

[0005] Firstly, this paper proposes an EEG attention classification model based on spatiotemporal feature fusion, comprising: a convolutional block configured to perform convolution operations on EEG signals in both temporal and spatial dimensions; a graph convolutional network configured to perform graph convolution operations on the feature maps output by the convolutional block to capture the spatial dependencies between electrodes; a temporal convolutional network configured to perform dilated causal convolution on the feature maps output by the graph convolutional network in the temporal dimension to capture the dynamic changes of EEG signals over time; and an attention feature fusion network configured to initially fuse the spatial feature maps extracted by the graph convolutional network and the temporal feature maps extracted by the temporal convolutional network to form an initial fused feature map F. fusion The initial fused feature map F is applied using attention weights α. fusion We perform weighting to obtain the weighted feature map F. attention For the initial fused feature map F fusion With the weighted feature map F attention We perform a weighted summation to obtain the final fusion feature F. final ; and a classifier, which is configured to fuse the final features F through a fully connected layer. final The vector is mapped to the number of categories, and the category probabilities are calculated using the Softmax function. Finally, the attention classification result is output.

[0006] In some examples, the convolutional block comprises: a first two-dimensional convolutional layer configured to perform independent convolution operations on the EEG signals of each channel, capturing dynamic characteristics across multiple time points and generating an initial temporal feature map reflecting the temporal correlation between channels; a first batch normalization layer configured to perform batch normalization on the initial temporal feature map generated by the first two-dimensional convolutional layer; a second two-dimensional convolutional layer configured to convolve the batch-normalized initial temporal feature map in the spatial dimension, extracting the relationship between different channels and generating an initial spatial feature map; a second batch normalization layer configured to perform batch normalization on the initial spatial feature map; an activation function configured to apply a nonlinear transformation to the batch-normalized initial spatial feature map; a pooling layer configured to perform average pooling on the initial spatial feature map after the activation function; and a depthwise separable convolutional layer configured to perform depthwise separable convolution operations on the average-pooled initial spatial feature map.

[0007] In some examples, depthwise separable convolutional layers include: depthwise convolution, which performs spatial convolution on each input channel, keeping the number of channels consistent with the input; and pointwise convolution, which performs cross-channel feature fusion.

[0008] In some examples, after spatial convolution is performed on each input channel by depthwise convolution, the feature maps of each channel are sequentially subjected to batch normalization and ReLU nonlinear activation.

[0009] In some examples, the ReLU activation function is applied after pointwise convolution.

[0010] In some examples, an average pooling layer is configured after the ReLU activation function, which performs spatial downsampling to reduce the spatial dimension of the feature map input to the graph convolutional network.

[0011] In some examples, the initial fused feature map F fusion Global average pooling is performed to obtain a global feature vector. The global feature vector is then input into a fully connected layer and passed through the non-linear activation function ReLU to generate attention weights α.

[0012] Secondly, a brainwave attention classification method is proposed, comprising: collecting brainwave signals from subjects; inputting the brainwave signals into the brainwave attention classification model based on spatiotemporal feature fusion, wherein the model outputs attention classification results.

[0013] Thirdly, a computer system is proposed, comprising: a processor; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the EEG attention classification method.

[0014] Fourthly, a computer-readable storage medium is proposed for storing non-transitory computer-readable instructions that, when executed by a computer, enable the EEG attention classification method.

[0015] This invention can effectively extract and fuse spatiotemporal features from EEG signals. Through the synergistic effect of various parts of the model, it highlights key features while suppressing redundant information, thereby significantly improving the accuracy of EEG attention classification.

[0016] This invention automatically identifies and assigns higher weights to the feature channels most important for the classification task by calculating attention weights, thereby enhancing attention to key spatiotemporal features and further optimizing classification performance. This mechanism enables the model to more accurately focus on task-related features, improving the accuracy of EEG attention classification.

[0017] The EEG attention classification model of this invention adopts a rational and modular design, which facilitates expansion and adjustment in practical applications and can flexibly adapt to different application scenarios and needs. By applying deep learning to EEG signal analysis and using artificial intelligence algorithms for feature extraction and classification, this invention not only improves classification accuracy but also makes it easy to promote and apply to fields such as daily mental health monitoring and attention state assessment. Attached Figure Description

[0018] Figure 1 This is a flowchart of an EEG attention classification method based on spatiotemporal feature fusion according to an embodiment of the present invention.

[0019] Figure 2 This is a block diagram of an EEG attention classification model based on spatiotemporal feature fusion according to an embodiment of the present invention.

[0020] Figure 3 This is a structural diagram of an EEG attention classification model based on spatiotemporal feature fusion according to an embodiment of the present invention.

[0021] Figure 4 This is an electrode distribution diagram for collecting electroencephalogram (EEG) signals according to an embodiment of the present invention. Detailed Implementation

[0022] This invention enables efficient and accurate classification of attention states, making it particularly suitable for applications such as education and healthcare that require real-time monitoring and assessment of attention states. The core of this invention lies in enhancing the classification model's ability to capture multi-dimensional information through deep fusion of spatiotemporal features, thereby significantly improving the accuracy of EEG signal classification. Compared to traditional methods, this invention can more comprehensively characterize the brain's attention states and, by introducing an attention mechanism, enables the model to focus on the most recognizable feature signals, further improving classification performance.

[0023] Figure 1 , Figure 2 , Figure 3 This paper demonstrates the workflow of an EEG attention classification method based on spatiotemporal feature fusion. Next, we will combine... Figure 1 , Figure 2 and Figure 3 This method will be explained in detail.

[0024] 1. EEG signal acquisition

[0025] Multichannel EEG signals were collected from subjects under different attentional states to provide a data foundation for subsequent EEG classification. For example, when collecting the subjects' EEG signals, the sampling rate was set to 1000Hz, and 32 channels covering the prefrontal, parietal, central, occipital, and temporal lobes of the brain were selected, specifically including Fz, Fp1, Fp2, F3, F4, F7, F8, FC1, FC2, FC5, FC6, FT9, FT10, Cz, C3, C4, T7, T8, CP1, CP2, CP5, CP6, TP9, TP10, Pz, P3, P4, P7, P8, Oz, O1, and O2. In these channels, Fz was used as the reference electrode, while the ground electrode (GND) was placed on the subject's forehead. Figure 4 As shown.

[0026] 2. EEG signal preprocessing

[0027] The collected EEG signals are preprocessed to improve data quality and consistency, ensuring the accuracy of subsequent analysis.

[0028] 2.1 Bandpass Filtering

[0029] The raw EEG signal was filtered using a bandpass filter of 0.5-30Hz to remove unwanted signals in the frequency range, including higher frequency electromyography (EMG) signals and lower frequency electrocardiogram (ECG), electrodermal (ED) signals, and respiratory signals.

[0030] 2.2 Independent Component Analysis (ICA) Noise Reduction

[0031] Independent component analysis (ICA) was used to remove electrooculography (EOG) signals from the original signal. First, the filtered electroencephalogram (EEG) signal X... filtered The mixed signal is decomposed into independent components, and the calculation formula is as follows:

[0032] S = WX filtered ;

[0033] Where S is the independent component matrix, and W is the inverse matrix of the mixed signal matrix A found by the ICA algorithm.

[0034] Then, the components of the electrooculogram (EOG) signal are identified, and these components are removed from the independent component matrix S. The calculation formula is as follows:

[0035] S EEG =SS EOG ;

[0036] Where S EOG It is a component of electrooculography (EOG) signal, S EEG It is the remaining brainwave component.

[0037] Finally, use the remaining independent component S. EEG The EEG signal is reconstructed using the mixed signal matrix A, and the calculation formula is as follows:

[0038] X clean =AS EEG ;

[0039] Where X clean This is the reconstructed / preprocessed EEG signal. This step effectively removes noise signals, improving the accuracy of subsequent analysis and classification.

[0040] 3. EEG Attention Classification Model Based on Spatiotemporal Feature Fusion

[0041] The preprocessed EEG signals are input into an EEG attention classification model (hereinafter referred to as "model" or "classification model") based on spatiotemporal feature fusion for analysis and classification, thereby accurately identifying the test subject's attention state.

[0042] like Figure 2 , Figure 3 As shown, the classification model includes convolutional blocks, graph convolutional networks, temporal convolutional networks, attention feature fusion networks, and a classifier.

[0043] 3.1 Convolutional Blocks

[0044] The convolutional block contains two two-dimensional convolutional layers and a depth-separable convolutional layer, used to process the temporal and spatial features of EEG signals.

[0045] First, the EEG signals of each channel are processed independently in the time dimension. Specifically, a two-dimensional convolutional kernel of size (1, 64) is used to capture dynamic characteristics across multiple time points. Through a sliding window mechanism, the convolutional kernel performs convolution operations on the time axis, generating feature maps that reflect the temporal correlation between each channel. Subsequently, these temporal feature maps undergo batch normalization to accelerate the model training process and stabilize the training results.

[0046] Next, following batch normalization, a second two-dimensional convolutional layer is applied, this time working in the spatial dimension to extract relationships between different channels. The kernel size used here is (32,1), which covers local regions of multiple channels, thereby capturing interaction information between EEG signal channels and generating new spatial feature maps. These spatial feature maps are then batch normalized again, and the ELU (Exponential Linear Unit) activation function is used to more effectively represent spatial features. Note that the kernel sizes of the two two-dimensional convolutional layers are merely illustrative.

[0047] Finally, average pooling is performed on the batch-normalized spatial feature maps. This step helps to further compress the feature representation, reduce data dimensionality, and retain important spatial information.

[0048] Depthwise separable convolution is performed on the pooled spatial feature maps. First, spatially independent depthwise convolutions (e.g., 32×1 kernels) are used to perform spatial convolution on each input channel, maintaining the same number of channels as the input. The feature maps for each channel are then subjected to batch normalization and ReLU non-linear activation. Next, cross-channel feature fusion is performed using 1×1 pointwise convolutions, utilizing a learnable weight matrix to flexibly adjust the number of channels, and a ReLU activation function is applied after this convolutional layer to enhance non-linear expressiveness. After feature fusion, an average pooling layer is used for spatial downsampling to reduce the spatial dimension of the feature maps. In practical applications, to prevent overfitting and improve the model's generalization ability, Dropout can be introduced between the average pooling layer of the depthwise separable convolution operation and subsequent layers (e.g., the convolutional layer in the figure). This randomly masks neurons to suppress excessive dependence on specific features during training.

[0049] 3.2 Graph Convolutional Networks

[0050] Spatial dependencies of features are extracted using two graph convolutional layers.

[0051] The feature map corresponding to the reduced spatial dimension of the previous step is represented as a node feature matrix H. (0) Each node's feature vector represents the temporal characteristics of an electrode. An adjacency matrix A is constructed based on the physical distance and signal correlation between electrodes, defining the connection structure in the graph. Where A... ij The strength of the connection between node i and node j can typically be defined using a Gaussian kernel function or other similarity metrics. ij The value is then used. Then, the first layer of graph convolution is performed, calculated using the following formula:

[0052]

[0053] Wherein: H (1) It is the node feature matrix of layer 1; It is the adjacency matrix after adding self-connections, and I is the identity matrix; yes The degree matrix is ​​defined as follows: W (0) σ is a trainable weight matrix, and σ(·) is the ReLU activation function.

[0054] Then the output H of the first layer (1) The input is fed into the second layer of graph convolution to further extract deeper spatial features, and the calculation formula is the same as above.

[0055]

[0056] Graph convolution operations aggregate node features through adjacency and degree matrices, which can capture the spatial dependencies between electrodes, thereby better modeling the features of EEG signals.

[0057] 3.3 Temporal Convolutional Networks

[0058] Temporal convolutional networks are used to capture the dynamic changes in EEG signals over time.

[0059] Using spatial features extracted by a graph convolutional network as input, a first layer of dilated causal convolution is performed in the temporal dimension with an expansion rate of 2. After the convolution operation, batch normalization and the ELU activation function are applied. To enhance the model's generalization ability, Dropout can be introduced into the first layer of dilated causal convolution. The second layer of dilated causal convolution is similar to the first layer, but with an expansion rate of 4, thus capturing dependencies at different time scales. Similarly, batch normalization and the ELU activation function are applied. Subsequently, the input is directly added to the output of the second layer of dilated causal convolution to form a residual connection. This design ensures that gradients can propagate effectively in deep networks, alleviating the gradient vanishing problem. Finally, the ELU activation function is applied again to the output after the residual connection to further enhance the ability to represent nonlinear features.

[0060] 3.4 Attention Feature Fusion Network

[0061] An Attentional Feature Fusion (AFF) network is used to focus on the most important features.

[0062] First, the spatial feature map extracted by the graph convolutional network and the temporal feature map extracted by the temporal convolutional network are initially fused to form an initial fused feature map F. fusion Next, the initial fused feature map F... fusion Global average pooling is performed to obtain global context information. The formula for calculating global average pooling is as follows:

[0063]

[0064] in, It is a global feature vector, C fusion The number of channels for the fused features is T, where T represents the total number of time steps, t is the time step, and c is the number of channels.

[0065] The global feature vector z is input into a fully connected layer, and after passing through the ReLU non-linear activation function, attention weights α are generated. The calculation formula is as follows:

[0066] α = ReLU(Wz + b);

[0067] Where α is the attention weight, and W and b are the learnable weight matrix and bias vector, respectively.

[0068] The calculated attention weight α is used to fuse the feature map F. fusion Weighting is applied to highlight important feature channels. The weighted feature map F attention The calculation formula is as follows:

[0069] F attention (t,c)=α c ·F fusion (t,c);

[0070] Among them, F attention It preserves temporal information while enhancing the expression of key features.

[0071] The original feature map and the attention-weighted feature map are summed by weight to obtain the final fused feature F. final The calculation formula is as follows;

[0072] F final =F fusion +λ·F attention ;

[0073] Among them, F final λ represents the final fused features, and λ is a learnable fusion weight used to balance the importance of the original features and the attention-weighted features.

[0074] 4. Classification and Output

[0075] The final fused features are mapped to a vector of class numbers through a fully connected layer, and then the class probabilities are calculated using the Softmax function. The classifier then outputs the attention classification result.

[0076] The test subjects' EEG data are collected, and a trained model is used to detect the test subjects' EEG signals and classify the test subjects' attention states.

[0077] When collecting EEG data from test subjects using EEG caps or other EEG acquisition devices, it should be noted that 32 channels must also be used for data acquisition, with a sampling rate set to 1000Hz.

[0078] The trained model is used to detect the EEG signals of the test subjects and classify their attentional states.

[0079] The present invention also provides an embodiment of a computer system. This computer system is a broad concept, encompassing not only traditional computers but also external devices related to brain-computer interfaces. A memory is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). A processor is responsible for executing these instructions; when the processor executes these non-transitory computer-readable instructions, it is able to implement one or more steps in the aforementioned EEG attention classification method.

[0080] The memory and processor can be interconnected via a bus system or other forms of connection to ensure efficient transmission of data and control signals. This design allows for efficient instruction execution and support for complex computational tasks, such as the implementation of EEG attention classification methods.

[0081] For example, a processor can be a central processing unit (CPU), a graphics processing unit (GPU), or other form of processing unit with data processing and / or program execution capabilities. For instance, a CPU can be based on x86 or ARM architectures. A processor can be a general-purpose processor or a special-purpose processor, and it can control other components in a computer to perform desired functions.

[0082] For example, memory can include any combination of one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory can include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory can include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), compact optical disc read-only memory (CD-ROM), USB storage, flash memory, etc. One or more computer program modules can be stored on the computer-readable storage medium, and the processor can run one or more computer program modules to implement various functions of the computer.

[0083] This invention also provides a computer-readable storage medium for storing non-transitory computer-readable instructions. When executed by a computer, these instructions can implement one or more steps in the aforementioned EEG attention classification method. When the EEG attention classification model and method provided in this invention are implemented in software and sold or used as independent products, they can be stored in a computer-readable storage medium. For further details regarding the storage medium, please refer to the corresponding description of memory in a computer above; further explanation is not provided here.

Claims

1. A brainwave attention classification method, characterized in that, Includes: collecting electroencephalogram (EEG) signals from subjects; EEG signals are input into an EEG attention classification model based on spatiotemporal feature fusion. The model outputs an attention classification result. The model includes: Convolutional blocks are configured to perform convolution operations on EEG signals in both temporal and spatial dimensions; Graph convolutional networks are configured to perform graph convolution operations on the feature maps output by convolutional blocks to capture spatial dependencies between electrodes. Temporal convolutional networks are configured to perform dilated causal convolutions on the feature maps output by graph convolutional networks in the temporal dimension to capture the dynamic changes of EEG signals over time. An attention-based feature fusion network is configured to initially fuse the spatial feature maps extracted by a graph convolutional network and the temporal feature maps extracted by a temporal convolutional network to form an initial fused feature map. ; the initial fused feature map Global average pooling is performed to obtain a global feature vector. This global feature vector is then input into a fully connected layer, and after passing through the non-linear activation function ReLU, attention weights are generated. Use attention weights For the initial fused feature map We perform weighting to obtain the weighted feature map. ; and through learnable fusion weights For the initial fused feature map Compared with the weighted feature map We perform a weighted summation to obtain the final fusion feature. ;as well as A classifier, configured to fuse the final features through fully connected layers. The vector is mapped to the number of categories, and the category probabilities are calculated using the Softmax function. Finally, the attention classification result is output.

2. The EEG attention classification method according to claim 1, characterized in that, Convolutional blocks contain: The first two-dimensional convolutional layer is configured to perform independent convolution operations on the EEG signals of each channel, capture dynamic characteristics across multiple time points, and generate an initial temporal feature map that reflects the temporal correlation between the channels. The first batch normalization layer is configured to perform batch normalization on the initial temporal feature map generated by the first two-dimensional convolutional layer; The second two-dimensional convolutional layer is configured to convolve the initial temporal feature map after batch normalization in the spatial dimension, extract the relationship between different channels, and generate the initial spatial feature map. The second batch normalization layer is configured to perform batch normalization on the initial spatial feature map; The activation function is configured to apply a nonlinear transformation to the initial spatial feature map after batch normalization. Pooling layers are configured to perform average pooling operations on the initial spatial feature maps after activation functions; as well as Depthwise separable convolutional layers are configured to perform depthwise separable convolution operations on the initial spatial feature map after average pooling.

3. The EEG attention classification method according to claim 2, characterized in that, Depthwise separable convolutional layers include: Depthwise convolution performs spatial convolution on each input channel, maintaining the same number of channels as the input; and Pointwise convolution is used to fuse cross-channel features.

4. The EEG attention classification method according to claim 3, characterized in that, After performing spatial convolution on each input channel via depthwise convolution, the feature maps of each channel are sequentially subjected to batch normalization and ReLU nonlinear activation.

5. The EEG attention classification method according to claim 4, characterized in that, Apply the ReLU activation function after pointwise convolution.

6. The EEG attention classification method according to claim 5, characterized in that, The ReLU activation function is followed by an average pooling layer, which performs spatial downsampling to reduce the spatial dimension of the feature map input to the graph convolutional network.

7. A computer system, characterized in that, include: processor; Memory, including one or more computer program modules; The one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include instructions for implementing the EEG attention classification method according to any one of claims 1-6.

8. A computer-readable storage medium for storing non-transitory computer-readable instructions, characterized in that, When the non-transitory computer-readable instructions are executed by a computer, the EEG attention classification method according to any one of claims 1-6 can be implemented.

Images