EEG target spatial localization method based on spatiotemporal feature extraction and attention mechanism

Abstract

The present invention discloses a method for spatial positioning of EEG targets based on spatiotemporal feature extraction based on an attention mechanism. The method adopts the rapid serial visual presentation paradigm in the brain-computer interface to collect EEG data of the subjects when performing a viewing task. Then, the channel spatial attention features and spatiotemporal features of the EEG data are extracted in sequence, and the deep spatiotemporal features are further extracted. Finally, the spatial position of the target is located based on the deep spatiotemporal features. This method adopts two lightweight convolutional neural networks and a compact convolutional neural network in the feature extraction process, which effectively reduces the computational complexity of the network and is suitable for data training with small data volumes. The attention mechanism is added to the spatial and temporal feature extraction, which explicitly decouples the spatial and temporal information, utilizes the rich multi-channel information of the entire EEG, and can spatially locate targets from four directions.

Description

Technical Field

[0001] The present invention belongs to the technical field of brain-computer collaborative intelligent target detection, and relates to a method for decoding and classifying brain spatial selection attention information and locating the target space of the subject's attention, and specifically to a method for extracting EEG target space positioning based on the spatiotemporal characteristics of attention. Background Art

[0002] Brain-computer interfaces (BCIs) can convert brain information into commands, enabling direct communication between the brain and external devices. Their applications include helping to extend the motor abilities of patients with diseases. For example, gaze-independent BCI systems can be used to restore communication in patients with eye movement disorders. Humans' powerful visual perception abilities can also be leveraged to collaborate with machines to achieve enhanced intelligence. For example, in the military, large numbers of images are used to search for target images. In EEG image target interpretation, a reliable mechanism is the brain's spatial attention mechanism, which uses the spatial distribution characteristics of the brain after an event stimulus to determine target features. Image target interpretation includes target detection and positioning. In most applications, only the presence of a target in an image can be identified, and there is little research on target positioning in specific scenarios.

[0003] In attentional selection research, a typical electrophysiological marker is the N2pc, which appears at the contralateral electrode position relative to the target visual hemifield within 170–300 ms after stimulus onset. The N2pc not only reflects the covert shift of spatial attention but also relates to the spatial selectivity mechanisms that occur after such a shift. Spatial localization techniques based on the N2pc can distinguish left and right attentional selection. However, in real-world visual search, the target location is highly variable. Simply using the N2pc for spatial localization does not fully utilize the rich information from high-density EEG and cannot distinguish target locations at a more fine-grained level. In recent years, with the development of deep neural networks, more and more researchers have used neural network techniques to achieve end-to-end EEG feature extraction, demonstrating superior performance. However, extracting better spatiotemporal information and learning the underlying invariants of EEG features remain significant challenges. Summary of the Invention

[0004] One purpose of the present invention is to address the shortcomings of existing technologies and application scenarios, propose an EEG target spatial positioning method based on spatiotemporal feature extraction of attention mechanism, and establish a model including a channel-based spatial attention module, a time-based multi-scale attention module, and a spatiotemporal feature depth extraction module to achieve the target spatial positioning task.

[0005] The method of EEG target spatial positioning based on spatiotemporal feature extraction of attention mechanism is as follows:

[0006] Step 1: Obtain spatially localized EEG data

[0007] The rapid serial visual presentation (RSVP) paradigm in the brain-computer interface (BCI) was used to collect EEG data of the subjects while they were performing the viewing task. After the collection, the EEG data was preprocessed to form n time segments. i = 1, 2, ..., n, where C represents the number of EEG electrodes, T represents the number of time points in each time segment, and n represents the number of samples.

[0008] Step 2: Channel-based spatial attention calibration module

[0009] The channel-based spatial attention calibration module adaptively recalibrates the channel response characteristics for each input feature by explicitly modeling the interdependence between different electrode channels, and explores the discriminative features in the spatial domain. The specific method is to learn from SENet to perform the calculation on each time segment Q. i Perform squeeze and extraction operations. The squeeze operation uses a global average pooling layer to compress the global temporal information into a set of channel descriptors z c , to solve the channel dependency problem. After forming the channel descriptor, a simple gating mechanism and sigmoid activation function are added to generate weight factors to characterize the importance of each channel electrode. Finally, the weight factor is multiplied by the input feature to calculate the time segment Q i Recalibrate and complete channel feature extraction.

[0010] Step 3: Temporal multi-scale attention module

[0011] After channel feature extraction in step 2, another lightweight model, SKNet, is used to form separable spatial and temporal features. SKNet enables neurons to adaptively adjust the size of their receptive fields based on input information at different scales, improving the network's receptive field range while reducing model parameters and computational complexity. SKNet consists of three steps: separation, fusion, and selection. The separation operation uses convolution kernels of different sizes to generate different feature maps, learning spatiotemporal information at different scales. To enable neurons to adaptively adjust the size of their receptive fields based on input content, a gating mechanism similar to that used in step 2 is used to control information. The fusion operation integrates information from multiple branches to generate weight factors for branches at different scales. The selection operation utilizes a soft attention mechanism to adaptively select spatial information at different scales.

[0012] Step 4: Deep extraction module based on spatiotemporal features

[0013] The deep spatiotemporal feature extraction module consists of two submodules, employing a compact convolutional neural network to further obtain deep representations of temporal and spatial features. In the first submodule, two convolution operations are performed sequentially to extract features in the temporal and spatial dimensions, respectively. In the second submodule, separable convolution operations are used to reduce the number of parameters.

[0014] Step 5: Classification

[0015] The final feature information obtained after steps 2 to 4 is input into the linear classification layer, and the output information is compared with the category label of the target positioning direction to obtain the final classification result.

[0016] The present invention has the following beneficial effects:

[0017] 1. The use of two lightweight convolutional neural networks and one compact convolutional neural network effectively reduces the network's computational workload and is suitable for training with small amounts of data. An attention mechanism is added to spatial and temporal feature extraction to explicitly decouple spatial and temporal information, improving the network's spatial and temporal representation capabilities.

[0018] 2. This method utilizes the rich multi-channel information of the entire EEG and can spatially locate targets from four directions. Compared with the spatial attention localization method based on the single electrophysiological marker N2pc, it is more suitable for complex scenarios in actual situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flowchart of the EEG target space localization method for attention mechanism;

[0020] Figure 2 Figure 2 is a diagram of the experimental paradigm used in the examples;

[0021] Figure 3 This is a framework diagram of the EEG target space localization method based on the attention mechanism;

[0022] Figure 4 Schematic diagram of the channel-based spatial attention calibration module;

[0023] Figure 5 Schematic diagram of the time-based multi-scale attention module;

[0024] Figure 6 Schematic diagram of the depth extraction module based on spatiotemporal features; DETAILED DESCRIPTION

[0025] The method of the present invention is described in detail below with reference to the accompanying drawings.

[0026] like Figure 1As shown in the figure, the EEG target spatial positioning method based on the spatiotemporal feature extraction of the attention mechanism is as follows:

[0027] Step 1: Obtain spatially localized EEG data

[0028] This embodiment adopts the rapid serial visual presentation (RSVP) paradigm in the brain-computer interface (BCI). The stimulus image is an aerial image that has been grayscale processed. The image dataset comes from the remote sensing image target dataset of Wuhan University. Aerial images containing aircraft targets and no targets are screened out. After image grayscale processing, the number of aircraft targets is 1, and the aircraft targets are located in any of the four quadrants of the two-dimensional coordinate system of the image. Five types of images were used in the experiment, including images without aircraft targets, images with aircraft targets located in the upper left corner of the screen, images with aircraft targets located in the upper right corner of the screen, images with aircraft targets located in the lower right corner of the screen, and images with aircraft targets located in the lower left corner of the screen. The specific experimental process is as follows. Figure 2 As shown, during the experiment, participants sat in a comfortable chair approximately 70 cm from the screen. EEG data were collected non-invasively using a 64-channel Neuroscan device, employing a 10-20 system electrode placement method with a sampling rate of 1000 Hz. The impedance of all electrodes was maintained below 15Ω before EEG acquisition. Participants completed 10 blocks, each consisting of 60 trials, with a total of 600 images presented, with a target-to-nontarget ratio of 1:9. At the beginning of each trial, a "+" sign appeared in the center of the screen for 5 seconds, requiring participants to focus their attention in preparation for the experiment. Afterward, 60 images were presented in succession, each lasting 1 second. Participants were instructed to stare at the center of the screen and then at the target after seeing it.

[0029] The collected EEG data were first processed by Butterworth bandpass filtering to 0.1-40 Hz, and then downsampled to downsample the 1000 Hz EEG data to 250 Hz. Finally, the continuous EEG data were divided into data segments from 0 to 1000 ms after the start of the stimulus, and the 300 ms data before the stimulus were used for baseline calibration.

[0030] Step 2: Channel-based spatial attention calibration module

[0031] like Figure 3 As shown, the EEG data of each time segment obtained in step 1 is Where C represents the number of EEG electrodes, T represents the number of time points in each time slice, and n represents the number of samples. For each time slice, the channel-based spatial attention calibration module explicitly models the interdependence between different channels, adaptively recalibrates the electrode channel response characteristics, and explores the discriminative features in the spatial domain. Figure 4As shown, each time segment is squeezed and extracted using Squeeze and Excitation Network (SENet). First, the squeeze operation uses a global average pooling layer on the input features to generate a channel descriptor z by aggregating feature maps in the spatial dimension. c , the role of the channel descriptor is to generate a global distribution embedding of channel feature responses, allowing information from the global receptive field of the network to be used by all its layers, thereby solving the channel dependency problem:

[0032]

[0033] Among them, F sq () represents the global average pooling operation, z ic represents the descriptor of the cth channel in the i-th sample, Q ic Represents the signal of the cth channel in the i-th sample q ic (j) represents the jth time point of the cth channel in the i-th sample. Then, in order to explore the correlation between electrode channels, a simple gating mechanism and sigmoid activation function are added. The gating mechanism uses two fully connected layers, namely a dimensionality reduction layer with a decay rate of r1, a ReLU activation layer, and a dimensionality increase layer to return to the same number of channels as z. The purpose of this is to convert the input-specific channel descriptor z c Mapped into a set of channel weights, which is regarded as a self-attention function on the electrode channels, the formula for generating the activation factor is as follows:

[0034] s ic =F ex (z, W)=σ(g(z, W))=σ(W2δ(W1z)

[0035] F ex () represents two fully connected operations, which are used to learn the correlation between channels, δ() represents the ReLU function, and σ() represents the sigmoid activation function. are the parameters of the two fully connected layers. The decay rate r1 is also a hyperparameter in this network. It affects the capacity and computational cost of the SE block in the network. Studies have shown that the performance of the network is stable to a certain range of deceleration, that is, reducing the value of r1 will increase the complexity of the network but will not monotonically increase the performance. Therefore, after considering the trade-off, r1 is set to 16.

[0036] Last activating factor ic is used to recalibrate the input features Q i The output of the module is in F scale (q ic, s ic ) refers to the activation factor s ic and feature map q ic Channel-wise element-wise multiplication.

[0037] Step 3: Temporal multi-scale attention module

[0038] After the channel feature extraction in step 2, another lightweight model Selective and Kernel Network (SKNet) is used to generate spatial and temporal separable features. SKNet enables the network to adaptively adjust the size of the receptive field based on multi-scale input information. Figure 5 As shown in the figure, the module is divided into three steps: separation, fusion, and selection.

[0039] Separation: The separation operation is to use different sizes of convolution kernels to generate different feature maps. This embodiment divides the input features into three different branches, using two-dimensional convolution kernels of sizes C*5, C*10, and C*20 respectively, where C represents the number of EEG channels. Three different branches are formed by using transformation methods corresponding to three different convolution kernels: Each transformation consists of a convolutional layer, a batch normalization layer, and a ReLU function, and C′×T′ represents the size of the feature map after transformation.

[0040] Fusion: The fusion operation integrates information from multiple branches, allowing neurons to adaptively adjust the size of their receptive fields based on the input content, and allowing multiple branches carrying information of different scales to flow into the next neuron. First, the separated feature maps are combined by element-by-element summation: Similar to step 2, a global average pooling operation is then used to compress the global information into channel descriptors. In order to adaptively guide the selection of convolution kernel size, a fully connected layer is used to generate compact features.

[0041]

[0042] Among them, F fc () indicates a full connection operation, is the batch normalization operation, W3 is the parameter of the fully connected layer, and r2 is the decay factor.

[0043] Selection: According to the feature descriptor S i , using the soft attention mechanism to adaptively select spatial information of different scales, applying the Softmax function to compact features, and generating the weight factors α of the three branches i , β i , γ i , for each channel c, αic +β ic +γ ic =1, so the final output feature V of this module i ={V i1 , V i2 ,…,V iC},in

[0044]

[0045] Step 4: Deep extraction module based on spatiotemporal features

[0046] This module uses a compact convolutional neural module to further extract deep temporal and spatial features, which is divided into two parts. Figure 6 As shown in the figure, in the first block, a convolution operation is performed sequentially, and a 1*64 two-dimensional convolution kernel is used to learn the time-frequency features, followed by a batch normalization operation and an ELU activation function. A 1*4 average pooling layer is used to reduce the time dimension features, and the Dropout method is used to regularize the model to prevent overfitting.

[0047] In the second block, the separable convolution operation is used, which combines the channel-by-channel convolution and point-by-point convolution. In the channel-by-channel convolution, a convolution kernel is responsible for one channel, and a channel is convolved by only one convolution kernel. A 1*16 two-dimensional convolution kernel is used here, and a 1*1 convolution kernel is used for point-by-point convolution. The feature map is weighted and summed in the depth direction to generate a new feature map. The advantage of this is that it reduces the number of fitting parameters and can explicitly decouple the relationship between the feature maps.

[0048] Step 5: Classification

[0049] After the above modules extract valid information, a fully connected layer and a softmax activation function are used for classification. The output information is compared with the category label of the aircraft target positioning direction to obtain the final classification result. The following table shows the effect of using this model to classify data.

[0050]

[0051] This embodiment uses three indicators to measure the performance of the model. All sample data are divided into positive samples P and negative samples N. Therefore, the prediction results of the model have the following four situations: (1) TP: positive samples are predicted as positive samples; (2) TN: negative samples are predicted as negative samples; (3) FP: negative samples are predicted as positive samples; (4) FN: positive samples are predicted as negative samples.

[0052] P=TP+FN

[0053] N=TN+FP

[0054] ACC stands for accuracy, which is the ratio of correctly predicted samples to the total number of samples. A higher accuracy indicates better model performance. Its calculation formula is as follows:

[0055]

[0056] F-Score stands for F-value, which is the harmonic mean of precision and recall. Precision is a measure of accuracy, indicating the ratio of correctly predicted positive samples to the total number of predicted positive samples. Recall is a measure of coverage, indicating the ratio of correctly predicted positive samples to the total number of true positive samples. These two indicators may conflict with each other. F-value can be used to evaluate algorithms in a balanced way:

[0057]

[0058] BA is a method for calculating balanced accuracy, which can be used to handle imbalanced datasets in binary or multi-class classification problems. It is the sum of the false positive rate (FPR) and the true positive rate (TPR) divided by 2:

[0059]

[0060]

[0061]

[0062] It can be seen from the data in the table that the present application has greatly improved the accuracy, F value and calculation balance accuracy compared with the existing technology.

Claims

1. An EEG target spatial localization method based on spatiotemporal feature extraction of attention mechanism, characterized by: The specific steps of this method are as follows: Step 1: Collect the EEG data of the subjects when they perform the viewing task as sample data, and record the corresponding target position as label data; Step 2: Use SENet to squeeze and extract the sample data obtained in step 1, assign different weights to the information of different channels, calibrate the spatial attention of the channels, and extract the channel features of the sample data; Step 3: Input the channel features obtained in step 2 into SKNet to further extract separable features of space and time; Step 4: For the feature extraction results of step 3, first perform two convolution operations to extract the features of the time dimension and the features of the space dimension in turn, and then perform a separable convolution operation to complete the deep extraction of spatiotemporal features; Step 5: Input the final feature information obtained after steps 2 to 4 into the linear classification layer, compare the output information with the label data recorded in step 1, and update the model parameters; then input the EEG data with unknown labels into the updated model to complete the target space positioning task.

2. The method for EEG target spatial localization based on spatiotemporal feature extraction of attention mechanism as claimed in claim 1, characterized in that: In step 1, the rapid serial visual presentation paradigm in the brain-computer interface is used to collect EEG data of the subjects when they are performing the viewing task.

3. The method for EEG target spatial localization based on spatiotemporal feature extraction using an attention mechanism as claimed in claim 1 or 2, characterized in that: During the viewing task, the subjects were presented with five types of images, including images without a target and images in which the targets were located in the four quadrants of a two-dimensional rectangular coordinate system.

4. The method for EEG target spatial localization based on spatiotemporal feature extraction using an attention mechanism as claimed in claim 1 or 2, characterized in that: The collected EEG data is processed by Butterworth bandpass filtering to retain the signal in the range of 0.1 to 40 Hz, and then downsampled. Finally, the EEG data before the image appears is used to calibrate the baseline of the data after the image appears, and then divided into n time segments. Where C represents the number of EEG electrodes, and T represents the number of time points in each time segment.

5. The method for EEG target spatial localization based on spatiotemporal feature extraction of attention mechanism as claimed in claim 1, characterized in that: In step 2, the sample data is first subjected to a global average pooling layer to generate a channel descriptor: Among them, F sq () represents the global average pooling operation, represents the descriptor of the cth channel in the i-th sample, represents the signal of the cth channel in the i-th sample, q ic (j) represents the jth time point of the cth channel in the i-th sample; then the activation factor is generated through two fully connected layers: s ic =F ex (z,W)=σ(g(z,W))=σ(W2δ(W1z) F ex () represents two fully connected operations, which are used to learn the correlation between channels, δ() represents the ReLU function, and σ() represents the sigmoid activation function. are the parameters of the two fully connected layers, and the decay rate r1 is also a hyperparameter in this network; finally, the activation factor s is used ic Perform channel-based element multiplication with the input sample data to obtain the channel features of the sample data.

6. The method for EEG target spatial localization based on spatiotemporal feature extraction of attention mechanism as claimed in claim 5, characterized in that: Set the decay rate r1=16.

7. The method for EEG target spatial localization based on spatiotemporal feature extraction of attention mechanism as claimed in claim 1, characterized in that: In step 3, SKNet is used to separate, fuse and select the channel features of the sample data in sequence; The separation operation divides the input features into three different branches. Each branch includes a cascade of a convolution layer, a batch normalization layer, and a ReLU function. The sizes of the convolution kernels in the three branches are two-dimensional convolution kernels of size C*5, C*10, and C*20, respectively, where C represents the number of channels of the sample data. The fusion operation combines the outputs of the three branches by element-by-element summation. Together, they are compressed into channel descriptors through a global average pooling operation. Then use the fully connected layer to generate compact features Among them F fc () indicates a full connection operation, is the batch normalization operation, W3 is the parameter of the fully connected layer, r2 is the decay factor, and σ() represents the sigmoid activation function; The selection operation is based on the feature descriptor S i , using the soft attention mechanism to adaptively select spatial information of different scales, applying the Softmax function to compact features, and generating the weight factors α of the three branches i , β i , γ i , for each channel c, α ic +β ic +γ ic =1, output spatiotemporal features V i ={V i1 ,V i2 ,…,V iC },in 8. The method for EEG target spatial localization based on spatiotemporal feature extraction of attention mechanism as claimed in claim 1, characterized in that: In step 4, a 1*64 two-dimensional convolution kernel is first used to learn the time-frequency features, followed by batch normalization and ELU activation function, and a 1*4 average pooling layer is used to reduce the time dimension features. The Dropout method is used to regularize the model to prevent overfitting; then a separable convolution operation is used, including a 1*16 channel-by-channel convolution and a 1*1 point-by-point convolution.

Images