A RSVP-like imbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning

Abstract

The present invention proposes a RSVP class imbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupled learning, aiming to improve the feature extraction and classification performance of RSVP EEG signals. First, the collected RSVP EEG signals are preprocessed, including common mean reference, bandpass filtering and normalization processing, and the training set and test set are divided. Then, multi-scale temporal and spatial features are extracted through parallel convolution layers and spatial convolution layers, and features at different levels are fused using interpolation method. The training process is decoupled into representation learning and classifier learning. Representation learning uses a triplet loss function based on difficulty sample sampling, and classifier learning uses optimized Focal Loss for feature learning and classifier optimization. The adaptive channel hybrid attention mechanism is introduced, and the flexibility and robustness of the model are enhanced through dynamic weighting and multi-level feature fusion. This method solves the category imbalance problem in the RSVP task.

Description

Technical Field

[0001] The present invention discloses a RSVP-type unbalanced EEG signal classification method based on an adaptive channel hybrid attention mechanism and decoupling learning, which can be used for EEG signal decoding in rapid sequence visual presentation tasks and belongs to the intersection of computer and biomedical engineering. Background Art

[0002] A brain-computer interface (BCI) is a communication system that controls a computer or external device by acquiring electroencephalogram (EEG) signals or other physiological signals related to human brain activity. RSVP-BCI is a specific BCI paradigm that detects a specific waveform induced by EEG signals, namely the event-related potential (ERP), to identify the P300 component of a target image. In a rapid serial visual presentation (RSVP-BCI) BCI system, letters or images are typically flashed on a screen at a fixed frequency. Subjects are asked to view these image sequences and count the number of times a target image (a specific type of picture) appears. The onset of the target stimulus triggers a P300 event-related potential (ERP), which is time-locked to the onset of the stimulus and has a fixed phase. By performing real-time classification of scalp-recorded EEG signals, RSVP-BCI has been widely used in target search and interactive control tasks, such as video surveillance and character spelling. When subjects observe a target image, a P300 component is triggered, allowing them to identify the target image through decoding. The RSVP task distinguishes target and non-target stimuli by extracting ERP features, so the detection of ERP components plays a key role in RSVP-BCI classification. The RSVP-BCI classification framework includes preprocessing, feature extraction, and classification, where effective feature extraction significantly improves classification accuracy. However, RSVP EEG signals suffer from class imbalance, with target images appearing much less frequently than non-target images, which affects classification performance.

[0003] In recent years, many studies have focused on feature extraction methods to improve the performance of RSVP-BCI. In terms of machine learning algorithms, Sajda et al. proposed a spatiotemporal mixed feature extraction method called Hierarchical Discriminant Component Analysis (HDCA), which extracts spatial features through Fisher linear discriminant (FLD) and uses logistic regression to obtain temporal features. Alpert et al. proposed a spatially weighted FLD-principal component analysis (SWFP) method, combining FLD and PCA to extract spatiotemporal mixed features. In addition, Alpert et al. developed an improved version of HDCA, called hierarchical discriminant principal component analysis (HDPCA), which uses PCA for dimensionality reduction in the temporal dimension. Xie et al. proposed a filter bank spatiotemporal component analysis (FBSCA) method, which decomposes EEG data in the gamma frequency band in the time-frequency space domain and extracts spatiotemporal features to improve the classification performance of the RSVP task. Xiao et al. proposed a discriminative canonical pattern matching method that can effectively detect small ERPs even with a small training set. Cui et al. designed a spatiotemporal hybrid common spatial pattern (CSP)-PCA algorithm to decode EEG signals in the RSVP task. This algorithm used CSP to extract spatial features and PCA to obtain temporal features. Variant methods based on spatiotemporal hybrid feature extraction have also contributed to improving the performance of RSVP-BCIs. Unlike machine learning algorithms, deep learning can automatically extract discriminative data representations from EEG recordings of brain activity. For example, Cecotti et al. used a convolutional neural network (CNN) to adaptively detect the P300 wave in the time domain, tested it on a P300 spelling task, and demonstrated improved performance. Later, Cecotti et al. proposed a supervised spatial filtering method in another study to enhance discriminative information in EEG data. They proposed a CNN model containing a specialized spatial filtering layer to detect ERPs and trained it based on maximizing the area under the receiver operating characteristic curve (AUC). Schirrmeister et al. proposed a CNN model called DeepConvNet for EEG decoding tasks in BCIs. The model consists of five convolutional layers and a softmax layer for classification. Lawhern et al. proposed a compact neural network, named EEGNet, for EEG-based BCI, and achieved remarkable performance in various EEG classification tasks. Vázquez et al. proposed EEG-Inception, which integrates the Inception module to effectively extract temporal features at different time scales for ERP classification.

[0004] The key to implementing RSVP-based BCI is to effectively classify the EEG signals generated during the RSVP task. However, the performance of current methods for RSVP EEG classification remains unsatisfactory. The literature suggests that the following two issues need to be addressed:

[0005] 1. The problem of class imbalance is most common in the general RSVP paradigm. During the visual presentation process, a sufficient time interval needs to be maintained between two adjacent target images to induce a P300 response. The number of non-target samples far exceeds the number of target samples in the paradigm design. This class imbalance problem seriously affects the performance of RSVP EEG signal classification because the majority class dominates, resulting in a high false positive rate.

[0006] 2. In brain-computer interface decoding research, rapid fluctuations in physiological signals, artifacts, and individual variability are key factors that reduce EEG signal classification performance. Deep learning methods focus on extracting key features from these signals to provide better decision-making. However, fully utilizing the key information in RSVP EEG signals remains challenging.

[0007] To address the first issue, Li et al. recently proposed a novel decoupled representation learning model based on multi-granularity information to eliminate the impact of class imbalance and extract multi-granularity EEG information. This model training process is decomposed into two steps: representation learning and classifier training. This approach captures the discriminative features of imbalanced RSVP EEG data while maintaining classification performance. First, the preprocessed EEG dataset is constructed into positive and negative sample pairs. Using a two-branch architecture, the contrastive loss is minimized by reducing the cosine similarity of positive pairs and increasing the cosine similarity of negative pairs in the representation space. For imbalanced datasets, the representation learning process generally has little effect on distinguishing target from non-target samples. However, for classifiers, training on highly imbalanced datasets can result in high classification accuracy but also extremely high false alarm rates. This is because most target samples are incorrectly classified, with many being misclassified as non-target samples. The model gains little benefit from easily classified samples, and should instead focus on difficult samples, regardless of their class. However, the combined contribution of a large number of easy samples can overshadow the contribution of difficult samples, resulting in low training efficiency. For general RSVP tasks, the class imbalance problem is very significant and warrants further consideration. Li et al.'s strategy discards many non-target samples that may be valuable for classification. Furthermore, undersampling fails to fit the entire training data distribution, leading to inconsistent data distribution within the imbalanced test set. Therefore, addressing the class imbalance problem in RSVP tasks requires exploring more effective solutions.

[0008] For the second question, Xiao et al. used attention-based bidirectional LSTM to explore the importance of different time slices. Different emotional states show differences in the activation levels of different frequency bands, time points, and brain regions. In specific frequency bands, time points, or brain regions, each emotional state has some stable and discriminative patterns.

[42] Jia et al.

[43] A parallel spatiotemporal attention module is designed to adaptively capture some brain regions, frequency bands, and time points of interest. Similarly, spatial attention is used to identify valuable brain regions to improve the accuracy of emotion recognition. To construct spatial attention, global average pooling (bGAP) in the frequency band dimension is used to reduce the frequency band dimension. However, there are still two challenging problems that need to be solved. One is how to effectively obtain and utilize the spatial information of feature maps of different scales to enrich the feature space. The second is that channel attention or spatial attention can only effectively capture local information and cannot obtain more representative features. Although some of the latest methods can effectively solve the above problems, they also bring huge computational burdens.

[0009] By discussing and analyzing the advantages and disadvantages of the above existing methods, a RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and multi-task collaboration is proposed. Compared with the classic EEGNet and DRL models, the classification method proposed in this paper can more effectively improve the decoding performance of RSVP EEG signals. Summary of the Invention

[0010] The present invention proposes a RSVP class imbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupled learning, which can effectively improve the feature extraction and classification performance of RSVP EEG signals. In order to solve the problem of extremely unbalanced EEG signal categories, the representation learning and classifier learning stages of the decoupled model are used. In the representation learning stage, a triplet loss based on difficult sample sampling is introduced, and in the classifier learning stage, an optimized FocalLoss function is used to improve the processing ability of difficult-to-classify samples and imbalanced categories by introducing sample difficulty weights and category weights. At the same time, an adaptive channel hybrid attention mechanism is used in feature extraction to extract more distinctive spatiotemporal features. Finally, the trained classifier is used to classify unknown samples. Compared with models such as EEGNet and DRL, the indicators such as AUC and RECALL in the method proposed by the present invention have been significantly improved.

[0011] To achieve the above invention objectives, after research, discussion and repeated practice, the final solution of this method is as follows:

[0012] First, the original RSVP EEG dataset is preprocessed, and then the dataset is divided into a training set and a test set. The two parts are input into the constructed ACMA-DL model for training and testing respectively. Finally, the model classification results are obtained and evaluated to verify the effectiveness of the method.

[0013] The specific steps of the technical solution of the present invention are as follows:

[0014] Step 1: Data preprocessing and dataset partitioning: Redundant channels were removed from the raw data channels, and each EEG block was divided into different trials. For each trial, the data was normalized to zero mean and unit variance. The signal was bandpass filtered using a bandpass filter, and the EEG signal dataset was divided into training and test sets.

[0015] Step 2: Construct an ACMA-DL model. For the RSVP-based EEG signal classification model using an adaptive channel mixing attention mechanism and decoupled learning (ACMA-DL), multi-scale temporal features are extracted through parallel convolutional layers. These convolutional layers use different time windows and filters to capture the various temporal variations in the EEG signal. Spatial convolutional layers are then used to extract spatial features from the signal, ensuring that the spatial relationships of the EEG signal are captured. Interpolation is then used to fuse features from different scales, improving the expressive power of features through multi-level feature fusion. During the fusion stage, the model applies an adaptive channel mixing attention mechanism to weight the fused features and enhance the representation of important features.

[0016] In step 3, the training process is decoupled into two phases. In the representation learning phase, a difficulty sampling method is used to select negative samples. Specifically, for each anchor sample and positive sample, the cosine similarity between all candidate negative samples and the anchor sample is calculated, and the negative sample with the highest similarity, known as the "hardest" negative sample, is selected. This method allows for more effective optimization of the triplet loss, as selecting the hardest negative sample can better widen the gap between positive and negative samples, thereby improving the model's discriminative ability. The training set constructed in step 1 for the representation learning phase is input into ACMA-DL for training. In the classifier learning phase, the feature extractor trained in the representation learning phase is loaded, and the classifier is optimized using the optimized Focal loss function. This addresses class imbalance by adjusting the prediction difficulty. By reducing the weight of easily classified samples and increasing the weight of difficult samples, the model focuses more on difficult samples during training, thereby improving its ability to recognize the minority class. This joint training improves classification performance and ensures that the classifier accurately utilizes the extracted features. The training set for the classifier learning phase constructed in step 1 is input into ACMA-DL for training.

[0017] Step 4. In the test and evaluation phase, use the trained model to perform predictions and evaluations on the test dataset constructed in Step 1. Indicators include accuracy, recall, precision, and F1 score to comprehensively measure model performance and determine its effectiveness and robustness in practical applications.

[0018] The present invention has the following advantages:

[0019] 1. Compared to networks with single-scale convolutional layers, networks using parallel multi-scale temporal and spatial convolutional layers can simultaneously extract both local and long-range temporal and spatial features from EEG signals. Through multiple Gaussian filtering and downsampling, these networks capture signal features at different time scales, achieving better accuracy and efficiency, further improving various metrics for RSVP classification tasks.

[0020] 2. By decoupling the representation learning and classification learning stages, and using triple loss combined with difficulty sampling in the representation learning stage and Focalloss combined with sample difficulty weight and category weight in the classifier learning stage to improve the processing ability of difficult-to-classify samples and imbalanced categories, it not only enhances the feature expression ability of positive and negative samples, but also effectively alleviates the class imbalance problem of the RSVP dataset.

[0021] 3. The introduction of an adaptive channel-wise hybrid attention mechanism effectively integrates spatial and channel-wise information, capturing long-range dependencies and global contextual features to enhance the representation of important features in the input signal. Furthermore, this mechanism improves the model's flexibility and robustness through dynamic weighting and multi-level feature fusion, enabling better adaptation to diverse input signal characteristics and enhancing overall classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Overall flow chart of the present invention

[0023] Figure 2 ACMA-DL network structure diagram

[0024] Figure 3 Attention module diagram DETAILED DESCRIPTION

[0025] This method aims to address and classify the class imbalance problem in rapid serial visual presentation (RSVP) EEG signals. It employs a deep learning model with an adaptive channel-by-channel hybrid attention mechanism and decoupled learning. The decoupled learning strategy ensures the effectiveness of feature representation, while the adaptive channel-by-channel hybrid attention mechanism enhances the discriminative power of features. The combination of the two improves EEG classification performance. Furthermore, a triplet loss based on sample difficulty sampling is introduced in the representation learning phase of the decoupled learning strategy to learn the features of different samples. A focal loss based on sample difficulty and class weights is introduced in the classifier learning phase to mitigate the class imbalance problem. The entire process of this method includes four main steps: data preprocessing and dataset partitioning, model construction, training, and testing. Experimental verification shows that this method not only effectively addresses the imbalance problem in EEG signals, but also improves metrics such as classification recall and model robustness. Figure 1 The overall flow chart of the present invention can be decomposed into the following steps:

[0026] Step 1: Data preprocessing and data set division.

[0027] Step 2: Build the ACMA-DL model.

[0028] Step 3: Use the training set to train the model.

[0029] Step 4: Test the model effect and evaluate indicators such as Recall.

[0030] The following is a detailed description of each step:

[0031] Step 1 is as follows:

[0032] (1) Two channels, EOG1 and EOG2, were removed from the 64-channel original data;

[0033] (2) Extracting 0.1-30 Hz EEG signals through a 4th-order Butterworth bandpass filter;

[0034] (3) Each EEG signal block was divided into different trials, and for each trial, the data were normalized to zero mean and unit variance;

[0035] (4) The EEG signal is segmented and selected, each segment represents a complete task, and the length of each segment of the EEG signal is 1 s;

[0036] (5) For the training dataset representing the learning phase, 40 triplets are generated for each positive sample, each of which includes an anchor sample, a positive sample, and multiple negative samples;

[0037] (6) For the training data set in the classifier learning phase, all training samples are used for training;

[0038] (7) For the training data set in the test phase, all test samples are used for testing;

[0039] Step 2 is as follows:

[0040] The ACMA-DL model consists of four parts: parallel convolutional layers, spatial convolutional layers, feature fusion layers, and an adaptive channel-by-channel attention mechanism. The model was built using PyTorch. The following is a detailed description of each part:

[0041] (1) Parallel convolutional layer

[0042] In the parallel convolutional layer, four parallel convolutional layers are used to extract multi-scale temporal features. These convolutional layers use the same time window setting to perform preliminary temporal feature extraction on the input EEG signal, capturing temporal variations within it. To ensure consistent output dimensions after the convolution operation during multi-scale temporal feature extraction, zero padding is applied to each convolutional layer. Specifically, each convolutional layer performs zero padding of kernel_length / 2 on both the left and right sides, with kernel_length set to 3, to ensure that the convolution operation does not alter the temporal dimension of the output feature map. This strategy ensures that the convolution output maintains the same temporal dimension as the input, which is crucial for subsequent feature processing. Experiments have shown that the optimal convolution kernel size is (1,3), which uses three time points as the convolution window in the temporal dimension, ensuring the extraction of sufficiently detailed temporal information. The default stride is 1, meaning that the convolution operation slides across time points, ensuring fine-grained feature extraction. The number of output channels for the convolution operation is set to 8, a configuration that has been experimentally verified as the optimal value. After the convolution operation, batch normalization is used to standardize the convolution output. The introduction of batch normalization can significantly prevent the occurrence of vanishing or exploding gradients during training, thereby accelerating model convergence and improving training stability. Batch normalization is performed after each convolutional layer to ensure that the multi-scale features extracted in each layer are trained within the same numerical range.

[0043] (2) Spatial convolution layer

[0044] In the spatial convolution layer, four parallel convolutional layers are used to extract multi-scale spatial features. Similar to temporal feature extraction, spatial feature extraction requires the use of different time windows and filters to capture the various spatial variations in the input EEG signal. In this step, the convolution operation targets the spatial dimension of the EEG signal to ensure that spatial information is extracted. After multiple experiments, the optimal parallel structure was determined as follows: First, the input signal is processed through a convolutional layer. The convolution kernel size is set to (62, 1), where 62 represents the number of channels and 1 represents the temporal dimension. This setting ensures that the convolution operation fully covers all channels in the spatial dimension, ensuring that the spatial features of the signal are fully extracted. The stride defaults to 1, meaning that the convolution operation slides channel by channel without skipping any spatial information. The number of input and output channels is set to 8, which has been verified as the optimal value through multiple experiments. To further improve computational efficiency, the model uses depthwise convolution, using groups = 8, so that each input channel is independently convolved with the corresponding output channel. This operation reduces the number of parameters and computational cost, making the model more efficient when processing large-scale EEG signals. After the convolution operation, batch normalization (Batch Normalization) is used to standardize the convolution output, ensuring that the output of the convolution layer remains stable within a numerical range, thereby avoiding vanishing or exploding gradients and accelerating the model training process. Batch normalization is performed after each convolutional layer to ensure the stability of the training process. Next, the ELU (Exponential Linear Unit) activation function is used to introduce nonlinearity to help the model alleviate the vanishing gradient problem. Compared with other activation functions, the ELU can better handle vanishing gradients, especially in deep networks, ensuring that the model better fits complex data distributions. Finally, a Dropout layer is added with a dropout ratio set to 0.6. The Dropout layer prevents model overfitting by randomly dropping some neurons, improving the model's generalization ability. Experimental verification has shown that this dropout ratio can reduce model overfitting while maintaining sufficient expressive power.

[0045] (3) Feature fusion layer

[0046] The purpose of the feature fusion layer is to fuse features from different layers to improve the model's expressiveness and its ability to capture multi-scale features. First, the input to the feature fusion layer is a list of four feature maps of different sizes, output from different convolutional layers, representing feature information at different scales and levels. Because these feature maps have different sizes due to different downsampling and convolution operations, they must be resized to a consistent size for effective feature fusion.

[0047] To achieve this goal, we first define the target size. Typically, the highest-resolution feature map size is chosen as the target size. In this dataset, the feature maps with the highest granularity in the temporal dimension (64, 8, 1, 31) were selected as the target size. The torch.nn.functional.interpolate function is then used to interpolate the feature maps, using bilinear interpolation (mode = 'bilinear') to smoothly resize feature maps of different scales to the same size. During interpolation, the parameter align_corners = False can prevent alignment issues at feature map boundaries, ensuring consistency across feature map boundaries and improving the quality of the interpolated feature maps.

[0048] After interpolation and adjustment, all feature maps are converted to the same target size, forming a list of feature maps of consistent size. This provides the foundation for subsequent feature fusion operations, allowing each feature map to be operated on pixel by pixel in the same spatial dimensions. This process ensures that information between feature maps is not lost, improves the expressiveness of the fused features, and supports the model's ability to capture multi-scale features. After feature fusion, weighted fusion or convolution can be used to further enhance feature complementarity and improve the model's classification performance.

[0049] (4) Adaptive Hybrid Attention Mechanism

[0050] In the adaptive channel-wise hybrid attention mechanism, input features are first processed using a global context branch. Specifically, feature maps are pooled to a size of (1,1) using global average pooling (GAP). This operation generates global context information for the entire input feature map by globally averaging all spatial locations. This global context captures global semantic information, thereby better understanding the overall distribution of features. The pooled features are then further extracted using two one-dimensional convolutional layers with a kernel size of 1 and 8 channels. These layers are then activated using a ReLU function to generate the final global context features. This design ensures that information at the global context level is fully extracted and used for subsequent processing. The adaptive channel-wise hybrid attention mechanism also includes a self-attention branch. This branch generates query, key, and value vectors using three one-dimensional convolutional layers. The query and key vectors have an output channel of 1, while the value vector has an output channel of 8. Attention weights are calculated by calculating the similarity between the query and key, and these weights are then applied to the value vector. The attention calculation formula is:

[0051]

[0052] Where Q and K are query and key vectors respectively, d k is the dimension of the key vector. This formula computes the dot product of the query and the key, obtains the attention distribution through a soft maximization operation, and finally applies it to the value vector to generate self-attention features. Next, the self-attention features and global context features are weighted together using a trainable parameter gamma. Gamma is a scalar parameter that controls the relative importance of the self-attention features and global context features in the final output. The combined feature map is resized to (1,1) using bilinear interpolation to ensure consistent spatial dimensions and prepare suitable input for subsequent processing steps. Furthermore, the channel attention branch generates global features of the input feature map through global average pooling and global max pooling. Global average pooling and global max pooling capture the average information and maximum response value of the input feature map, respectively. These two pooling operations feed the feature map into a fully connected layer for further processing to generate channel attention weights. The fully connected layer has 8 intermediate channels and uses the ReLU activation function to enhance nonlinear representation. Finally, the generated channel attention weights are multiplied channel-by-channel with the original feature map to enhance the representation of important channels, thereby causing the model to focus more on features that contribute to classification.

[0053] Step 3 is as follows:

[0054] In the representation learning stage, the ACMA-DL model uses the triplet loss function (Triplet Loss) for feature learning. Each positive sample is paired with multiple positive samples and negative samples to generate 40 triplets. In each training cycle, the model extracts anchor samples, positive samples, and multiple negative samples from the training set and loads them onto the device. 64 segments of EEG signals are input into the network for training each time, and iterates 30 rounds. In order to select the most difficult negative sample, the model calculates the cosine similarity between each negative sample and the anchor sample, and selects the negative sample with the highest similarity as the most difficult negative sample. The triplet loss function is then used to calculate the loss, and the model parameters are optimized by backpropagation and gradient descent. Cosine Similarity (a, n) is used to calculate the similarity between two vectors a (anchor sample) and n (negative sample), and the formula is as follows:

[0055]

[0056] Where a·n represents the dot product of the anchor sample a and the negative sample n, ||a|| and ||n|| are the modulo (i.e., vector length) of vectors a and n respectively. Given multiple negative samples n1, n2, ..., n k , the model calculates the cosine similarity between each negative sample and the anchor sample and selects the negative sample with the highest similarity. It can be expressed as the following formula:

[0057]

[0058] Then, the hardest negative sample and the positive sample are substituted into the triplet loss function together, and the formula is as follows:

[0059] L(a,p,n)=max(0,||f(a)-f(p)|| 2 -||f(a)-f(n)|| 2 +margin)

[0060] where f(a), f(p), and F(n) represent the feature representations of anchor samples, positive samples, and negative samples, respectively. Margin is a hyperparameter used to ensure that the distance between negative samples and anchor samples is at least a constant greater than the distance between positive samples and anchor samples.

[0061] At the end of each epoch, the average loss is calculated. If the current loss is lower than the optimal loss, the model is saved. This method effectively learns the feature distances between samples of different categories, making the feature distances between samples of the same category closer and the feature distances between samples of different categories farther apart, thereby improving feature representation and classification performance. The Adam optimizer is used during network training to mitigate gradient oscillation, and the learning rate is set to 0.001.

[0062] During the classifier learning phase, the ACMA-DL model loads the feature extractor trained during the learning phase and performs further optimization on the classifier. First, it starts by loading the pre-trained feature extractor. Class weights are calculated by combining all labels, and the optimized Focal Loss function is used to address the problem of inter-class imbalance. The optimized Focal Loss function improves the ability to handle difficult-to-classify samples and imbalanced classes by introducing sample difficulty weights and class weights. Class weights are calculated based on the number of samples in each class. A larger weight value indicates a smaller number of samples in that class. The class weight formula is as follows:

[0063]

[0064] Where N is the total number of samples and ni is the number of samples in category i. The sample difficulty weight combines factors such as the entropy of the sample's predicted probability, the probability variance, and the predicted margin. The difficulty weight formula is as follows:

[0065] Difficulty Weight=softmax(Entropy+Variance Impact-Margin Impact)+1

[0066] The formula weights three factors (Entropy, Variance Impact, and Margin Impact) and uses a softmax function to calculate the relative difficulty weight of each sample. The result is added by 1 to ensure that the weight will not be negative.

[0067] Entropy=-∑(Probs·log(Probs+1e-10))

[0068] Entropy is used to measure classification uncertainty. The greater the entropy, the less certain the model is about the classification of the sample. Probs represents the predicted probability distribution for each class. This formula calculates the uncertainty of the sample classification by multiplying the probability of each class by its logarithm and summing the results. 1e-10 is added to prevent encountering zero probabilities when taking the logarithm.

[0069] Variance Impact=sigmoid(Variance×10)

[0070] Variance represents the variance of the probability distribution of the model output, and 10 is a scaling factor to enhance the impact of variance on the results.

[0071] MarginImpact=1-sigmoid(Margin×10)

[0072] Margin represents the difference between the first two predicted probabilities. This difference is mapped to the range (0, 1) using the sigmoid function, and the result is subtracted from 1 to reflect that smaller margins increase classification difficulty. The value is multiplied by 10 to amplify this effect.

[0073] Margin=Top Two Probs[0]-Top Two Probs[1]

[0074] Among them, Top Two Probs[0] is the model's predicted probability for the highest probability class, and Top Two Probs[1] is the predicted probability for the second highest probability class. The larger this difference is, the more confident the model is that its classification is correct and the lower the difficulty is. The final Focal Loss is calculated by combining the class weight and the sample difficulty weight, and its formula is:

[0075] FocalLoss = α·(1-p t ) γ ·CELoss weighted

[0076] CELoss weighted=CELoss×Difficulty Weight×Class Weight

[0077] Among them, CELoss is the cross entropy loss, which is used to calculate the difference between the model prediction result and the true label. α is a parameter that controls sample imbalance and its value is 0.25. γ is a parameter that controls the influence of difficulty weight and its value is 2.0.

[0078] p t =exp(-CE Loss weighted )

[0079] Among them, p t This is a probability value calculated based on weighted cross-entropy loss, which is used to measure the model's confidence in the prediction of a particular sample. During the training cycle, samples from the dataset are loaded onto the device, and 128 EEG signals are fed into the network for training at a time. For 60 iterations, forward propagation is performed to calculate the output logits. Then, the loss is calculated and backpropagation is performed to update the model parameters. After each cycle, the average loss and classification accuracy are calculated and recorded. If the average loss of the current cycle is lower than the previous minimum loss, the current model state is saved. This process optimizes the model's classification performance, ensuring accurate classification using the extracted features. The Adam optimizer is used during network training to mitigate gradient oscillation, and the learning rate is set to 0.001.

[0080] Step 4 is as follows:

[0081] Input the test set in step 1 into the trained model in step 3 for classification and recognition, and evaluate the classification indicators such as AUC, BA, recall rate, F1_score, etc.

[0082] The data sets and experimental results used in the method of the present invention are described as follows:

[0083] 1. Dataset

[0084] The present invention uses the RSVP-based brain-computer interface benchmark dataset provided by Tsinghua University. This dataset contains EEG data from 64 healthy subjects. The data of each subject was collected through 64 electrode channels with a sampling rate of 250Hz. In the experiment, the subjects performed a target image detection task. Each subject performed two sets of experiments, each consisting of two blocks. Each block consisted of 40 trials, and each trial randomly presented 100 stimulus images at a frequency of 10Hz. The stimuli included two categories: target images and non-target images, and the probability of presenting target images was 1% to 4%.

[0085] In this study, 62 electrodes (1–32, 34–42, and 44–64) were selected for further processing. The EEG data were then filtered using a Butterworth filter with a bandwidth of 0.1–30 Hz. After the onset of each image stimulus, a segment spanning 0–1000 ms was extracted and divided into data segments containing 250 sampling points. Thus, the EEG data format for a single trial was structured as a 62-electrode by 250 sampling point matrix.

[0086] The present invention uses the average values ​​of AUC, BA, Recall and F1 of 64 subjects as the performance indicators of the model.

[0087] 2. Experimental Results

[0088] To verify the effectiveness and versatility of the proposed method, we conducted a variety of machine learning experiments on the RSVP brain-computer interface benchmark dataset, involving various classic and traditional models such as HDCA, xDAWN, and XGB-DIM. We also conducted a variety of deep learning experiments, involving various classic and traditional models such as EEGNet, EEG-Inception, and DRL. Six methods were compared with the proposed method, and classification metrics such as AUC, BA, recall, and F1_score were evaluated, with the average results across 64 subjects taken. The table data is as follows:

[0089] Table 1 Comparison indicators of several models

[0090]

[0091] Ultimately, the ACMA-DL method achieved the highest overall performance. It outperformed all evaluation metrics, including classification AUC, balanced precision, recall, and F1_score, significantly surpassing other methods. These results demonstrate that deep learning methods, particularly ACMA-DL, have significant advantages in improving the classification accuracy and stability of the RSVP-BCI system. This research not only validates the effectiveness of the new method but also demonstrates its potential for practical applications.

Claims

1. A RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning, characterized by: The specific steps include: Step 1, data preprocessing and dataset division: First, preprocess the raw EEG signal data, remove unnecessary redundant channels to reduce noise interference, and retain channel information that is useful for the classification task; Secondly, each EEG signal block was divided into different trials, and the signal of each trial was normalized, that is, the signal was transformed to zero mean and unit variance to eliminate the dimensional differences between different signals; then, the EEG signal was filtered using a bandpass filter to retain the frequency components related to the RSVP task and filter out low-frequency and high-frequency noise interference; finally, the preprocessed dataset was divided into training and test sets; Step 2: Construct an ACMA-DL model. For the RSVP-like imbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupled learning (ACMA-DL), the model first extracts multi-scale temporal features through parallel convolutional layers, using different time windows and filter parameters to capture the various temporal variations of EEG signals. By extracting multi-scale temporal features, it captures signal variations in different frequency bands, thereby improving the feature expression capability of the temporal dimension. Next, the model uses spatial convolutional layers to extract spatial features from the signal, ensuring that the spatial dependencies between brain regions are captured in multi-electrode EEG signals. By extracting spatial features, the model can learn the relationships between different electrode channels and improve its ability to model the spatial characteristics of EEG signals. After extracting multi-scale features, the model uses interpolation to fuse temporal and spatial features from different scales, improving the expressiveness and generalization capabilities of features through multi-level feature fusion. Finally, the model introduces an adaptive channel hybrid attention mechanism to weight the fused multi-scale features and dynamically adjust the weights of each channel. In step 3, the training process is decoupled into two phases. In the representation learning phase, a triplet loss optimization strategy based on difficult samples is used. This strategy calculates the similarity between each anchor sample and its positive sample and selects the negative sample with the highest cosine similarity, i.e., the "hardest" negative sample. In the representation learning phase, the training set constructed in step 1 is used as input to the ACMA-DL model to train the feature extractor. In the classifier learning phase, the feature extractor trained in the representation learning phase is used in combination with the optimized Focal loss function to optimize the classifier. The optimized Focal loss function solves the class imbalance problem by reducing the weight of easy-to-classify samples and increasing the weight of difficult-to-classify samples. In this phase, the training set of the classifier learning phase constructed in step 1 is loaded to train the ACMA-DL model.

2. The RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning as claimed in claim 1, characterized in that: Step 1 is as follows: (1) Two channels, EOG1 and EOG2, were removed from the 64-channel original data; (2) Extracting 0.1-30 Hz EEG signals through a 4th-order Butterworth bandpass filter; (3) Each EEG signal block was divided into different trials, and for each trial, the data were normalized to zero mean and unit variance; (4) The EEG signal is segmented and selected, each segment represents a complete task, and the length of each segment of the EEG signal is 1 s; (5) For the training dataset representing the learning phase, 40 triplets are generated for each positive sample, each of which includes an anchor sample, a positive sample, and multiple negative samples; (6) For the training data set in the classifier learning phase, all training samples are used for training.

3. The RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning as claimed in claim 1, characterized in that: Step 2 is as follows: The specific structure of the ACMA-DL model can be summarized into four parts: parallel convolution layer, spatial convolution layer, feature fusion layer and adaptive channel hybrid attention mechanism. Pytorch is used to build the model. The following is a detailed description of each part: (1) Parallel convolutional layer In the parallel convolutional layer part, four parallel convolutional layers are used to extract multi-scale temporal features. These convolutional layers use the same time window setting to perform preliminary temporal feature extraction on the input EEG signal to capture the temporal changes in the EEG signal. To ensure the consistency of the output size after the convolution operation during the process of extracting multi-scale temporal features, zero padding is applied in each convolutional layer. Specifically, each convolutional layer performs a zero padding operation of kernel_length / 2 on the left and right sides, and kernel_length is set to 3 to ensure that the convolution operation does not change the temporal dimension of the output feature map. The kernel size is set to (1,3), which means that three time points are used as the convolution window in the time dimension. The stride defaults to 1, which means that the convolution operation slides from time point to time point. The number of output channels of the convolution operation is set to 8. After the convolution operation, batch normalization is used to standardize the convolution output. (2) Spatial convolution layer In the spatial convolution layer, four parallel convolution layers are used to extract multi-scale spatial features; The parallel structure is as follows: First, the input signal is processed through the convolution layer; the convolution kernel size is set to (62, 1), where 62 represents the number of channels and 1 is the time dimension; the stride defaults to 1, which means that the convolution operation slides channel by channel without skipping any spatial information; The number of input channels and output channels are both set to 8; After the convolution operation, batch normalization is used to standardize the convolution output to ensure that the output of the convolution layer remains stable within the numerical range, thereby avoiding the gradient vanishing or exploding problem and accelerating the model training process; batch normalization is performed after each convolution layer; Next, we use the ELU (Exponential Linear Unit) activation function to introduce nonlinearity to help the model alleviate the vanishing gradient problem. Finally, we add a Dropout layer with a dropout ratio of 0.

6. (3) Feature fusion layer First, the input to the feature fusion layer is a list of features containing 4 feature maps of different sizes. These feature maps are output from different convolutional layers and represent feature information of different scales and levels. To achieve this goal, we first define the target size and select the feature map size with the highest resolution as the target size. We also select the feature map with the highest granularity in the time dimension (64, 8, 1, 31) as the target size. Then, we use the torch.nn.functional.interpolate function to interpolate the feature map, using bilinear interpolation. After interpolation adjustment, all feature maps will be converted to the same target size to form a list of feature maps of consistent size; (4) Adaptive Hybrid Attention Mechanism In the adaptive channel hybrid attention mechanism, the input features are first processed by the global context branch. Specifically, the feature map is pooled to a size of (1,1) using global average pooling. This operation can generate global context information for the entire input feature map by globally averaging all spatial positions. The pooled features are further extracted through two one-dimensional convolutional layers with a convolution kernel size of 1 and 8 channels, and then activated by the ReLU function to generate the final global context features. At the same time, the adaptive channel hybrid attention mechanism also includes a self-attention branch; This branch generates query, key, and value vectors through three one-dimensional convolutional layers. The number of output channels for query and key vectors is 1, and the number of output channels for value vectors is 8. Attention weights are obtained by calculating the similarity between query and key, and then these weights are applied to the value vector. The attention calculation formula is: Where Q and K are query and key vectors respectively, d k is the dimension of the key vector; Next, the self-attention features and global context features are weightedly combined using the trainable parameter gamma; gamma is a scalar parameter that controls the relative importance of the self-attention features and global context features in the final output; the combined feature map is resized to (1,1) using bilinear interpolation; In addition, the channel attention branch generates global features of the input feature map through global average pooling and global maximum pooling; global average pooling and global maximum pooling capture the average information and maximum response value of the input feature map respectively. The feature map after these two pooling operations is input into the fully connected layer to obtain the channel attention weight; the number of intermediate channels in the fully connected layer is 8, and the ReLU activation function is used to enhance the nonlinear expression ability; the generated channel attention weight will be multiplied with the original feature map channel by channel.

4. The RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning as claimed in claim 1, characterized in that: Step 3 is as follows: In the representation learning phase, the ACMA-DL model uses a triplet loss function for feature learning. Each positive sample is paired with multiple positive and negative samples to generate 40 triplets. In each training cycle, the model extracts anchor samples, positive samples, and multiple negative samples from the training set and loads them onto the device. 64 EEG signals are input into the network for training each time, with 30 iterations. To select the most difficult negative sample, the model calculates the cosine similarity between each negative sample and the anchor sample, and selects the negative sample with the highest similarity as the most difficult negative sample. The triplet loss function is then used to calculate the loss, and the model parameters are optimized through backpropagation and gradient descent. Cosine Similarity(a,n) is used to calculate the similarity between two vectors a, namely the anchor sample and the negative sample n. The formula is as follows: Among them, a·n represents the dot product of anchor sample a and negative sample n, ||a|| and ||n|| are the modulos (i.e., vector lengths) of vectors a and n respectively; given multiple negative samples n1, n2, ..., n k , the model calculates the cosine similarity between each negative sample and the anchor sample and selects the negative sample with the highest similarity, which is expressed as the following formula: Then, the hardest negative sample and the positive sample are substituted into the triplet loss function together, and the formula is as follows: L(a,p,n)=max(0,||f(a)-f(p)|| 2 -||f(a)-f(n)|| 2 +margin) Where f(a), f(p), and f(n) represent the feature representations of anchor samples, positive samples, and negative samples, respectively; margin is a hyperparameter used to ensure that the distance between negative samples and anchor samples is at least a constant greater than the distance between positive samples and anchor samples; At the end of each cycle, the average loss is calculated. If the current loss is lower than the best loss, the model is saved. The Adam optimizer is used to alleviate gradient oscillation during network training, and the learning rate is set to 0.

001. During the classifier learning phase, the ACMA-DL model loads the feature extractor trained in the learning phase and performs further optimization on the classifier. It starts by loading the pre-trained feature extractor. Class weights are calculated by combining all labels, and the optimized Focal Loss function is used to address inter-class imbalance. The optimized Focal Loss function improves the ability to handle difficult-to-classify samples and imbalanced classes by introducing sample difficulty weights and class weights. Class weights are calculated based on the number of samples in each class. A larger weight indicates a smaller number of samples in that class. The class weight formula is as follows: Where N is the total number of samples, n i is the number of samples of category i; The difficulty weight formula is as follows: Difficulty Weight = softmax(Entropy + Variance Impact - Margin Impact) + 1. The formula weights the three factors of Entropy, Variance Impact, and Margin Impact and calculates the relative difficulty weight of each sample through the softmax function. The result is added by 1 to ensure that the weight will not be negative. Entropy=-∑(Probs·log(Probs+1e-10)) Among them, entropy is used to measure the uncertainty of classification; the larger the entropy, the less certain the model is about the classification of this sample; Probs represents the predicted probability distribution of each category; Variance Impact=sigmoid(Variance×10) Among them, Variance represents the variance of the probability distribution of the model output, and 10 is a scaling factor; MarginImpact=1-sigmoid(Margin×10) Margin represents the difference between the first two predicted probabilities. This difference is mapped to (0, 1) using the sigmoid function, and then the result is subtracted from 1. Margin = Top Two Probs[0] - Top Two Probs[1], where Top Two Probs[0] is the model's predicted probability for the highest probability category, and Top Two Probs[1] is the predicted probability for the second highest probability category. The final FocalLoss is calculated by combining the category weight and the sample difficulty weight, and its formula is: FocalLoss=α·(1-p t ) γ ·CELoss weighted CELoss weighted =CELoss×Difficulty Weight×Class Weight Among them, CELoss is the cross entropy loss, which is used to calculate the difference between the model prediction result and the true label. α is a parameter that controls sample imbalance and its value is 0.

25. γ is a parameter that controls the influence of difficulty weight and its value is 2.

0. p t =exp(-CE Loss weighted ) Among them, p t It is a probability value calculated based on the weighted cross entropy loss, which is used to measure the model's prediction confidence on the sample; during the training cycle, the samples in the dataset are loaded onto the device, and 128 EEG signals are input into the network for training each time, iterating 60 rounds, and performing forward propagation to calculate the output logits; then, the loss is calculated and backpropagation is performed to update the model parameters; after each cycle, the average loss and classification accuracy are calculated and recorded; if the average loss of the current cycle is lower than the previous minimum loss, the current model state is saved; during the network training process, the Adam optimizer is used to alleviate gradient oscillation, and the learning rate is set to 0.

001.

5. The RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning as claimed in claim 1, characterized in that: Step 4 is also set, and the specific steps of step 4 are: Input the test set in step 1 into the trained model in step 3 for classification and recognition, and evaluate the classification AUC, BA, recall rate or F1_score.

Images