RSVP type unbalanced electroencephalogram signal classification method based on adaptive channel mixed attention mechanism and decoupling learning

Abstract

The invention provides an RSVP type unbalanced electroencephalogram signal classification method based on a self-adaptive channel mixed attention mechanism and decoupling learning, and aims to improve the feature extraction and classification performance of RSVP electroencephalogram signals. Firstly, collected RSVP electroencephalogram signals are preprocessed, including common average reference, band-pass filtering and standardization processing, and a training set and a test set are divided; then, multi-scale time and space features are extracted through a parallel convolution layer and a space convolution layer, and the features of different levels are fused through an interpolation method; the training process is decoupled into representation learning and classifier learning, the representation learning uses a triple loss function based on difficult sample sampling, and the classifier learning uses the optimized Focal Loss to perform feature learning and classifier optimization. A self-adaptive channel mixed attention mechanism is introduced, and the flexibility and robustness of the model are enhanced through dynamic weighting and multi-level feature fusion. According to the method, the problem of class imbalance in the RSVP task is solved.

Description

Technical Field

[0001] The present invention discloses a RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning, which can be used for EEG signal decoding of rapid sequence visual presentation tasks and belongs to the intersection of computer and biomedical engineering. Background Art

[0002] Brain-Computer Interface (BCI) is a communication system that controls computers or external devices by collecting electroencephalogram (EEG) or other physiological signals related to human brain activity. RSVP-BCI is a specific paradigm of BCI that identifies the P300 component in the target image by detecting special waveforms triggered by EEG signals, namely event-related potentials (ERP). In the rapid serial visual presentation brain-computer interface system (RSVP-BCI), letters or images are usually displayed on the screen in sequence with a fixed frequency. Subjects are asked to watch these image sequences and count the number of times the target image (a specific type of picture) appears. The appearance of the target stimulus triggers the P300 event-related potential, which is time-locked with the onset of the stimulus and has a fixed phase. By real-time classification of EEG signals recorded from the scalp, RSVP-BCI is widely used in target search and interactive control tasks such as video monitoring and character spelling. When the subject observes the target image, the P300 component is triggered, and the target image can be identified through decoding. The RSVP task distinguishes target and non-target stimuli by extracting ERP features, so the detection of ERP components plays an important role in the classification of RSVP-BCI. The classification framework of RSVP-BCI includes preprocessing, feature extraction and classification, among which effective feature extraction significantly improves the classification accuracy. However, there is a class imbalance problem in RSVP EEG signals, and the frequency of target images is much lower than that of non-target images, which affects the classification performance.

[0003] In recent years, many studies have focused on feature extraction methods to improve the performance of RSVP-BCI. In terms of machine learning algorithms, Sajda et al. proposed a spatiotemporal mixed feature extraction method called Hierarchical Discriminant Component Analysis (HDCA), which extracts spatial features through Fisher linear discriminant (FLD) and uses logistic regression to obtain temporal features. Alpert et al. proposed a spatially weighted FLD-principal component analysis (SWFP) method, which combines FLD and PCA to extract spatiotemporal mixed features. In addition, Alpert et al. also developed an improved version of HDCA, called hierarchical discriminant principal component analysis (HDPCA), which uses PCA for dimensionality reduction in the temporal dimension. Xie et al. proposed a filter bank spatiotemporal component analysis (FBSCA) method to decompose EEG data in the γ frequency band in the time-frequency space domain and extract spatiotemporal features to improve the classification performance of the RSVP task. Xiao et al. proposed a discriminative canonical pattern matching method that can effectively detect tiny ERPs even with a small training set. Cui et al. designed a spatiotemporal hybrid common spatial pattern (CSP)-PCA algorithm to decode EEG signals in RSVP tasks, which used CSP to extract spatial features and PCA to obtain temporal features. Variant methods based on spatiotemporal hybrid feature extraction have also contributed to improving the performance of RSVP-BCI. Different from machine learning algorithms, deep learning can automatically extract discriminative data representations from EEG records of brain activity. For example, Cecotti et al. used convolutional neural networks (CNNs) to adaptively detect the P300 wave in the time domain, tested it in the P300 spelling task, and showed improved performance. Later, Cecotti et al. proposed a supervised spatial filtering method in another study to enhance the discriminative information in EEG data. They proposed a CNN model containing a dedicated spatial filtering layer to detect ERPs and trained it based on maximizing the area under the receiver operating characteristic curve (AUC). Schirrmeister et al. proposed a CNN model called DeepConvNet for EEG decoding tasks in BCI, which consists of five convolutional layers and a softmax layer for classification. Lawhern et al. proposed a compact neural network named EEGNet for EEG-based BCI and achieved remarkable performance in various EEG classification tasks. Vázquez et al. proposed EEG-Inception, which integrates the Inception module to effectively extract temporal features at different time scales for ERP classification.

[0004] To realize RSVP-based BCI, the key is to effectively classify the EEG signals generated in the RSVP task. However, the performance of current methods for RSVP EEG classification is still unsatisfactory. The literature shows that the following two problems need to be solved:

[0005] 1. The problem of class imbalance is most common in the general RSVP paradigm. During the visual presentation process, a sufficient time interval needs to be maintained between two adjacent target images to induce a P300 response. The number of non-target samples far exceeds the number of target samples in the paradigm design. This class imbalance problem seriously affects the performance of RSVP EEG signal classification because the majority class dominates, resulting in a high false positive rate.

[0006] 2. In the research of brain-computer interface decoding, rapid fluctuations of physiological signals, artifacts and individual differences are key factors that reduce the classification performance of EEG signals. Deep learning methods focus on how to extract key features from these signals to provide better decisions. However, how to make full use of the key information in RSVP EEG signals remains challenging.

[0007] For the first problem, in recent years, Li et al. proposed a new decoupled representation learning model based on multi-granularity information to eliminate the impact of the class imbalance problem and extract multi-granularity EEG information. The model training process is decomposed into two processes: representation learning and classifier training to capture the discriminative features of unbalanced RSVP EEG data while maintaining classification performance. First, the preprocessed EEG signal dataset is used to construct positive and negative sample pairs. Using a dual-branch architecture, the contrast loss is minimized by reducing the cosine similarity of positive pairs in the representation space and increasing the cosine similarity of negative pairs. For unbalanced datasets, the representation learning process generally has little effect on the distinction between target samples and non-target samples. However, for classifiers, although the classification accuracy is high when trained with highly unbalanced datasets, the false alarm rate of the model is also extremely high. This is because most of the target samples are not correctly classified, and many target samples are judged as non-target samples. The model gains little from samples that are easy to classify. The model should focus on samples that are difficult to classify, regardless of which category it belongs to. However, the contribution of a large number of easy samples will overshadow the difficult samples, making the training efficiency very low. For general RSVP tasks, the class imbalance problem is very obvious and should be considered in depth. The strategy of Li et al. discards many non-target samples that may be valuable for classification. At the same time, undersampling cannot fit the entire training data distribution, resulting in inconsistency in the data distribution in the imbalanced test set. Therefore, solving the class imbalance problem in RSVP tasks requires exploring more effective solutions.

[0008] For the second question, Xiao et al. used attention-based bidirectional LSTM to explore the importance of different time slices. Different emotional states show differences in the activation levels of different frequency bands, time points, and brain regions. In specific frequency bands, time points, or brain regions, each emotional state has some stable and discriminative patterns.

[42] Jia et al.

[43] A parallel spatiotemporal attention module is designed to adaptively capture some brain regions, frequency bands, and time points of interest. Similarly, spatial attention is used to identify valuable brain regions to improve the accuracy of emotion recognition. To construct spatial attention, global average pooling (bGAP) in the frequency band dimension is used to reduce the frequency band dimension. However, there are still two challenging problems to be solved. One is how to effectively obtain and utilize the spatial information of feature maps of different scales to enrich the feature space. The second is that channel attention or spatial attention can only effectively capture local information and cannot obtain more representative features. Although some of the latest methods can effectively solve the above problems, they also bring huge computational burdens.

[0009] By discussing and analyzing the advantages and disadvantages of the above existing methods, a RSVP type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and multi-task collaboration is proposed. Compared with the classic EEGNet and DRL models, the classification method proposed in this paper can more effectively improve the decoding performance of RSVP EEG signals. Summary of the invention

[0010] The present invention proposes a RSVP class imbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupled learning, which can effectively improve the feature extraction and classification performance of RSVP EEG signals. Aiming at the problem of extremely unbalanced EEG signal categories, the representation learning and classifier learning stages of the decoupled model are used to introduce triple loss based on difficult sample sampling in the representation learning stage and use the optimized FocalLoss function in the classifier learning stage to improve the processing ability of difficult-to-classify samples and imbalanced categories by introducing sample difficulty weights and category weights; at the same time, an adaptive channel hybrid attention mechanism is used in feature extraction to extract more distinctive spatiotemporal features; finally, the unknown samples are classified by the trained classifier. Compared with models such as EEGNet and DRL, the indicators such as AUC and RECALL in the method proposed by the present invention are significantly improved.

[0011] In order to achieve the above invention purpose, after research, discussion and repeated practice, the final solution of this method is as follows:

[0012] Firstly, the original RSVP EEG dataset is preprocessed, and then the dataset is divided into two parts: training set and test set. They are input into the constructed ACMA-DL model for training and testing respectively. Finally, the model classification results are obtained and evaluated to verify the effectiveness of the method.

[0013] The specific steps of the technical solution of the present invention are as follows:

[0014] Step 1, data preprocessing and data set division: Remove redundant channels from the original data channels, and divide each EEG block into different trials. For each trial, normalize the data to zero mean and unit variance. Use a bandpass filter to perform bandpass filtering on the signal, and divide the EEG signal data set into a training set and a test set.

[0015] Step 2, build the ACMA-DL model. For the RSVP-based EEG signal classification model using an adaptive channel mixing attention mechanism and decoupled learning (ACMA-DL), multi-scale temporal features are extracted through parallel convolutional layers, which use different time windows and filters to capture various temporal changes in EEG signals. Then, spatial features in the signal are extracted through spatial convolutional layers to ensure that the spatial relationship of EEG signals is captured. After that, interpolation is used to fuse features from different scales, and the expressiveness of features is improved through multi-level feature fusion. In the fusion stage, the model applies an adaptive channel mixing attention mechanism to weight the fused features and enhance the expression of important features.

[0016] Step 3, the training process is decoupled into two stages: In the representation learning stage, a difficulty sampling method is used to select negative samples. Specifically, for each anchor sample and positive sample, the cosine similarity of all candidate negative samples with the anchor sample is calculated, and the negative sample with the highest similarity, that is, the "hardest" negative sample, is selected. With this method, we can optimize the triplet loss more effectively, because selecting the hardest negative sample can better increase the distance between positive and negative samples, thereby improving the model's discrimination ability. The training set for the representation learning stage constructed in step 1 is input into ACMA-DL for training. In the classifier learning stage, the feature extractor trained in the representation learning stage is loaded, and the classifier is optimized using the optimized Focal loss function to solve the class imbalance problem by adjusting the prediction difficulty. It reduces the weight of easy-to-classify samples and increases the weight of difficult-to-classify samples, so that the model can better focus on difficult-to-classify samples during training, thereby improving the recognition ability of minority classes. Improve the classification performance through joint training to ensure that the classifier can accurately use the extracted features for classification. The training set for the classifier learning phase constructed in step 1 is input into ACMA-DL for training.

[0017] Step 4. In the test and evaluation phase, use the trained model to perform predictions and evaluations on the test dataset constructed in step 1. The indicators include accuracy, recall, precision, and F1 score to comprehensively measure the model performance and determine its effectiveness and robustness in practical applications.

[0018] The present invention has the following advantages:

[0019] 1. Compared with the network with single-scale convolution layer, the network with parallel multi-scale temporal convolution layer and spatial convolution layer can extract local and long-range temporal and spatial features of EEG signals at the same time. Through multiple Gaussian filtering and downsampling, the characteristics of the signal at different time scales are captured, which can achieve better accuracy and efficiency, thereby further improving various indicators of RSVP classification tasks.

[0020] 2. By decoupling the representation learning and classification learning stages, and using the triplet loss combined with difficulty sampling in the representation learning stage and the Focalloss combined with sample difficulty weight and category weight in the classifier learning stage to improve the processing ability of difficult-to-classify samples and unbalanced categories, not only the feature expression ability of positive and negative samples is enhanced, but also the class imbalance problem of the RSVP dataset is effectively alleviated.

[0021] 3. Introduce an adaptive channel hybrid attention mechanism module to effectively integrate spatial and channel information, and enhance the expression of important features in the input signal by capturing long-distance dependencies and global context features. In addition, this mechanism improves the flexibility and robustness of the model through dynamic weighting and multi-level feature fusion, which can better adapt to different input signal characteristics and improve overall classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Overall flow chart of the present invention

[0023] Figure 2 ACMA-DL network structure diagram

[0024] Figure 3 Attention module diagram DETAILED DESCRIPTION

[0025] This method aims to solve the class imbalance problem in rapid serial visual presentation (RSVP) EEG signals and classify them. It adopts a deep learning model of adaptive channel hybrid attention mechanism and decoupled learning. The decoupled learning strategy ensures the effectiveness of feature representation, while the adaptive channel hybrid attention mechanism enhances the discriminative ability of features. The combination of the two can improve the classification performance of EEG signals. At the same time, the triple loss based on difficult sample sampling is introduced in the representation learning stage of the decoupled learning strategy to learn the features of different samples. The class imbalance problem of the signal is alleviated by introducing focal loss based on sample difficulty and category weight in the classifier learning stage. The whole process of this method includes four main steps: data preprocessing and data set division, model construction, training and testing. After experimental verification, this method not only effectively handles the imbalance problem in EEG signals, but also improves indicators such as classification recall rate and model robustness. Figure 1 It can be decomposed into the overall flow chart of the present invention, which can be decomposed into the following steps:

[0026] Step 1: Data preprocessing and data set division.

[0027] Step 2: Build the ACMA-DL model.

[0028] Step 3: Use the training set to train the model.

[0029] Step 4: Test the model effect and evaluate indicators such as Recall.

[0030] The following is a detailed description of each step:

[0031] Step 1 is as follows:

[0032] (1) Two channels, EOG1 and EOG2, were removed from the 64-channel original data;

[0033] (2) Extract the 0.1-30 Hz EEG signal through a 4th-order Butterworth bandpass filter;

[0034] (3) Each EEG signal block was divided into different trials, and for each trial, the data were normalized to zero mean and unit variance;

[0035] (4) The EEG signal is segmented and each segment represents a complete task. The length of each segment of the EEG signal is 1 s.

[0036] (5) For the training data set representing the learning phase, 40 triplets are generated for each positive sample, each of which includes an anchor sample, a positive sample, and multiple negative samples;

[0037] (6) For the training data set in the classifier learning phase, all training samples are used for training;

[0038] (7) For the training data set in the test phase, all test samples are used for testing;

[0039] Step 2 is as follows:

[0040] The specific structure of the ACMA-DL model is mainly summarized into four parts: parallel convolution layer, spatial convolution layer, feature fusion layer and adaptive channel hybrid attention mechanism. Pytorch is used to build the model here. The following is a detailed description of each part:

[0041] (1) Parallel convolutional layers

[0042] In the parallel convolution layer part, 4 parallel convolution layers are used to extract multi-scale temporal features. These convolution layers use the same time window setting, the purpose is to perform preliminary temporal feature extraction on the input EEG signal to capture the temporal changes in the EEG signal. In order to ensure that the output size after the convolution operation is consistent in the process of extracting multi-scale temporal features, we apply zero padding in each convolution layer. Specifically, each convolution layer performs zero padding operations of kernel_length / 2 on the left and right, and kernel_length is 3 to ensure that the convolution operation does not change the temporal dimension of the output feature map. This strategy can ensure that the output after convolution maintains the same temporal dimension as the input, which is crucial for subsequent feature processing. Experiments show that the optimal convolution kernel size is set to (1,3), that is, 3 time points are used as convolution windows in the time dimension to ensure that sufficiently fine temporal information can be extracted. The stride defaults to 1, which means that the convolution operation slides one time point at a time to ensure the fineness of feature extraction. The number of output channels of the convolution operation is set to 8, which is the optimal value obtained after experimental verification. After the convolution operation, batch normalization is used to standardize the convolution output. The introduction of batch normalization can significantly prevent the gradient vanishing or gradient exploding problems during training, thereby accelerating the convergence of the model and improving the stability of training. Batch normalization is performed after each convolution layer to ensure that the multi-scale features extracted in each layer can be trained within the same numerical range.

[0043] (2) Spatial convolution layer

[0044] In the spatial convolution layer, four parallel convolution layers are used to extract multi-scale spatial features. Similar to the extraction of temporal features, the extraction of spatial features requires the use of different time windows and filters to capture the various spatial changes in the input EEG signal. In this step, the convolution operation processes the spatial dimension of the EEG signal to ensure that the spatial information in the signal can be extracted. After many experiments, the best parallel structure is as follows: First, the input signal is processed through the convolution layer. The convolution kernel size is set to (62,1), where 62 represents the number of channels and 1 is the time dimension. This setting ensures that the convolution operation fully covers all channels in the spatial dimension and ensures that the spatial features of the signal are fully extracted. The stride defaults to 1, which means that the convolution operation slides channel by channel without skipping any spatial information. The number of input channels and the number of output channels are both set to 8, which is the optimal value verified by many experiments. In order to further improve the computational efficiency, the model adopts the depthwise convolution method, that is, each input channel is independently convolved with the corresponding output channel through groups=8. Such operations reduce the number of parameters and computational costs, making the model more efficient when processing large-scale EEG signals. After the convolution operation, batch normalization is used to standardize the convolution output to ensure that the output of the convolution layer remains stable within the numerical range, thereby avoiding the problem of gradient vanishing or exploding and accelerating the training process of the model. Batch normalization is performed after each convolution layer to ensure the stability of the training process. Next, the ELU (Exponential Linear Unit) activation function is used to introduce nonlinearity to help the model alleviate the problem of gradient vanishing. Compared with other activation functions, ELU can better handle gradient vanishing, especially in deep networks, ensuring that the model better fits complex data distributions. Finally, a Dropout layer is added with a dropout ratio set to 0.6. The Dropout layer prevents model overfitting by randomly dropping some neurons and improves the generalization ability of the model. Experimental verification shows that this dropout ratio can reduce model overfitting while maintaining sufficient expression ability.

[0045] (3) Feature Fusion Layer

[0046] The purpose of the feature fusion layer is to fuse features from different levels to improve the model's expressiveness and its ability to capture multi-scale features. First, the input to the feature fusion layer is a list of features containing four feature maps of different sizes. These feature maps are output from different convolutional layers and represent feature information of different scales and levels. Since these feature maps have different sizes after different downsampling and convolution operations, they need to be adjusted to a consistent size for effective feature fusion.

[0047] To achieve this goal, we first define the target size. Usually, the feature map size with the highest resolution is selected as the target size. In this dataset, the feature map with the highest granularity in the time dimension (64, 8, 1, 31) is selected as the target size. Subsequently, the torch.nn.functional.interpolate function is used to interpolate the feature map, using bilinear interpolation (mode = 'bilinear') to smoothly adjust feature maps of different scales to the same size. During interpolation, the parameter align_corners = False can avoid alignment problems at the boundaries of the feature map, ensure the consistency of the feature map boundaries, and improve the quality of the interpolated feature map.

[0048] After interpolation adjustment, all feature maps will be converted to the same target size, forming a list of feature maps of the same size. This provides a basis for subsequent feature fusion operations, allowing each feature map to be operated pixel by pixel on the same spatial dimension. This process ensures that information between feature maps is not lost, improves the ability to express features after fusion, and provides support for the model to capture multi-scale features. After feature fusion, the complementarity of features can be further enhanced through weighted fusion or convolution to improve the classification performance of the model.

[0049] (4) Adaptive Hybrid Attention Mechanism

[0050] In the adaptive channel hybrid attention mechanism, the input features are first processed by the global context branch. Specifically, the feature map is pooled to a size of (1,1) using Global Average Pooling (GAP). This operation can generate global context information for the entire input feature map by globally averaging all spatial positions. This global context can capture global semantic information, thereby better understanding the overall distribution of features. After that, the pooled features are further extracted through two one-dimensional convolutional layers with a convolution kernel size of 1 and a channel number of 8, and then through the ReLU activation function to generate the final global context features. This design ensures that the information at the global context level can be fully extracted and used for subsequent processing. At the same time, the adaptive channel hybrid attention mechanism also includes a self-attention branch. This branch generates query, key, and value vectors through three one-dimensional convolutional layers. The number of output channels of the query and key vectors is 1, and the number of output channels of the value vector is 8. The attention weights are obtained by calculating the similarity between the query and the key, and then these weights are applied to the value vector. The calculation formula for attention is:

[0051]

[0052] where Q and K are query and key vectors respectively, d k is the dimension of the key vector. This formula calculates the dot product of the query and the key, obtains the attention distribution through a soft maximization operation, and finally applies it to the value vector to generate the self-attention feature. Then, the self-attention feature and the global context feature are weighted combined by the trainable parameter gamma. Gamma is a scalar parameter that controls the relative importance of the self-attention feature and the global context feature in the final output. The combined feature map is resized to (1,1) by bilinear interpolation to ensure the consistency of spatial dimensions and prepare suitable inputs for subsequent processing steps. In addition, the channel attention branch generates global features of the input feature map through global average pooling and global maximum pooling. Global average pooling and global maximum pooling capture the average information and maximum response value of the input feature map respectively. The feature map after these two pooling operations is input to the fully connected layer for further processing to obtain the channel attention weight. The number of intermediate channels in the fully connected layer is 8, and the ReLU activation function is used to enhance the nonlinear expression ability. Finally, the generated channel attention weights will be multiplied with the original feature map channel by channel to enhance the expression ability of important channels, so that the model pays more attention to the features that contribute to the classification.

[0053] Step 3 is as follows:

[0054] In the representation learning stage, the ACMA-DL model uses the triplet loss function (Triplet Loss) for feature learning. Each positive sample is paired with multiple positive samples and negative samples to generate 40 triplets. In each training cycle, the model extracts anchor samples, positive samples, and multiple negative samples from the training set and loads them onto the device. Each time, 64 EEG signals are input into the network for training, and iterates for 30 rounds. In order to select the most difficult negative sample, the model calculates the cosine similarity between each negative sample and the anchor sample, and selects the negative sample with the highest similarity as the most difficult negative sample. The loss is then calculated using the triplet loss function, and the model parameters are optimized by backpropagation and gradient descent. CosineSimilarity(a,n) is used to calculate the similarity between two vectors a (anchor sample) and n (negative sample), and the formula is as follows:

[0055]

[0056] Where a·n represents the dot product of anchor sample a and negative sample n, and ||a|| and ||n|| are the moduli (i.e., vector lengths) of vectors a and n, respectively. Given multiple negative samples n1, n2, …, n k , the model calculates the cosine similarity between each negative sample and the anchor sample and selects the negative sample with the highest similarity. It can be expressed as the following formula:

[0057]

[0058] Then, the hardest negative sample and the positive sample are substituted into the triplet loss function together, and the formula is as follows:

[0059] L(a,p,n)=max(0,||f(a)-f(p)|| 2 -||f(a)-f(n)|| 2 +margin)

[0060] Where f(a), f(p), and F(n) represent the feature representations of anchor samples, positive samples, and negative samples, respectively. Margin is a hyperparameter used to ensure that the distance between negative samples and anchor samples is at least a constant greater than the distance between positive samples and anchor samples.

[0061] At the end of each cycle, the average loss is calculated, and if the current loss is lower than the optimal loss, the model is saved. In this way, the model can effectively learn the feature distance between samples of different categories, making the feature distance of samples of the same category closer and the feature distance of samples of different categories farther, thereby improving the feature representation ability and classification performance. The Adam optimizer is used to alleviate gradient oscillation during network training, and the learning rate is set to 0.001.

[0062] In the classifier learning phase, the ACMA-DL model loads the feature extractor trained in the learning phase and performs further optimization on the classifier. First, start by loading the pre-trained feature extractor. The category weights are calculated by combining all labels, and the optimized Focal loss function is used to handle the imbalance between classes. The optimized Focal Loss function improves the processing capabilities of difficult-to-classify samples and unbalanced categories by introducing sample difficulty weights and category weights. Category weights are calculated based on the number of samples in each category. The larger the weight value, the fewer samples in that category. The category weight formula is as follows:

[0063]

[0064] Where N is the total number of samples and ni is the number of samples in category i. The sample difficulty weight combines factors such as the entropy of the predicted probability of the sample, the probability variance, and the predicted margin. The difficulty weight formula is as follows:

[0065] Difficulty Weight=softmax(Entropy+Variance Impact-Margin Impact)+1

[0066] The formula weights the three factors (Entropy, Variance Impact, and Margin Impact) and calculates the relative difficulty weight of each sample through the softmax function. The result is added by 1 to ensure that the weight will not be negative.

[0067] Entropy=-∑(Probs·log(Probs+1e-10))

[0068] Among them, entropy is used to measure the uncertainty of classification. The larger the entropy, the less certain the model is about the classification of this sample. Probs represents the predicted probability distribution of each category. This formula calculates the uncertainty of sample classification by multiplying the logarithm of each category probability and summing them up. Adding 1e-10 is to prevent the situation of zero probability when taking the logarithm.

[0069] Variance Impact=sigmoid(Variance×10)

[0070] Variance represents the variance of the probability distribution of the model output, and 10 is a scaling factor to enhance the impact of the variance on the results.

[0071] MarginImpact=1-sigmoid(Margin×10)

[0072] Margin is the difference between the first two predicted probabilities. The difference is mapped to (0,1) through the sigmoid function, and then 1 is subtracted from the result to reflect that a smaller margin gap will increase the difficulty of classification. Multiplying by 10 is to amplify this effect.

[0073] Margin=Top Two Probs[0]-Top Two Probs[1]

[0074] Among them, Top Two Probs[0] is the model's predicted probability for the highest probability category, and Top Two Probs[1] is the predicted probability for the second highest probability category. The larger the difference, the more confident the model is that its classification is correct and the lower the difficulty. The final Focal Loss is calculated by combining the category weight and the sample difficulty weight, and its formula is:

[0075] FocalLoss = α·(1-p t ) γ ·CELoss weighted

[0076] CELoss weighted=CELoss×Difficulty Weight×Class Weight

[0077] Among them, CELoss is the cross entropy loss, which is used to calculate the difference between the model prediction result and the true label, α is a parameter to control the sample imbalance, and its value is 0.25, and γ is a parameter to control the influence of the difficulty weight, and its value is 2.0.

[0078] p t =exp(-CE Loss weighted )

[0079] Among them, p t It is a probability value calculated based on the weighted cross entropy loss, which is used to measure the model's prediction confidence on the sample. During the training cycle, the samples in the dataset are loaded onto the device, and 128 EEG signals are input into the network for training each time. 60 rounds are iterated, and forward propagation is performed to calculate the output logits. Then, the loss is calculated and back propagation is performed to update the model parameters. After each cycle, the average loss and classification accuracy are calculated and recorded. If the average loss of the current cycle is lower than the previous minimum loss, the state of the current model is saved. Through this process, the classification performance of the model is optimized to ensure that the extracted features can be accurately used for classification. The Adam optimizer is used to alleviate gradient oscillation during network training, and the learning rate is set to 0.001.

[0080] Step 4 is as follows:

[0081] Input the test set in step 1 into the trained model in step 3 for classification and recognition, and evaluate the classification indicators such as AUC, BA, recall rate, F1_score, etc.

[0082] The data set and experimental results used by the method of the present invention are described as follows:

[0083] 1. Dataset

[0084] The present invention uses the RSVP-based brain-computer interface benchmark dataset provided by Tsinghua University, which contains EEG data of 64 healthy subjects. The data of each subject is collected through 64 electrode channels with a sampling rate of 250Hz. In the experiment, the subjects performed the target image detection task. Each subject conducted two groups of experiments, each group containing two blocks. Each block included 40 trials, and each trial randomly presented 100 stimulus images at a frequency of 10Hz. The stimulus includes two categories: target images and non-target images, and the probability of presentation of target images is 1% to 4%.

[0085] In this study, 62 electrodes (1–32, 34–42, 44–64) were selected for further processing. Subsequently, the EEG data were filtered using a Butterworth filter with a bandwidth of 0.1–30 Hz. After the onset of each image stimulus, a segment of 0–1000 ms was extracted and the segment was divided into data containing 250 sampling points. Therefore, the EEG data format of a single trial was structured as a matrix of 62 electrodes by 250 sampling points.

[0086] The present invention uses the average values ​​of AUC, BA, Recall and F1 of 64 subjects as the performance indicators of the model.

[0087] 2. Experimental results

[0088] In order to verify the effectiveness and versatility of the method of the present invention, a variety of experiments based on machine learning were conducted on the RSVP brain-computer interface benchmark dataset, involving a variety of classic and traditional models, such as HDCA, xDAWN, and XGB-DIM. A variety of experiments based on deep learning were also conducted, involving a variety of classic and traditional models, such as EEGNet, EEG-Inception, and DRL. The six methods and the method proposed in the present invention were used as comparisons to evaluate the classification indicators such as AUC, BA, recall rate, F1_score, and the average value of 64 subjects was taken. The table data is as follows:

[0089] Table 1 Comparison indicators of several models

[0090]

[0091] Finally, the ACMA-DL method achieved the highest overall performance. This method performed well in all evaluation indicators, including classification AUC, balanced accuracy, recall rate and F1_score, significantly surpassing other methods. These results show that deep learning methods, especially ACMA-DL, have significant advantages in improving the classification accuracy and stability of the RSVP-BCI system. This research result not only verifies the effectiveness of the new method, but also demonstrates its potential in practical applications. 。

Claims

1. A RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning, characterized in that: The specific steps include: Step 1, data preprocessing and data set division: First, the original EEG signal data is preprocessed to delete unnecessary redundant channels to reduce noise interference and retain channel information that is useful for the classification task; Secondly, each EEG signal block was divided into different trials, and the signal of each trial was normalized, that is, the signal was transformed into zero mean and unit variance to eliminate the dimensional differences between different signals; then, the EEG signal was filtered using a bandpass filter to retain the frequency components related to the RSVP task and filter out low-frequency and high-frequency noise interference; finally, the preprocessed data set was divided into a training set and a test set; Step 2, construct the ACMA-DL model; for the RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupled learning (ACMA-DL), the model first extracts multi-scale temporal features through parallel convolutional layers, and uses different time windows and filter parameters to capture various temporal variation information of EEG signals; through multi-scale extraction of temporal features, signal changes in different frequency bands are captured, thereby improving the feature expression ability of the time dimension; Next, the model uses a spatial convolution layer to extract spatial features from the signal, ensuring that the spatial dependencies between brain regions are captured in multi-electrode EEG signals. By extracting spatial features, the model can learn the relationship between different electrode channels and improve its ability to model the spatial characteristics of EEG signals. After extracting multi-scale features, the model uses interpolation to fuse temporal and spatial features from different scales, improving the expressiveness and generalization of features through multi-level feature fusion. Finally, the model introduces an adaptive channel hybrid attention mechanism to weight the fused multi-scale features and dynamically adjust the weights of each channel. Step 3: The training process is decoupled into two stages. In the representation learning stage, a triplet loss optimization strategy based on difficult samples is adopted. This strategy calculates the similarity between each anchor sample and its positive sample and selects the negative sample with the highest cosine similarity, i.e., the "hardest" negative sample. In the representation learning stage, the training set constructed in step 1 is used to input the ACMA-DL model to train the feature extractor. In the classifier learning stage, the feature extractor trained in the representation learning stage is used, and the classifier is optimized in combination with the optimized Focal loss function; the optimized Focal loss function solves the class imbalance problem by reducing the weight of easy-to-classify samples and increasing the weight of difficult-to-classify samples. In this stage, the training set of the classifier learning stage constructed in step 1 is loaded to train the classifier of the ACMA-DL model.

2. The RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning as claimed in claim 1, characterized in that: Step 1 is as follows: (1) Two channels, EOG1 and EOG2, were removed from the 64-channel original data; (2) Extract the 0.1-30 Hz EEG signal through a 4th-order Butterworth bandpass filter; (3) Each EEG signal block was divided into different trials, and for each trial, the data were normalized to zero mean and unit variance; (4) The EEG signal is segmented and each segment represents a complete task. The length of each segment of the EEG signal is 1 s. (5) For the training data set representing the learning phase, 40 triplets are generated for each positive sample, each of which includes an anchor sample, a positive sample, and multiple negative samples; (6) For the training data set in the classifier learning phase, all training samples are used for training.

3. The RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning as claimed in claim 1, characterized in that: Step 2 is as follows: The specific structure of the ACMA-DL model is summarized into four parts: parallel convolution layer, spatial convolution layer, feature fusion layer and adaptive channel hybrid attention mechanism; Pytorch is used to build the model here; the following is a detailed description of each part: (1) Parallel convolutional layers In the parallel convolution layer part, 4 parallel convolution layers are used to extract multi-scale temporal features. These convolution layers use the same time window setting to perform preliminary temporal feature extraction on the input EEG signal to capture the temporal changes in the EEG signal. In order to ensure the consistency of the output size after the convolution operation in the process of extracting multi-scale temporal features, zero padding is applied in each convolution layer. Specifically, each convolution layer performs zero padding operations of kernel_length / 2 on the left and right, and kernel_length is 3 to ensure that the convolution operation does not change the temporal dimension of the output feature map. The kernel size is set to (1,3), which means that three time points are used as the convolution window in the time dimension. The stride defaults to 1, which means that the convolution operation slides one time point at a time. The number of output channels of the convolution operation is set to 8. After the convolution operation, batch normalization is used to standardize the convolution output. (2) Spatial convolution layer In the spatial convolution layer, four parallel convolution layers are used to extract multi-scale spatial features. The parallel structure is as follows: First, the input signal is processed by the convolution layer; the convolution kernel size is set to (62,1), where 62 represents the number of channels and 1 is the time dimension; the stride defaults to 1, which means that the convolution operation slides channel by channel without skipping any spatial information; The number of input channels and output channels are both set to 8; After the convolution operation, batch normalization is used to standardize the convolution output to ensure that the output of the convolution layer remains stable within the numerical range, thereby avoiding the gradient vanishing or exploding problem and accelerating the model training process; batch normalization is performed after each convolution layer; Next, the ELU (Exponential Linear Unit) activation function is used to introduce nonlinearity to help the model alleviate the gradient vanishing problem; finally, the Dropout layer is added, and the dropout ratio is set to 0.6; (3) Feature Fusion Layer First, the input to the feature fusion layer is a list of features containing 4 feature maps of different sizes. These feature maps are output from different convolutional layers and represent feature information of different scales and levels. To achieve this goal, we first define the target size and select the feature map size with the highest resolution as the target size. We select the feature map with the highest granularity in the time dimension (64, 8, 1, 31) as the target size. Then, we use the torch.nn.functional.interpolate function to interpolate the feature map using bilinear interpolation. After interpolation adjustment, all feature maps will be converted to the same target size to form a list of feature maps of consistent size; (4) Adaptive Hybrid Attention Mechanism In the adaptive channel hybrid attention mechanism, the input features are first processed by the global context branch. Specifically, the feature map is pooled to a size of (1,1) using global average pooling. This operation can generate global context information for the entire input feature map by globally averaging all spatial positions. The pooled features are further extracted through two one-dimensional convolutional layers with a convolution kernel size of 1 and 8 channels, and then passed through the ReLU activation function to generate the final global context features. At the same time, the adaptive channel hybrid attention mechanism also includes a self-attention branch; This branch generates query, key, and value vectors through three one-dimensional convolutional layers; the number of output channels of query and key vectors is 1, and the number of output channels of value vector is 8; the attention weights are obtained by calculating the similarity between query and key, and then these weights are applied to the value vector; the attention calculation formula is: where Q and K are query and key vectors respectively, d k is the dimension of the key vector; Next, the self-attention feature and the global context feature are weighted combined by the trainable parameter gamma; gamma is a scalar parameter that controls the relative importance of the self-attention feature and the global context feature in the final output; the combined feature map is adjusted to the size of (1,1) by bilinear interpolation; In addition, the channel attention branch generates global features of the input feature map through global average pooling and global maximum pooling; global average pooling and global maximum pooling capture the average information and maximum response value of the input feature map respectively. The feature map after these two pooling operations is input into the fully connected layer to obtain the channel attention weight; the number of intermediate channels in the fully connected layer is 8, and the ReLU activation function is used to enhance the nonlinear expression ability; the generated channel attention weight will be multiplied with the original feature map channel by channel.

4. The RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning as claimed in claim 1, characterized in that: Step 3 is as follows: In the representation learning stage, the ACMA-DL model uses the triplet loss function for feature learning; each positive sample is paired with multiple positive samples and negative samples to generate 40 triplets; in each training cycle, the model extracts anchor samples, positive samples and multiple negative samples from the training set and loads them onto the device; 64 EEG signals are input into the network for training each time, and iterates 30 rounds; in order to select the most difficult negative sample, the model calculates the cosine similarity between each negative sample and the anchor sample, and selects the negative sample with the highest similarity as the most difficult negative sample; then the triplet loss function is used to calculate the loss, and the model parameters are optimized through back propagation and gradient descent; Cosine Similarity (a, n) is used to calculate the similarity between two vectors a, namely the anchor sample and the negative sample n, and the formula is as follows: Where a·n represents the dot product of anchor sample a and negative sample n, ||a|| and ||n|| are the moduli (i.e., vector lengths) of vectors a and n, respectively; given multiple negative samples n1, n2, …, n k , the model calculates the cosine similarity between each negative sample and the anchor sample and selects the negative sample with the highest similarity, which is expressed as the following formula: Then, the hardest negative sample and the positive sample are substituted into the triplet loss function together, and the formula is as follows: L(a,p,n)=max(0,||f(a)-f(p)|| 2 -||f(a)-f(n)|| 2 +margin) Where f(a), f(p), and f(n) represent the feature representations of anchor samples, positive samples, and negative samples, respectively; margin is a hyperparameter used to ensure that the distance between negative samples and anchor samples is at least a constant greater than the distance between positive samples and anchor samples; At the end of each cycle, the average loss is calculated. If the current loss is lower than the best loss, the model is saved. The Adam optimizer is used to alleviate gradient oscillation during network training, and the learning rate is set to 0.

001. In the classifier learning phase, the ACMA-DL model loads the feature extractor trained in the learning phase and performs further optimization on the classifier. First, it starts with loading the pre-trained feature extractor. The category weights are calculated by combining all labels, and the optimized Focal loss function is used to handle the imbalance between classes. The optimized Focal Loss function improves the processing capabilities of difficult-to-classify samples and imbalanced categories by introducing sample difficulty weights and category weights. The category weights are calculated based on the number of samples in each category. The larger the weight value, the fewer samples in that category. The category weight formula is as follows: Where N is the total number of samples, n i is the number of samples of category i; The difficulty weight formula is as follows: Difficulty Weight = softmax(Entropy + Variance Impact - Margin Impact) + 1, where the formula weights the three factors of Entropy, Variance Impact, and Margin Impact and calculates the relative difficulty weight of each sample through the softmax function; the result is added by 1 to ensure that the weight will not be negative; Entropy=-∑(Probs·log(Probs+1e-10)) Among them, entropy is used to measure the uncertainty of classification; the larger the entropy, the less certain the model is about the classification of this sample; Probs represents the predicted probability distribution of each category; Variance Impact=sigmoid(Variance×10) Among them, Variance represents the variance of the probability distribution of the model output, and 10 is a scaling factor; MarginImpact=1-sigmoid(Margin×10) Among them, Margin represents the difference between the first two predicted probabilities; this difference is mapped to (0,1) through the sigmoid function, and then the result is subtracted from 1; Margin = Top Two Probs[0] - Top Two Probs[1], where Top Two Probs[0] is the model's predicted probability for the highest probability category, and Top Two Probs[1] is the predicted probability for the second highest probability category. The final FocalLoss is calculated by combining the category weight and sample difficulty weight, and the formula is: FocalLoss=α·(1-p t ) γ ·CELoss weighted CELoss weighted =CELoss×Difficulty Weight×Class Weight Among them, CELoss is the cross entropy loss, which is used to calculate the difference between the model prediction result and the true label, α is a parameter to control sample imbalance, and its value is 0.25, and γ is a parameter to control the influence of difficulty weight, and its value is 2.0; p t =exp(-CE Loss weighted ) Among them, p t It is a probability value calculated based on the weighted cross entropy loss, which is used to measure the model's prediction confidence on the sample. During the training cycle, the samples in the dataset are loaded onto the device, and 128 EEG signals are input into the network for training each time. After 60 rounds of iteration, forward propagation is performed to calculate the output logits. Then, the loss is calculated and back propagation is performed to update the model parameters. After each cycle, the average loss and classification accuracy are calculated and recorded. If the average loss of the current cycle is lower than the previous minimum loss, the state of the current model is saved. During the network training process, the Adam optimizer is used to alleviate gradient oscillation, and the learning rate is set to 0.

001.

5. The RSVP-type unbalanced EEG signal classification method based on adaptive channel hybrid attention mechanism and decoupling learning as claimed in claim 1, characterized in that: You can also set step 4, which is as follows: Input the test set in step 1 into the trained model in step 3 for classification and recognition, and evaluate the classification AUC, BA, recall rate or F1_score.

Images