SSVEP (Steady-State Visual Evoked Potential) classification method based on time-frequency collaborative channel attention and multistage fusion

Abstract

A steady-state visual evoked potential classification method based on a multistage time-frequency collaborative channel attention mechanism comprises the steps that 1, tested electroencephalogram signals under different frequencies are collected to serve as sample signals, the sample signals are preprocessed to obtain a sample data set, and the sample data set is divided into a training set, a verification set and a test set according to a set proportion; constructing an MSFDCA-Net network which comprises a branch-level time-frequency collaborative channel attention convolution layer, a multi-branch feature fusion layer, a final feature extraction convolution layer and an output layer in sequence from an input layer; the training set is input into the MSFDCA-Net network for training, and a trained MSFDCA-Net network is obtained; and verifying the trained MSFDCA-Net network by using the verification set to obtain a final MSFDCA-Net network. And inputting the test set into the final MSFDCA-Net network to complete identification of different electroencephalogram signals in the test set. Through the learnable adaptive filter bank and the multi-level attention fusion mechanism, the limitation of time-frequency cooperation deficiency and feature fusion roughness of an existing method is overcome, the accuracy and robustness of SSVEP classification are remarkably improved, and the method is particularly suitable for a high-speed brain-computer interface system under a short time window.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface technology, specifically to a steady-state visual evoked potential classification method based on time-frequency collaborative channel attention and multi-level fusion mechanism. Background Technology

[0002] Steady-state visual evoked potentials (SSVEPs), as a typical non-invasive BCI paradigm, have become a research hotspot due to their advantages such as simple operation and no need for training. However, SSVEP signals are characterized by low signal-to-noise ratio, susceptibility to interference, and complex multi-band harmonic characteristics. Traditional methods such as canonical correlation analysis and filter bank CCA, although computationally efficient, are difficult to capture nonlinear features, thus limiting their performance.

[0003] Machine learning methods based on EEG have made some progress in the field of steady-state visual evoked potential classification. The process includes four stages: EEG signal acquisition, signal preprocessing (interference removal), feature extraction, and classification. This method involves manually extracting features from the EEG signals to reduce the data volume, and then using methods such as Support Vector Machines, K-Nearest Neighbors, Random Forests, and Naive Bayes to classify and recognize the EEG signals. However, this manual feature extraction method is time-consuming and labor-intensive when dealing with large amounts of EEG data, is prone to information loss, and the classification accuracy is difficult to achieve optimally.

[0004] Deep learning methods based on EEG can achieve end-to-end mapping, directly inputting preprocessed EEG signals into deep neural networks for training, eliminating the need for complex feature extraction methods and helping to solve nonlinear problems. For example, CNN, LSTM, and Transformer have been introduced into the SSVEP classification task, but problems such as missing time-frequency coordination, fixed filter banks, and coarse feature fusion still exist. This prevents the full utilization of the temporal and spatial information of EEG, resulting in poor model accuracy and generalization ability. Therefore, there is an urgent need for a novel SSVEP classification method that can collaboratively model temporal dynamics and frequency harmonic structures, and possess adaptive feature fusion capabilities, to improve classification accuracy and system robustness. Summary of the Invention

[0005] The technical problem to be solved by this invention is to overcome the defects of the separation of time-domain and frequency-domain features and the coarse fusion mechanism in existing SSVEP decoding methods, and to provide an SSVEP classification method that can synergistically utilize the dynamics of the time domain and the harmonic characteristics of the frequency domain, and achieve refined feature optimization through multi-level progressive fusion.

[0006] This invention provides an SSVEP classification method based on time-frequency collaborative channel attention and multi-level fusion, comprising the following steps:

[0007] Step 1: Collect EEG signals from the subjects at different frequencies as sample signals, preprocess the sample signals to obtain a sample dataset, and divide the sample dataset into a training set, a validation set, and a test set according to a set ratio;

[0008] Step 2: Construct the MSFDCA-Net network; The structure of the MSFDCA-Net is as follows: starting from the input layer, there are branch-level time-frequency co-channel attention convolutional layers, multi-branch feature fusion layers, final feature extraction convolutional layers, and the output layer.

[0009] The input layer receives four parallel EEG signals; the branch-level time-frequency co-channel attention convolutional layer extracts and enhances local spatiotemporal features of the signals from the four parallel input branches; the multi-branch feature fusion layer aggregates and calibrates the features from the four branches; the final feature extraction convolutional layer performs deeper transformations and refinements on the fused and calibrated features, enhancing key features and suppressing irrelevant information in both the frequency and time domains; and the output layer outputs the classification of the identified input signals.

[0010] Step 3: Input the training set into the MSFDCA-Net network for training to obtain the trained MSFDCA-Net network; and use the validation set to validate the trained MSFDCA-Net network to obtain the final MSFDCA-Net network.

[0011] Step 4: Input the test set into the final MSFDCA-Net network to complete the recognition of different EEG signals in the test set.

[0012] Furthermore, step 1 includes:

[0013] Step 1.1: The user watches the screen flashing program at different frequencies in sequence. Each frequency contains multiple trials. According to the international standard 10-20 electrode placement system, the corresponding EEG signal data samples are collected and each sample signal is labeled.

[0014] Step 1.2: The sample signal is preprocessed and decomposed into four sub-bands: 3-14Hz, 9-26Hz, 14-38Hz, and 19-50Hz. Low-frequency and high-frequency noise, as well as interference from ECG signals and power line frequencies, are removed. The data sampling rate is reduced from 1000Hz to 250Hz through downsampling. Then, DC offset is eliminated through baseline correction to ensure that the zero baseline of the signal is consistent with the reference electrode. Interference signals from electrooculography and electromyography are removed through artifact removal. Finally, the signal is divided into multiple windows of different lengths to form a sample dataset through signal segmentation.

[0015] Step 1.3: Divide the sample dataset into training set, validation set and test set according to the set ratio, and set the number of sample signals to be processed each time EEG signal recognition is performed.

[0016] Furthermore, the structure of the MSFDCA-Net network also includes:

[0017] The three convolutional layers and the subsequent independent convolutional layer in the sub-band feature extraction module all use the Gelu activation function; the four convolutional units are respectively denoted as convolutional unit 1, convolutional unit 2, convolutional unit 3 and convolutional unit 4; wherein convolutional layer 1 is connected in sequence with convolution operation 1, batch normalization layer 1, Gelu activation function 1, and Dropout layer 1; convolutional layer 2 is connected in sequence with convolution operation 2, batch normalization layer 2, Gelu activation function 2, and Dropout layer 2; convolutional layer 3 is connected in sequence with convolution operation 3, batch normalization layer 3, Gelu activation function 3, and Dropout layer 3; and convolutional layer 4 is connected in sequence with convolution operation 4, batch normalization layer 4, Gelu activation function 4, and Dropout layer 4.

[0018] The structure of the time-frequency collaborative channel attention module is as follows: starting from the input layer, there are parallel frequency domain channel attention (FCA) layers and time domain channel attention (TCA) layers, weight concatenation layer, fusion convolutional layer, and output layer.

[0019] The input layer is used to receive the original feature input;

[0020] The Frequency Domain Channel Attention (FCA) architecture consists of the following layers: a zero-padding layer, a discrete wavelet transform layer, a global average pooling layer, a channel reduction and reshaping layer, and a weight generation network layer. The zero-padding layer pads the height and width of the input feature map to an even number of dimensions to meet the requirements of the Discrete Wavelet Transform (DWT) algorithm. The discrete wavelet transform layer decomposes the spatial feature map into four sub-bands (LL, LH, HL, HH) of different frequencies, transforming the spatial information into the frequency domain to capture global information of different frequency components in the features. The global average pooling layer performs global compression on the feature map of each frequency band channel, reducing the frequency of each channel... The domain feature information is aggregated into a single scalar value to generate a global frequency domain descriptor vector. The channel reduction and reshaping layer is used to aggregate the descriptor vectors generated by DWT with 4 times the number of channels (4C) back to the original number of channels (C) by summing, generating a comprehensive descriptor for each original channel that integrates all its frequency band information. The weight generation network layer is used to input the comprehensive frequency domain descriptor into a small neural network (two 1x1 convolutions) to learn the nonlinear mapping between frequency domain information and channel importance, and outputs a weight vector between 0 and 1 through the Sigmoid activation function, representing the importance of each channel based on frequency domain information.

[0021] The Temporal Channel Attention (TCA) layer has the following structure: parallel global average pooling and global max pooling layers, followed by a weight generation network layer. The global average pooling layer compresses the spatial dimension, extracting the average value of each channel's feature map to form a global context descriptor. The global max pooling layer also compresses the spatial dimension, extracting the maximum value of each channel's feature map. Its function is to capture the most salient feature responses across the entire spatial range, supplementing the average descriptor. The weight generation network comprises two independent one-dimensional convolutional networks. These networks nonlinearly transform the descriptors generated by average pooling and max pooling, respectively, learning the relationship between spatial statistical features and channel importance. The outputs of the two networks are then summed to fuse the information extracted by the two different pooling methods (average features and salient features), thereby generating a more robust attention weight vector, which is then output through a sigmoid activation function.

[0022] The weight splicing layer is used to splice the frequency domain weight vector generated by the FCA path and the spatial domain weight vector generated by the TCA path in the channel dimension to form a composite weight vector that integrates information from both the frequency domain and the spatial domain.

[0023] The fusion convolutional layer described therein is used to adaptively learn how to most effectively fuse information from the frequency and spatial domains. It first reduces the dimensionality of the concatenated high-dimensional weights to save computation and increase nonlinearity, and then projects them back to the original number of channels, finally outputting a set of optimized, comprehensive attention weights.

[0024] Furthermore, the process of building the MSFDCA-Net network also includes: designing the kernel size, stride, and number of kernels for each layer based on the data size (i.e., time window length) of the input signal of the MSFDCA-Net network and the number of sampling points set within the signal period of each input signal.

[0025] Furthermore, step 3 includes:

[0026] Step 3.1: Input the training set EEG data into the MSFDCA-Net network in batches of 64, with each sample containing multi-channel SSVEP signals; use the multi-branch sub-band feature extraction module and the frequency domain-time domain dual attention module (FDCALayer) in the network to process signals of different frequency bands in parallel, extract discriminative spatiotemporal features, and enhance key frequency response features through the attention mechanism;

[0027] Step 3.2: Input the fusion features extracted in Step 3.1 into the final convolution and attention optimization layer for advanced feature extraction. Map the features to probability distribution vectors of different stimulus frequency categories through a classifier and establish the correspondence between the output and the SSVEP stimulus frequency.

[0028] Step 3.3: Using cross-entropy as the loss function, the Adam optimizer is used to perform end-to-end optimization of the network parameters, and the learning rate is dynamically adjusted using the ReduceLROnPlateau strategy; the training process lasts for 4000 epochs, and early stopping (training is terminated when the validation set loss no longer decreases for several consecutive epochs) and model checkpointing techniques are used to save the optimal weights; if the model does not converge, the parameters are updated and the process returns to step 3.1 for iterative optimization;

[0029] Step 3.4: Use the saved optimal weight model to perform performance verification on the validation set, and finally evaluate the classification accuracy and information transmission rate on the test set through three-fold cross-validation. All experiments use fixed random seeds to ensure the reproducibility and statistical robustness of the results.

[0030] The working principle of the method of the present invention is as follows:

[0031] S1. Collect EEG signal data; The user watches a pre-set screen flashing program of different frequencies in sequence. Each frequency contains multiple trials, and there is a rest time between every two trials. After the rest time, there is a prompt sound to remind the user to open their eyes. According to the EEG channel positions of the international standard 10-20 electrode placement system, the corresponding EEG signal data is collected.

[0032] S2. Data Preprocessing: The acquired EEG signals were preprocessed by decomposing them into four sub-bands (3-14Hz, 9-26Hz, 14-38Hz, and 19-50Hz) using a filter bank to remove low-frequency and high-frequency noise as well as ECG signals. Power line frequency interference was removed using a power frequency notch filter. The data rate was reduced from 1000Hz to 250Hz by downsampling to reduce the amount of data and computational burden while retaining the main information of the signal. Then, baseline correction was used to eliminate DC offset and ensure that the zero baseline of the signal was consistent with the reference electrode. Artifact removal methods such as independent component analysis were used to remove interference signals such as electrooculography and electromyography. Finally, the signal was divided into multiple windows of different lengths by signal segmentation to verify the superiority of the MSFDCA-Net model at different time lengths.

[0033] S3. Temporal Feature Extraction: A multi-branch temporal convolutional network (TCNN) with shared weights is used to process each sub-band signal and extract primary spatiotemporal features. The signal is sequentially passed through a two-dimensional convolutional layer with input channel 1, output channel 32, and kernel size (62, 1) for cross-channel spatial filtering. Then, it is passed through a two-dimensional convolutional layer with input channel 32, output channel 32, kernel size (1, win_train), where win_train represents the kernel length in the time dimension, stride (5, 5) indicates a stride of 5 in both the height and width directions, and padding (0, 23) indicates no padding in the height direction and 23 padding points on each side of the width (time) direction for temporal convolution and downsampling. Finally, advanced temporal features are extracted through a two-dimensional convolutional layer with input channel 32, output channel 32, and kernel size (1, 5). Each convolutional layer is followed by a batch normalization, dropout layer with a dropout rate of 0.3, and a GELU activation function.

[0034] S4. Three-level progressive feature fusion;

[0035] This step requires the use of the Time-Frequency Cooperative Channel Attention (FDCA) module. It consists of two parts: Temporal Channel Attention (TCA) performs Global Average Pooling (GAP) and Global Max Pooling (GMP) on the input features, generating two sets of temporal attention weights through a shared bottleneck structure. These weights are then summed and activated by a Sigmoid function to obtain the temporal attention map W. T Frequency domain channel attention (FCA) maps the input features to the frequency domain using discrete wavelet transform (DWT), resulting in a frequency domain representation. Subsequently, a dual-path aggregation and bottleneck transform, similar to TCA, are performed on the frequency domain representation to generate a frequency domain attention map W. F Next, W T With W F The data is concatenated along the channel dimension, and an optimal combination of weights is learned through a lightweight fusion network to generate the final collaborative attention map W. final Finally, use W final The original input features are weighted channel by channel to achieve adaptive feature enhancement.

[0036] The three-level progressive feature fusion process first involves intra-subband time-frequency domain feature fusion, where features extracted from each sub-band branch are input into the FDCA module. This module computes temporal channel attention (TCA) and frequency channel attention (FCA) in parallel and integrates them through an adaptive fusion network, outputting sub-band features optimized by time-frequency collaboration. Next, cross-subband time-frequency domain feature fusion is performed, where the optimized sub-band features are added together and then input into the FDCA module again to evaluate and weight the contributions of different sub-band features, achieving cross-subband collaboration. Finally, the final layer of time-frequency domain feature fusion transforms the cross-subband fused features into a high-dimensional space through convolutional layers, and then refines them using the FDCA module for the final layer.

[0037] S5. Classification and Output: The final optimized features are flattened and fed into the classifier (two-layer fully connected network) to obtain the classification result.

[0038] S6. Model Training and Optimization: The model training adopts an end-to-end single-task learning strategy with the goal of minimizing cross-entropy loss. The Adam optimizer is used to optimize the model parameters, and the learning rate is dynamically adjusted through the ReduceLROnPlateau strategy to improve convergence. The training process lasts for a total of 4000 epochs with a batch size of 64. Early stopping and model checkpointing techniques are used to monitor the performance on the validation set and save the best weights. Finally, the model is evaluated on the test set with classification accuracy and information transfer rate as the core indicators. All experiments use fixed random seeds and three-fold cross-validation to ensure the reproducibility and statistical robustness of the results.

[0039] This invention overcomes the limitations of existing methods, such as the lack of time-frequency collaboration and coarse feature fusion, by using a learnable adaptive filter bank and a multi-level attention fusion mechanism. It significantly improves the accuracy and robustness of SSVEP classification, and is especially suitable for high-speed brain-computer interface systems with short time windows.

[0040] The beneficial effects of this invention are as follows:

[0041] This invention comprehensively preprocesses the acquired EEG signals, including downsampling, bandpass filtering, multi-subband signal decomposition, and time window truncation, effectively improving signal quality and highlighting SSVEP response characteristics. By designing a multi-branch temporal convolutional structure to replace the traditional fixed filter bank, adaptive extraction of subband features is achieved. An innovative time-frequency collaborative channel attention (FDCA) mechanism is introduced, enabling parallel mining of temporal dynamic patterns and frequency harmonic structures, and achieving collaborative optimization of time-frequency information through an adaptive fusion network. A three-level progressive attention fusion strategy is employed, realizing a refined processing flow from intra-subband feature optimization and cross-subband fusion to high-level semantic enhancement, significantly improving the model's ability to focus on key features. The entire MSFDCA-Net network combines high performance with low complexity, achieving state-of-the-art (SOTA) classification accuracy and information transfer rate (ITR) on public datasets, particularly excelling in real-time decoding within short time windows, providing an effective solution for building an efficient and robust real-time SSVEP-BCI system. Attached Figure Description

[0042] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the method of the present invention.

[0044] Figure 2 This is a schematic diagram of the 62-channel electrode distribution of the present invention.

[0045] Figure 3 This is a schematic diagram of the MSFDCA-Net model framework of the present invention. Detailed Implementation

[0046] The following will describe clearly and completely the SSVEP classification method and system based on time-frequency collaborative channel attention and multi-level fusion of the present invention with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0047] Example 1

[0048] This embodiment relates to an SSVEP classification method based on time-frequency collaborative channel attention and multi-level fusion. The implementation process is as follows: Figure 1As shown, it mainly includes the following steps:

[0049] Step 1: Collect EEG signals from the subjects at different frequencies as sample signals, preprocess the sample signals to obtain a sample dataset, and divide the sample dataset into a training set, a validation set, and a test set according to a set ratio;

[0050] Step 1.1: Using a 62-lead EEG device conforming to the international 10-20 system, collect EEG signals generated when the user gazes at four different frequencies (5.45Hz, 6.67Hz, 8.57Hz, 12Hz) flashing stimuli on the screen. Each frequency contains 20 trials, each trial contains a 10-second stimulus presentation time, and there is a 5-second rest period between every two trials. Before the end of the rest period, a prompt tone reminds the subject to open their eyes. The sampling frequency is 1000Hz.

[0051] Step 1.2: Preprocessing the acquired raw signal. First, the data rate is reduced from 1000Hz to 250Hz by downsampling. Second, a fourth-order infinite impulse response (IIR) bandpass filter (passband 3-50Hz) is used to filter out low-frequency and high-frequency noise. Subsequently, a power line notch filter (50Hz) is used to eliminate power line interference. DC offset is eliminated by baseline correction. Independent component analysis (ICA) is used to remove artifacts such as electrooculography (EOG) and electromyography (EMG). Finally, the signal is passed through a filter bank consisting of four sixth-order Butterworth bandpass filters, decomposing it into four sub-band signals (3-14Hz, 9-26Hz, 14-38Hz, 19-50Hz). To simulate stimulus delay and increase data diversity, a starting point is randomly sampled from a uniform distribution U(0.14, 10.14) seconds, and a time window of length t seconds is extracted.

[0052] Step 1.3: Divide the sample dataset into training set, validation set and test set according to the set ratio, and set the number of sample signals to be processed each time EEG signal recognition is performed.

[0053] In this embodiment, since the SSVEP classification method based on time-frequency collaborative channel attention and multi-level fusion was proposed in the context of signals with low signal-to-noise ratio and easy interference, in order to verify its effectiveness, the dataset proposed by Lee et al. in 2019 was used as the data sample to construct the sample dataset, and the sample dataset was divided into training set, validation set and test set in a ratio of 7:1:2. At the same time, the number of samples to be processed each time signal recognition was set to 64 to accelerate the learning speed of MSFDCA-Net network parameters.

[0054] Step 2: Set up the MSFDCA-Net network;

[0055] The structure of MSFDCA-Net is as follows: starting from the input layer, it consists of a branch-level time-frequency co-channel attention convolutional layer, a multi-branch feature fusion layer, a final feature extraction convolutional layer, and an output layer. The three convolutional layers in the sub-band feature extraction module and the subsequent independent convolutional layer all use the Gelu activation function; its formula is approximately:

[0056]

[0057] Where x represents the function input value and is also the only variable, and tanh represents the hyperbolic tangent activation function.

[0058] The four convolutional units are designated as Convolutional Unit 1, Convolutional Unit 2, Convolutional Unit 3, and Convolutional Unit 4, respectively. Convolutional layer 1 sequentially connects Convolution Operation 1, Batch Normalization Layer 1, Gelu Activation Function 1, and Dropout Layer 1, using 32 convolutional kernels of size (62,1) with a stride of 1. Convolutional layer 2 sequentially connects Convolution Operation 2, Batch Normalization Layer 2, Gelu Activation Function 2, and Dropout Layer 2, using 32 convolutional kernels of size (1,win_train) with a stride of (5,5) and padding of (0,23). Convolutional layer 3 sequentially connects Convolution Operation 3, Batch Normalization Layer 3, Gelu Activation Function 3, and Dropout Layer 3, using 32 convolutional kernels of size (1,5). Convolutional layer 4 sequentially connects Convolution Operation 4, Batch Normalization Layer 4, Gelu Activation Function 4, and Dropout Layer 4, using 64 convolutional kernels of size (1,6). The dropout probability for all Dropout layers is 0.3.

[0059] The time-frequency collaborative channel attention module described therein has the following structure: starting from the input layer, there are parallel frequency domain channel attention (FCA) layers and time domain channel attention (TCA) layers, weight concatenation layer, fusion convolutional layer, and output layer.

[0060] The input layer is used to receive the original feature input;

[0061] The Frequency Domain Channel Attention (FCA) architecture consists of the following layers: a zero-padding layer, a discrete wavelet transform layer, a global average pooling layer, a channel reduction and reshaping layer, and a weight generation network layer. The zero-padding layer pads the height and width of the input feature map to an even number of dimensions to meet the requirements of the Discrete Wavelet Transform (DWT) algorithm. The discrete wavelet transform layer decomposes the spatial feature map into four sub-bands (LL, LH, HL, HH) of different frequencies, transforming the spatial information into the frequency domain to capture global information of different frequency components in the features. The global average pooling layer performs global compression on the feature map of each frequency band channel, reducing the frequency of each channel... The domain feature information is aggregated into a single scalar value to generate a global frequency domain descriptor vector. The channel reduction and reshaping layer is used to aggregate the descriptor vectors generated by DWT with 4 times the number of channels (4C) back to the original number of channels (C) by summing, generating a comprehensive descriptor for each original channel that integrates all its frequency band information. The weight generation network layer is used to input the comprehensive frequency domain descriptor into a small neural network (two 1x1 convolutions) to learn the nonlinear mapping between frequency domain information and channel importance, and outputs a weight vector between 0 and 1 through the Sigmoid activation function, representing the importance of each channel based on frequency domain information.

[0062] The Temporal Channel Attention (TCA) layer has the following structure: parallel global average pooling and global max pooling layers, followed by a weight generation network layer. The global average pooling layer compresses the spatial dimension, extracting the average value of each channel's feature map to form a global context descriptor. The global max pooling layer also compresses the spatial dimension, extracting the maximum value of each channel's feature map. Its function is to capture the most salient feature responses across the entire spatial range, supplementing the average descriptor. The weight generation network comprises two independent one-dimensional convolutional networks. These networks nonlinearly transform the descriptors generated by average pooling and max pooling, respectively, learning the relationship between spatial statistical features and channel importance. The outputs of the two networks are then summed to fuse the information extracted by the two different pooling methods (average features and salient features), thereby generating a more robust attention weight vector, which is then output through a sigmoid activation function.

[0063] The weight splicing layer is used to splice the frequency domain weight vector generated by the FCA path and the spatial domain weight vector generated by the TCA path in the channel dimension to form a composite weight vector that integrates information from both the frequency domain and the spatial domain.

[0064] The fusion convolutional layer described therein is used to adaptively learn how to most effectively fuse information from the frequency and spatial domains. It first reduces the dimensionality of the concatenated high-dimensional weights to save computation and increase nonlinearity, and then projects them back to the original number of channels, finally outputting a set of optimized, comprehensive attention weights.

[0065] From an overall perspective, the MSFDCA-Net network achieves progressive feature optimization through a three-level cascaded FDCA module, with its multi-level time-frequency collaborative feature fusion. The first step is intra-subband optimization, where the feature tensor X output from each subband branch is optimized. i ∈R 32×H×W Each input is a separate FDCA module. This module computes temporal attention (TCA) and frequency domain attention (FCA) in parallel, and generates a collaborative attention graph through an adaptive fusion network. The formula is:

[0066]

[0067] in It is the collaborative attention weight, W T W is a one-dimensional weight vector in the time domain channel. F It is a one-dimensional weight vector in the frequency domain channel.

[0068] Next, the original sub-band features are recalibrated using the following formula:

[0069]

[0070] in X represents the optimized output feature. i Represents the original subband characteristics. It represents the Kronecker product.

[0071] After in-subband optimization, cross-subband fusion is performed. The four optimized subband features are added together to obtain the fused features, which are represented as follows:

[0072]

[0073] Where X fusion Represents the characteristics after fusion. This represents the four optimized sub-band features.

[0074] The fused features are then input into a second FDCA module, which evaluates the contribution of features from different sub-bands and learns their optimal weighted combination, outputting cross-sub-band co-optimized features.

[0075] Finally, high-level semantic enhancement is performed. The features are transformed by an additional 2D convolutional layer (with 32 input channels, 64 output channels, and a 1×6 kernel) to map them to a higher-dimensional space. The output of this convolutional layer is then processed sequentially through a batch normalization layer, a GELU activation function layer, and finally through a random deactivation layer with a dropout ratio of 0.3. The result is then fed into a third FDCA module for final refinement, outputting highly purified features.

[0076] Furthermore, the process of building the MSFDCA-Net network also includes: designing the kernel size, stride, and number of kernels for each layer based on the data size (i.e., time window length) of the input signal of the MSFDCA-Net network and the number of sampling points set within the signal period of each input signal.

[0077] The SSVEP classification method based on time-frequency collaborative channel attention and multi-level fusion proposed in this embodiment achieves signal classification by extracting features in the frequency and time domains of EEG signals. Therefore, for different sampling frequencies, the size of the convolution kernel and stride in the MSFDCA-Net network is designed according to the number of sampling points of the input signal in the same period to extract features in the frequency and time domains of the input signal.

[0078] In this embodiment, the proposed MSFDCA-Net network possesses multi-band spatiotemporal feature fusion capabilities, enabling direct collaborative processing of the time-frequency domain features of multi-channel SSVEP signals. The convolutional layers and frequency-time dual attention modules are core components, jointly responsible for extracting discriminative features from the input data. The quality of the extracted features is closely related to the convolutional kernel configuration, multi-branch structure, and attention weight calculation method; therefore, the overall network architecture is determined based on the multi-band response characteristics of SSVEP signals and the spatiotemporal characteristics of EEG signals. This embodiment utilizes the feature differences in the time, frequency, and spatial dimensions of SSVEP signals induced by different frequency stimuli to achieve frequency category identification. Figure 3As shown, the established MSFDCA-Net network includes: 4 convolutional layers (Conv), 4 batch normalization layers (BN), 4 GELU activation layers, 4 dropout layers, 3 frequency-temporal dual attention modules (FDCALayer), 1 feature fusion module, and 1 softmax classification layer. The cascaded design of the multi-branch convolutional structure and attention module is used to extract deep features from signals of different frequency bands and perform adaptive feature calibration. The frequency-temporal dual attention module significantly enhances the ability to capture key frequency response features by fusing frequency domain analysis of wavelet transform with temporal global pooling. The feature fusion module integrates multi-band information through a weighted fusion strategy to form a discriminative joint feature representation. Finally, the classification layer maps high-dimensional features to the probability distribution of stimulus frequency categories, achieving end-to-end SSVEP frequency recognition. Since EEG signals induced by stimuli of different frequencies exhibit significant differences in time-frequency characteristics, by combining the multi-channel characteristics and frequency band distribution patterns of SSVEP signals, and rationally designing parameters such as the size of the multi-branch convolutional kernel, stride, and dimensionality reduction ratio of the attention module, the time-frequency characteristics of multi-class SSVEP signals can be efficiently extracted. This allows for accurate identification of stimulus targets at different frequencies based on the differences in time-frequency characteristics. In this embodiment, the specific parameters of the MSFDCA-Net network are shown in Table 1.

[0079] Table 1 MSFDCA-Net Network Parameters:

[0080]

[0081] Step 3: Input the training set into the MSFDCA-Net network for training to obtain the trained MSFDCA-Net network; and use the validation set to validate the trained MSFDCA-Net network to obtain the final MSFDCA-Net network.

[0082] Step 3.1: Input the training set EEG data into the MSFDCA-Net network in batches of 64, with each sample containing multi-channel SSVEP signals; use the multi-branch sub-band feature extraction module and the frequency domain-time domain dual attention module (FDCALayer) in the network to process signals of different frequency bands in parallel, extract discriminative spatiotemporal features, and enhance key frequency response features through the attention mechanism;

[0083] In this embodiment, such as Figure 3As shown, for SSVEP multichannel EEG signals with significant frequency domain features, the output of the sub-band feature extraction module can be understood as the spatiotemporal features of the signal in a specific frequency band. The Frequency-Time Dual Attention Module (FDCALayer) is specifically designed to process data with both time and frequency characteristics, effectively mining the frequency domain structure and temporal correlation information contained in the input features. Therefore, the spatiotemporal features extracted from multiple frequency bands of the SSVEP signal are input into the FDCALayer. Based on the extraction of single-band features, wavelet transform can be used to obtain global frequency domain structure information, and an attention weight recalibration mechanism can be used to enhance discriminative features and suppress irrelevant noise, thereby achieving cross-frequency band and cross-spatiotemporal feature optimization and fusion.

[0084] Step 3.2: Input the fusion features extracted in Step 3.1 into the final convolution and attention optimization layer for advanced feature extraction. Map the features to probability distribution vectors of different stimulus frequency categories through a classifier and establish the correspondence between the output and the SSVEP stimulus frequency.

[0085] Step 3.3: Use cross-entropy as the loss function, the formula of which is as follows:

[0086]

[0087] The Adam optimizer is used to perform end-to-end optimization of network parameters, and the learning rate is dynamically adjusted through the ReduceLROnPlateau strategy. The training process lasts for 4000 epochs, and early stopping (training is terminated when the validation set loss no longer decreases for several consecutive epochs) and model checkpointing techniques are used to save the optimal weights. If the model does not converge, the parameters are updated and the process returns to step 3.1 for iterative optimization.

[0088] Step 3.4: Use the saved optimal weight model to perform performance verification on the validation set, and finally evaluate the classification accuracy and information transmission rate on the test set through three-fold cross-validation. All experiments use fixed random seeds to ensure the reproducibility and statistical robustness of the results.

[0089] In this implementation, the training set is input into the MSFDCA-Net network in batches according to the set batch size. Cross-entropy is selected as the loss function, and the Adam optimizer is used to iteratively train the network. The initial learning rate is set to 0.001. During training, if the loss function does not decrease for 10 consecutive rounds, the learning rate is decayed to 0.5 using the ReduceLROnPlateau strategy, and the early stopping patience is set to 30. If the loss function does not decrease for 30 consecutive rounds, the network is considered to have converged, and training is stopped, at which point the trained MSFDCA-Net model is obtained. Otherwise, the network parameters are updated and retraining is performed until a trained model is obtained. Model checkpointing is used to save the network weights with the best performance on the validation set. Finally, the validation set from the sample dataset is input into the trained MSFDCA-Net network, and the network performance is further evaluated using the three-fold cross-validation method. The data is divided into three parts, and two parts are used as the training set and one part as the test set in turn. The average validation result of the three validations is taken as the final classification accuracy, completing the construction of the MSFDCA-Net network and obtaining the final classification model.

[0090] Step 4: Input the test set into the final MSFDCA-Net network to classify the SSVEP signals induced by stimuli of different frequencies in the test set.

[0091] In this embodiment, the test set is input into the MSFDCA-Net network to obtain the classification results of different stimulus frequency categories in the test set. The classification results are compared with the true category labeled for each sample signal in the test set, and the classification accuracy of the network is calculated. Under a time window of 1 second, the average classification accuracy of the network reaches 97.85%, which effectively improves the performance and practicality of the SSVEP brain-computer interface system.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A steady-state visual evoked potential classification method based on time-frequency collaborative channel attention and multi-level fusion, comprising the following steps: Step 1: Collect EEG signals from the subjects at different frequencies as sample signals, preprocess the sample signals to obtain a sample dataset, and divide the sample dataset into a training set, a validation set, and a test set according to a set ratio; Step 2: Construct the MSFDCA-Net network; The structure of the MSFDCA-Net is as follows: starting from the input layer, there are branch-level time-frequency co-channel attention convolutional layers, multi-branch feature fusion layers, final feature extraction convolutional layers, and the output layer. The input layer is used to receive four parallel EEG signals; the branch-level time-frequency co-channel attention convolutional layer is used to extract and enhance the local spatiotemporal features of the signals from the four parallel input branches; and the multi-branch feature fusion layer is used to aggregate and calibrate the four-branch features. The final feature extraction convolutional layer is used to perform deeper transformations and refinements on the fused and calibrated features, and to enhance key features and suppress irrelevant information in both the frequency and time domains. The output layer is used to output the classification of the identified input signals; Step 3: Input the training set into the MSFDCA-Net network for training to obtain the trained MSFDCA-Net network; and use the validation set to validate the trained MSFDCA-Net network to obtain the final MSFDCA-Net network. Step 4: Input the test set into the final MSFDCA-Net network to complete the recognition of different EEG signals in the test set.

2. The steady-state visual evoked potential classification method based on time-frequency collaborative channel attention and multi-level fusion as described in claim 1, characterized in that, Step 1 includes: Step 1.1: The user watches the strobe program at different frequencies in sequence. Each frequency contains multiple trials. According to the international standard 10-20 electrode placement system, the corresponding EEG signal data samples are collected and each sample signal is labeled. Step 1.2: The sample signal is preprocessed and decomposed into four sub-bands: 3-14Hz, 9-26Hz, 14-38Hz, and 19-50Hz. Low-frequency and high-frequency noise, as well as interference from ECG signals and power line frequencies, are removed. The data sampling rate is reduced from 1000Hz to 250Hz through downsampling. Then, DC offset is eliminated through baseline correction to ensure that the zero baseline of the signal is consistent with the reference electrode. Interference signals from electrooculography and electromyography are removed through artifact removal. Finally, the signal is divided into multiple windows of different lengths to form a sample dataset through signal segmentation. Step 1.3: Divide the sample dataset into training set, validation set and test set according to the set ratio, and set the number of sample signals to be processed each time EEG signal recognition is performed.

3. The steady-state visual evoked potential classification method based on time-frequency collaborative channel attention and multi-level fusion as described in claim 2, characterized in that, The structure of the MSFDCA-Net network in step two further includes: The sub-band feature extraction module uses the GELU activation function for all three convolutional layers and the subsequent independent convolutional layer. The four convolutional units are denoted as Convolutional Unit 1, Convolutional Unit 2, Convolutional Unit 3, and Convolutional Unit 4. Convolutional Unit 1 sequentially connects Convolution Operation 1, Batch Normalization Layer 1, GELU activation function 1, and Dropout Layer 1, using 32 convolutional kernels of size (62,1) with a stride of 1. Convolutional Unit 2 sequentially connects Convolution Operation 2, Batch Normalization Layer 2, and GELU activation function 2. Dropout layer 2 uses 32 convolutional kernels of size (1, win_train) with a stride of (5, 5) and padding of (0, 23). Convolutional layer 3 consists of convolution operation 3, batch normalization layer 3, GELU activation function 3, and Dropout layer 3, using 32 convolutional kernels of size (1, 5). Convolutional layer 4 consists of convolution operation 4, batch normalization layer 4, GELU activation function 4, and Dropout layer 4, using 64 convolutional kernels of size (1, 6). The dropout probability of all Dropout layers is 0.

3. The structure of the time-frequency collaborative channel attention module is as follows: starting from the input layer, there are parallel frequency domain channel attention FCA layer and time domain channel attention TCA layer, weight concatenation layer, fusion convolutional layer, and output layer. The input layer is used to receive the original feature input; The frequency domain channel attention (FCA) layer comprises, in sequence, a zero-padding layer, a discrete wavelet transform layer, a global average pooling layer, a channel reduction and reshaping layer, and a weight generation network layer. The zero-padding layer pads the height and width of the input feature map to an even number of dimensions to meet the requirements of the Discrete Wavelet Transform (DWT) algorithm. The discrete wavelet transform layer decomposes the spatial domain feature map into four sub-bands (LL, LH, HL, HH) of different frequencies, transforming the spatial information into the frequency domain, thereby capturing the global information of different frequency components in the features. The global average pooling layer performs global compression on the feature map of each frequency band channel, reducing the frequency domain of each channel... Feature information is aggregated into a single scalar value to generate a global frequency domain descriptor vector. The channel reduction and reshaping layer is used to aggregate the descriptor vectors generated by DWT (4 times the number of channels, 4C) back to the original number of channels (C) by summing, generating a comprehensive descriptor for each original channel that integrates all its frequency band information. The weight generation network layer is used to input the comprehensive frequency domain descriptor into a small neural network (two 1x1 convolutions) to learn the nonlinear mapping between frequency domain information and channel importance, and outputs a weight vector between 0 and 1 through the Sigmoid activation function, representing the importance of each channel based on frequency domain information. The temporal channel attention TCA layer includes parallel global average pooling and global max pooling layers, followed by a weight generation network layer. The global average pooling layer is used to compress the spatial dimension, extracting the average value of each channel feature map to form a global context descriptor. The global max pooling layer is used to compress the spatial dimension, extracting the maximum value of each channel feature map, capturing the most significant feature response in the entire spatial range as a supplement to the average descriptor. The weight generation network contains two independent one-dimensional convolutional networks, which perform nonlinear transformations on the descriptors generated by average pooling and max pooling, respectively, to learn the relationship between spatial statistical features and channel importance. Then, the outputs of the two networks are added together to fuse the information extracted by the two different pooling methods (average features and salience features), thereby generating a more robust attention weight vector, which is output through a sigmoid activation function. The weight splicing layer is used to splice the frequency domain weight vector generated by the FCA path and the spatial domain weight vector generated by the TCA path in the channel dimension to form a composite weight vector that integrates information from both the frequency domain and the spatial domain. The fusion convolutional layer is used to adaptively learn how to most effectively fuse information from the frequency domain and the spatial domain. First, the high-dimensional weights of the concatenation are reduced in dimensionality to save computation and increase nonlinearity. Then, they are projected back to the original number of channels, and finally output a set of optimized, comprehensive attention weights.

4. The steady-state visual evoked potential classification method based on time-frequency collaborative channel attention and multi-level fusion as described in claim 3, characterized in that, The formula for the GELU activation function is as follows: Where x represents the function input value and is also the only variable, and tanh represents the hyperbolic tangent activation function.

5. The steady-state visual evoked potential classification method based on time-frequency collaborative channel attention and multi-level fusion as described in claim 3, characterized in that, The MSFDCA-Net network achieves progressive feature optimization through a three-level cascaded FDCA module, employing multi-level time-frequency collaborative feature fusion. The first step is intra-subband optimization, where the feature tensor X output from each subband branch is optimized. i ∈R 32×H×W Independent FDCA modules are input separately, and time-domain attention (TCA) and frequency-domain attention (FCA) are computed in parallel. A collaborative attention graph is then generated through an adaptive fusion network. The formula is: in It is the collaborative attention weight, W T W is a one-dimensional weight vector in the time domain channel. F It is a one-dimensional weight vector in the frequency domain channel; Next, the original sub-band features are recalibrated using the following formula: in X represents the optimized output feature. i Represents the original subband characteristics. Indicates the Kronecker product; After in-subband optimization, cross-subband fusion is performed. The four optimized subband features are added together to obtain the fused features, which are represented as follows: Where X fusion Represents the characteristics after fusion. These represent the four optimized sub-band features; The fused features are input into the second FDCA module to evaluate the contribution of features from different sub-bands and learn their optimal weighted combination, outputting the cross-sub-band co-optimized features. Finally, high-level semantic enhancement is performed. An additional 2D convolutional layer with 32 input channels, 64 output channels, and a 1×6 kernel transforms the features, mapping them to a higher-dimensional space. The output of this convolutional layer is then processed sequentially through a batch normalization layer, a GELU activation function layer, and finally a random deactivation layer with a dropout ratio of 0.

3. The result is then fed into a third FDCA module for final refinement, outputting highly purified features.

6. The steady-state visual evoked potential classification method based on time-frequency collaborative channel attention and multi-level fusion as described in claim 3, characterized in that, The process of building the MSFDCA-Net network also includes: designing the kernel size, stride, and number of kernels for each layer based on the data size (i.e., time window length) of the input signal of the MSFDCA-Net network and the number of sampling points set within the signal period of each input signal.

7. The steady-state visual evoked potential classification method based on time-frequency collaborative channel attention and multi-level fusion as described in claim 6, characterized in that, Step 3 includes: Step 3.1: Input the training set EEG data into the MSFDCA-Net network in batches of 64, with each sample containing multi-channel SSVEP signals; use the multi-branch sub-band feature extraction module and the frequency domain-time domain dual attention module FDCALayer in the network to process signals of different frequency bands in parallel, extract discriminative spatiotemporal features, and enhance key frequency response features through the attention mechanism; Step 3.2: Input the fusion features extracted in Step 3.1 into the final convolution and attention optimization layer for advanced feature extraction. Map the features to probability distribution vectors of different stimulus frequency categories through a classifier and establish the correspondence between the output and the SSVEP stimulus frequency. Step 3.3: Using cross-entropy as the loss function, the Adam optimizer is used to perform end-to-end optimization of the network parameters, and the learning rate is dynamically adjusted using the ReduceLROnPlateau strategy; the training process lasts for 4000 epochs, and early stopping (training is terminated when the validation set loss no longer decreases for several consecutive epochs) and model checkpointing techniques are used to save the optimal weights; if the model does not converge, the parameters are updated and the process returns to step 3.1 for iterative optimization; Step 3.4: Use the saved optimal weight model to perform performance verification on the validation set, and finally evaluate the classification accuracy and information transmission rate on the test set through three-fold cross-validation. All experiments use fixed random seeds to ensure the reproducibility and statistical robustness of the results.

Images