Electroencephalogram auditory attention classification method and system based on time-frequency attention mechanism

Abstract

The application discloses an electroencephalogram auditory attention classification method and system based on a time-frequency attention mechanism, and belongs to the field of pattern recognition and biomedical signal processing. Specifically, first, the original electroencephalogram signal is subjected to multi-band filtering, re-referencing, multi-band electroencephalogram splicing and windowed slicing preprocessing. Then, different weights are assigned to different frequency bands in the multi-band electroencephalogram signal, the multi-band electroencephalogram is fused into a two-dimensional electroencephalogram signal, the frequency domain features of the electroencephalogram signal are extracted, and the time sequence features of the electroencephalogram signal are extracted. Finally, the electroencephalogram signal with the extracted frequency domain features and time domain features is used to extract deep auditory attention features in the electroencephalogram signal by using a lightweight residual neural network, and auditory attention classification is performed. Experimental results show that the method can extract more discriminative features from the original electroencephalogram signal, and in the execution of auditory attention classification, 96.5% and 76.3% average classification accuracies are achieved in the within-subject and cross-subject experiments under a one-second decision window, and the classification effect is obviously improved compared with other methods.

Description

Technical Field

[0001] The present invention belongs to the field of pattern recognition and biomedical signal processing, and specifically relates to an EEG auditory attention classification method and system based on a time-frequency attention mechanism. Background Art

[0002] In a noisy environment with multiple sound sources, the brain's selective attention allows humans to consistently track one sound source, a phenomenon known as the "cocktail party effect." Neuroscience research has shown that the brain's selective attention can be decoded from human brain signals. In recent years, with the continuous development of artificial intelligence, many researchers have begun using deep learning methods to process EEG signals for selective attention decoding, a process known as auditory attention detection. Auditory attention detection based on EEG signals can be divided into two categories: speaker recognition, which aims to identify the sound that a subject selectively focuses on in a multi-sound source environment; and spatial attention detection, which aims to detect the spatial orientation of a subject's attention.

[0003] Research in the field of speaker recognition has used the cross-attention mechanism and the spatial-band attention mechanism combined with the frequency-band attention mechanism in neural networks to build classification models. The accuracy rates for a 2s decision window are approximately 90%-96%, and for a 1s decision window, approximately 90%-94%. These studies all used clean speech envelopes as model input, but this is not practical in real-world scenarios, as only mixed speech signals are available.

[0004] In the field of spatial attention detection, researchers have used convolutional neural network models to decode the spatial location of participants' attention without using the speech envelope. The results showed an accuracy of approximately 80.8% for a 1-second decision window. Zhang et al. employed a multi-task learning model, supplemented by an envelope reconstruction task, for direct auditory attention classification, achieving an accuracy of approximately 82% for a 2-second decision window. These approaches attempt to directly classify auditory attention without using the speech envelope. However, they fail to consider the complex nature of EEG signals, which contain diverse information across frequency bands and a significant amount of temporal information, thus failing to effectively improve classification results. Furthermore, the shallow convolutional neural networks (CNNs) used in these approaches have limitations that restrict the model's ability to fully learn the effective information from EEG signals. Specifically, due to their simple structure, shallow CNNs are unable to deeply exploit the complex features in EEG data. Furthermore, while pooling layers reduce feature dimensionality, they can lose valuable information, making it difficult for the model to effectively represent subtle differences between EEG signals focused on different spatial locations. These factors directly affect the overall performance of the model, especially in cross-subject experiments, because the EEG signal characteristics of different individuals may vary significantly, and shallow CNNs lack sufficient capacity to adapt to such individual differences, resulting in decreased classification accuracy. Summary of the Invention

[0005] The purpose of the present invention is to provide an EEG auditory attention classification method and system based on the time-frequency attention mechanism, so as to solve the problem that the original EEG signal has weak representation ability and cannot effectively improve the classification effect.

[0006] To achieve the purpose of the present invention, the technical solutions provided by the present invention are as follows:

[0007] First aspect

[0008] This application provides an EEG auditory attention classification method based on the time-frequency attention mechanism, comprising the following steps:

[0009] Step S1: obtaining EEG signals;

[0010] Step S2: filtering the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information;

[0011] Step S3: re-referencing the multi-band EEG signal to eliminate interference information in the EEG signal;

[0012] Step S4: splicing the re-referenced multi-band EEG signals and performing window slicing operations to input them into the neural network;

[0013] Step S5: assigning different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to enhance the frequency band information that is useful for representing auditory attention;

[0014] Step S6: fusing the weighted multi-band EEG signals into a two-dimensional EEG signal, extracting the frequency domain features of the EEG signal, and outputting the EEG signal with the extracted frequency domain features;

[0015] Step S7: extracting time series features from the EEG signal whose frequency domain features have been extracted, and outputting the EEG signal whose time series features have been extracted;

[0016] Step S8: Classify the EEG signal from which the time series features have been extracted.

[0017] Second aspect

[0018] Corresponding to the above method, the present application provides an EEG auditory attention classification system based on the time-frequency attention mechanism, which includes the following modules: a signal acquisition module, a filtering module, a re-reference module, a splicing and slicing module, a frequency band attention module, a multi-band EEG fusion module, a time attention module, and a classification module;

[0019] The signal acquisition module is used to acquire EEG signals;

[0020] The filtering module is used to filter the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information;

[0021] The re-reference module is used to re-reference the multi-band EEG signal to eliminate interference information in the EEG signal;

[0022] The splicing and slicing module is used to splice the re-referenced multi-band EEG signals and perform window slicing operations so as to input them into the neural network;

[0023] The frequency band attention module is used to assign different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, thereby enhancing the frequency band information that is beneficial to the representation of auditory attention;

[0024] The multi-band EEG fusion module is used to fuse the weighted multi-band EEG signals into a two-dimensional EEG signal, extract the frequency domain features of the EEG signal, and output the EEG signal with the extracted frequency domain features;

[0025] The temporal attention module is used to extract time series features from the EEG signal from which frequency domain features have been extracted, and output the EEG signal from which time series features have been extracted;

[0026] The classification module is used to classify the EEG signals from which time series features have been extracted.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] Compared with simple feature selection, the frequency band attention module and the time attention module are introduced into the model, relying on the learning ability of the network to obtain the attention of different frequency bands and different times of the current sample. It does not require too much manual intervention and relies entirely on the generalization ability of the model. At the same time, the use of deep convolutional neural networks in the classification module can make the model have a stronger ability to capture auditory attention features, solving the problem that the existing technology is insufficient in extracting the time-frequency information of the original EEG signal, the original EEG information is not strong in representation, and it cannot effectively improve the auditory attention classification effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a method flow chart provided in an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of the system framework provided by an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of a frequency band attention module in an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of the temporal attention module in an embodiment of the present invention.

[0033] Figure 5 Schematic diagram of a classification module in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment provides an EEG auditory attention classification method based on the time-frequency attention mechanism, comprising the following steps:

[0037] Step S1: obtaining EEG signals;

[0038] Step S2: filtering the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information;

[0039] Step S3: re-referencing the multi-band EEG signal to eliminate interference information in the EEG signal;

[0040] Step S4: splicing the re-referenced multi-band EEG signals and performing window slicing operations to input them into the neural network;

[0041] Step S5: assigning different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to enhance the frequency band information that is useful for representing auditory attention;

[0042] Step S6: fusing the weighted multi-band EEG signals into a two-dimensional EEG signal, extracting the frequency domain features of the EEG signal, and outputting the EEG signal with the extracted frequency domain features;

[0043] Step S7: extracting time series features from the EEG signal whose frequency domain features have been extracted, and outputting the EEG signal whose time series features have been extracted;

[0044] Step S8: Classify the EEG signal from which the time series features have been extracted.

[0045] Preferably, in step S2, the EEG signal is filtered to extract a multi-band EEG signal for representing auditory attention information, specifically:

[0046] Filters were used to extract the five frequency bands of EEG signals: δ (1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (12-30 Hz), and γ (30-50 Hz).

[0047] Specifically, the filter adopts a Butterworth bandpass filter, and the transfer function of the Butterworth filter is shown in the following formula:

[0048]

[0049] Where n is the order of the filter, ω c is the cutoff frequency, ω p is the passband edge frequency;

[0050] According to the characteristics of the bandpass filter, different upper and lower cutoff frequencies need to be set when extracting different frequency bands, so that only signals in specific frequency bands are allowed to pass.

[0051] Preferably, in step S3, the multi-band EEG signal is re-referenced, specifically:

[0052] performing average re-referencing on the multi-band EEG signals;

[0053] Assume that each frequency band of the multi-band EEG signal is X i (t), where i is the i-th electrode and t is the sampling point of the current electrode at time t. The average re-referenced EEG signal is shown as follows:

[0054]

[0055] Where N is the number of electrodes, Y i(t) is the EEG signal after re-reference, X i (t) is the original EEG signal.

[0056] Preferably, the step S4 specifically includes:

[0057] The obtained EEG signals of the five frequency bands are spliced ​​along the third dimension to obtain the multi-band EEG data X∈R C×T×B , where C is the number of electrodes, T is the sampling point, and B is the number of frequency bands;

[0058] Then, split the spliced ​​EEG data into segments of a set length. You can choose a time length of 1 second for segmentation, specifically:

[0059] A 1-second sliding window is set to perform non-overlapping segmentation on the EEG data. The resulting sample format is: number of channels × sampling points × number of frequency bands, where the number of channels is 64, the number of sampling points is sampling rate × time, which is 128 sampling points here, and the number of frequency bands is 5.

[0060] Preferably, the step S5 specifically includes:

[0061] For the segmented multi-band EEG samples F∈R C×T×B , convolution is performed along the channel dimension, and the 2D feature map obtained after convolution is shown as follows:

[0062] F s ∈R B×T

[0063] F s Represents the feature map after convolution, whose output dimension is B×T;

[0064] The obtained output is activated using the Elu function. The calculation formula of Elu is as follows:

[0065]

[0066] Among them, alpha is an optional parameter, usually the default value is 1.0;

[0067] The activated feature map is average pooled along the time dimension. The feature map obtained by average pooling is shown in the following formula:

[0068] F sp ∈R B×1

[0069] The pooled feature map is connected to two fully connected layers. Each fully connected layer is activated using the Tanh activation function to explore the relationship between different frequency bands and finally obtain the frequency band attention mask. The Tanh activation function formula is shown as follows:

[0070]

[0071] The attention mask obtained using full connection is shown as follows:

[0072] Mask∈R 1×B

[0073] Use the obtained band attention mask Mask∈R 1×B , assigning weights to different sub-bands of the EEG signal.

[0074] Preferably, the step S6 specifically includes:

[0075] The output feature map is convolved, and the feature map obtained after convolution is shown in the following formula:

[0076] F t ∈R B×T

[0077] The obtained feature map is then activated using the Relu function;

[0078] The Relu activation function formula is as follows:

[0079]

[0080] Finally, the feature map is averaged along the time dimension. The feature map obtained after average pooling is shown in the following formula:

[0081] F tp ∈R B×32 .

[0082] Preferably, in step S7, a self-attention mechanism is used to extract temporal features from the EEG signal. Specifically, the self-attention calculation formula is as follows:

[0083]

[0084] Among them, Q, K, and V represent query, key, and value matrices, respectively, which are obtained by mapping EEG data three times. k Represents the dimension of the key.

[0085] Preferably, the step S8 specifically includes:

[0086] The extracted frequency domain features are combined with the time series features to enhance the auditory attention detection effect and complete the auditory attention classification task. Specifically, it includes:

[0087] For the EEG samples F∈R that have been extracted from the temporal features C×T , input into the lightweight residual neural network for auditory attention feature extraction, and the resulting 2D feature map is shown in the following formula:

[0088] F s ∈R 512×1

[0089] F s Represents the feature map after convolution, whose output dimension is 512×1;

[0090] Next, a fully connected layer is used to prepare for classification. In a fully connected layer, each neuron in each layer is connected to all neurons in the previous layer, but neurons in the same layer have no connections between each other. The function of a fully connected layer is to integrate the information from all feature maps obtained by the network before the output layer, outputting it as a feature vector.

[0091] Finally, a Sigmoid classifier is used. The output of the dense neural layer is connected, and the Sigmoid output is set to 2. The Sigmoid function is used to map the resulting feature vector to a range of 0-1, which is the probability of each category classification output by the neuron. The model uses the result with the highest probability output by the neuron as the final recognition result. The Sigmoid calculation formula is as follows:

[0092]

[0093] Example 2

[0094] like Figure 2 As shown, this embodiment provides an EEG auditory attention classification system based on the time-frequency attention mechanism, including the following modules: a signal acquisition module, a filtering module and a re-reference module, a splicing and slicing module, a frequency band attention module, a multi-band EEG fusion module, a time attention module, and a classification module;

[0095] The signal acquisition module is used to acquire EEG signals;

[0096] The filtering module is used to filter the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information;

[0097] The re-reference module is used to re-reference the multi-band EEG signal to eliminate interference information in the EEG signal;

[0098] The splicing and slicing module is used to splice the re-referenced multi-band EEG signals and perform window slicing operations so as to input them into the neural network;

[0099] The frequency band attention module is used to assign different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, thereby enhancing the frequency band information that is beneficial to the representation of auditory attention;

[0100] The multi-band EEG fusion module is used to fuse the weighted multi-band EEG signals into a two-dimensional EEG signal, extract the frequency domain features of the EEG signal, and output the EEG signal with the extracted frequency domain features;

[0101] The temporal attention module is used to extract time series features from the EEG signal from which frequency domain features have been extracted, and output the EEG signal from which time series features have been extracted;

[0102] The classification module is used to classify the EEG signals from which time series features have been extracted.

[0103] It should be noted that the overall framework of this embodiment is divided into two parts.

[0104] The first part is the EEG signal preprocessing part. The preprocessing part uses the signal acquisition module, filtering module, re-reference module, and splicing and slicing module to perform multi-band filtering, re-reference, multi-band EEG splicing, and window slicing on the EEG signal in sequence.

[0105] The second part is the EEG auditory attention classification network based on the time-frequency attention mechanism. This part uses the frequency band attention module to assign different weights to EEGs of different frequency bands to strengthen the weight of useful information; uses the multi-band EEG fusion module to extract the frequency domain features of EEG signals; uses the time attention module to obtain complex time series features; uses the classification module to first input into the lightweight residual neural network for auditory attention feature extraction, and then input into the Sigmoid classifier for auditory attention detection.

[0106] Preferably, the filtering module is used to filter the EEG signal and extract a multi-band EEG signal for characterizing auditory attention information, specifically: using a filter to extract the five frequency bands of the EEG signal δ (1-4Hz), θ (4-8Hz), α (8-12Hz), β (12-30Hz), and γ (30-50Hz).

[0107] Among them, the Butterworth bandpass filter is used, and the transfer function of the Butterworth filter is shown as follows:

[0108]

[0109] Where n is the order of the filter, ω c is the cutoff frequency, ω p is the passband edge frequency.

[0110] According to the characteristics of the bandpass filter, different upper and lower cutoff frequencies need to be set when extracting different frequency bands, so that only signals in specific frequency bands are allowed to pass.

[0111] Preferably, the re-reference module is used to perform average re-reference on the multi-band EEG signals to eliminate interference information in the EEG signals.

[0112] Specifically, let each frequency band of the multi-band EEG signal be X i (t), where i is the i-th electrode and t is the sampling point of the current electrode at time t. The average re-referenced EEG signal is shown as follows:

[0113]

[0114] Where N is the number of electrodes, Y i (t) is the EEG signal after re-reference, X i (t) is the original EEG signal.

[0115] Preferably, the slicing splicing module is specifically used to:

[0116] The obtained EEG signals of the five frequency bands are spliced ​​along the third dimension to obtain the multi-band EEG data X∈R C×T×B , where C is the number of electrodes, T is the sampling points, and B is the number of frequency bands.

[0117] The EEG data is then divided into segments of a set length. Here, a 1-second time length is selected for illustration. Specifically:

[0118] A 1-second sliding window is set to perform non-overlapping segmentation on the EEG data. The resulting sample format is: number of channels × sampling points × number of frequency bands, where the number of channels is 64, the number of sampling points is sampling rate × time, which is 128 sampling points here, and the number of frequency bands is 5.

[0119] The EEG auditory attention classification network based on the time-frequency attention mechanism consists of four modules: frequency band attention module, multi-band EEG fusion module, temporal attention module and classification module.

[0120] The band attention module consists of 3D convolution, mean pooling, and two fully connected layers. The specific parameters of the band attention module are shown in Table 1.

[0121] Table 1 Specific parameters of the frequency band attention module

[0122]

[0123] The multi-band EEG fusion module consists of a convolutional layer and a pooling layer. The specific parameters of the multi-band EEG fusion module are shown in Table 2.

[0124] Table 2 Specific parameters of multi-band EEG fusion module

[0125]

[0126] The classification module consists of multiple convolutional layers, pooling layers, and residual connections. The specific parameters of the classification module are shown in Table 3.

[0127] Table 3 Specific parameters of the auditory attention feature extraction module

[0128]

[0129]

[0130] The frequency band attention module is used to assign different weights to different frequency bands in the multi-band EEG signal, thereby enhancing the frequency band information that is beneficial for characterizing auditory attention. Specifically,

[0131] The frequency band attention module can make up for the deficiency of convolutional neural network in extracting the frequency domain features of EEG signals and improve the suppression of irrelevant frequency domain information. Figure 3 As shown in Figure 2, the principle of the frequency band attention mechanism is to generate a frequency band attention mask by analyzing the internal relationships of the feature maps obtained by the CNN. Frequency band attention focuses on the "where" of the effective information in the feature map. The frequency band attention module enables the model to obtain more representative sub-bands from the EEG signal for auditory attention classification.

[0132] In the frequency band attention module, the sample is sequentially passed through a 3D convolution layer, an Elu activation function, a mean pooling layer, and two fully connected layers.

[0133] For the segmented multi-band EEG samples F∈R C×T×B , convolution is performed along the channel dimension, and the 2D feature map obtained after convolution is shown as follows:

[0134] F s ∈R B×T

[0135] F s Represents the feature map after convolution, whose output dimension is B×T;

[0136] The obtained output is activated using the Elu function. The calculation formula of Elu is as follows:

[0137]

[0138] Among them, alpha is an optional parameter and usually takes a default value of 1.0.

[0139] The activated feature map is average pooled along the time dimension. The feature map obtained by average pooling is shown in the following formula:

[0140] F sp ∈R B×1

[0141] The pooled feature map is connected to two fully connected layers. Each fully connected layer is activated using the Tanh activation function to explore the relationship between different frequency bands and finally obtain the frequency band attention mask. The Tanh activation function formula is shown as follows:

[0142]

[0143] The attention mask obtained using full connection is shown as follows:

[0144] Mask∈R 1×B

[0145] Use the obtained band attention mask Mask∈R 1×B , assigning weights to different sub-bands of the EEG signal.

[0146] The multi-band EEG fusion module is used to fuse the multi-band EEG into a two-dimensional EEG signal and extract the frequency domain features of the EEG signal, specifically including:

[0147] After the frequency band attention module, each sub-band with dynamically assigned weights is obtained, which will help this module to better extract the frequency domain features in the EEG signal and better complete the EEG auditory attention classification task.

[0148] In the multi-band EEG fusion module, the samples are sequentially passed through a 2D convolution layer, a ReLU activation function, and a mean pooling layer.

[0149] The feature map output by the frequency band attention module is convolved, and the feature map obtained after convolution is shown in the following formula:

[0150] F t ∈R B×T

[0151] The obtained feature map is then activated using the Relu function.

[0152] The Relu activation function formula is as follows:

[0153]

[0154] Finally, the feature map is averaged along the time dimension. The feature map obtained after average pooling is shown in the following formula:

[0155] F tp ∈R B×32

[0156] The temporal attention module is used to extract the temporal features of EEG signals, including:

[0157] Psychoacoustic research provides ample evidence that human attention itself is a dynamic and time-based activity, and the auditory system is very sensitive to time. Therefore, a temporal attention module is introduced in this scheme to effectively extract the complex temporal features in EEG signals, focusing on the auditory information expressed by the EEG signals at specific time nodes.

[0158] The EEG signal that has undergone frequency domain feature extraction is further extracted with time dimension time series features. In this solution, the self-attention mechanism is used to extract the time series features of the EEG signal. The implementation method is as follows: Figure 4 shown.

[0159] The self-attention mechanism mainly uses scaled dot product attention. Its input consists of three matrices: Q (Queries), K (Key), and V (Value). The output is a weighted sum of the similarity and value of Q and K.

[0160] The self-attention calculation formula is as follows:

[0161]

[0162] Among them, Q, K, and V represent query, key, and value matrices, respectively, which are obtained by mapping EEG data three times. k Represents the dimension of the key.

[0163] The classification module is used to combine the extracted frequency domain features with the time series features to enhance the auditory attention detection effect and complete the auditory attention classification task. Specifically, it includes:

[0164] The lightweight residual network model provided in this embodiment is used to extract the features of EEG signals to realize auditory attention detection. Figure 5 As shown in the figure. This model is based on the ResNet architecture, but is simplified to use only two layers of residual connections, which reduces the complexity of the model and the number of parameters while retaining the advantages of ResNet in solving the gradient vanishing and gradient exploding problems. In this simplified structure, when the input is x, the learned feature is H(x). We hope that the network will learn the residual F(x), whose formula is F(x) = H(x) - x, so the actual output of the network is F(x) + x. Even if the residual is 0, the network still performs the identity mapping, which does not affect performance. However, through non-zero residuals, the stacked layers can further learn new features based on the input features, improving the accuracy and efficiency of the model.

[0165] Next, a fully connected layer is used to prepare for classification. In a fully connected layer, each neuron in each layer is connected to all neurons in the previous layer, but neurons in the same layer have no connections between each other. The function of a fully connected layer is to integrate the information from all feature maps obtained by the network before the output layer, outputting it as a feature vector.

[0166] Finally, a Sigmoid classifier is used. The output of the dense neural layer is connected, and the Sigmoid output is set to 2. The Sigmoid function is used to map the resulting feature vector to a range of 0-1, which is the probability of each category classification output by the neuron. The model uses the result with the highest probability output by the neuron as the final recognition result. The Sigmoid calculation formula is as follows:

[0167]

[0168] It should be noted that the EEG auditory attention classification method based on the time-frequency attention mechanism of this application uses EEG signals collected by the BioSemi ActiveTwo device to perform auditory attention classification tasks in a laboratory scenario.

[0169] Here is an introduction to this dataset: The dataset used contains EEG data of 16 normal hearing subjects who were asked to selectively focus on one of two speakers speaking simultaneously. The speech stimuli consisted of four Dutch stories told by three male Flemish speakers. The stimuli were presented binaurally (one speaker per ear) or filtered by head-related transfer function (HRTF) to simulate speech coming from 90° to the left and 90° to the right. Throughout the experiment, the order of condition presentation was randomized for different subjects, and the number of experiments focusing on the left and right ears was equal. 64-channel EEG signals were recorded using a BioSemi ActiveTwo device with a sampling rate of 8192Hz. A total of 8×6 minutes of EEG data were collected for each subject, for a total of 12.8 hours of EEG data for the 16 subjects.

[0170] The neural network model needs to be trained with samples in advance before it can be used for auditory attention classification.

[0171] Specifically, the training process includes:

[0172] Within-subject experiments: Each subject's EEG data was divided using a four-fold cross-validation method. Specifically, each subject's eight experiments (four focusing on the left side and four focusing on the right side) were divided into four groups, each containing one experiment focusing on the left side and one experiment focusing on the right side. One of these groups was selected as the test set, and the remaining three groups were used as training sets. The data of all 16 subjects were divided using this method. During training, the training data of all subjects were mixed together, and during testing, the test data of all subjects were mixed together.

[0173] Cross-subject experiment: In this cross-subject experiment, a 15-fold cross-validation method was used to partition the data. Each time, one of the 16 subjects was selected as the test set, and the data of the remaining 15 subjects was used as the training set. This cycle was repeated to ensure that each subject's data was used as the test set for validation. This experiment was used to test the model's ability to generalize across different subjects.

[0174] The training process uses the stochastic gradient descent (SGD) optimizer. The model is trained over 200 epochs. For the first 25 epochs, the learning rate is set to 0.01, and then reduced to 0.001 for the next 175 epochs. The batch_size is set to 256 to ensure that each training batch contains enough data samples for effective gradient updates.

[0175] To verify the effectiveness of the model, the proposed method was compared with different models, including a convolutional neural network (CNN) and STA-NET. The comparison results of different models are shown in Tables 4 and 5.

[0176] Table 4 shows the comparison results of different models within the subjects.

[0177]

[0178] Table 5 shows the comparison results of different models across subjects.

[0179]

[0180] As can be seen from Tables 4 and 5, compared with the CNN and STA-NET methods, the method proposed in this application has the highest accuracy in different time windows in both within-subject and across-subject experiments.

[0181] The above describes in detail the optional implementation methods of the embodiments of the present invention in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation methods. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical solutions of the embodiments of the present invention, and these simple modifications all fall within the scope of protection of the embodiments of the present invention.

Claims

1. A method for EEG auditory attention classification based on time-frequency attention mechanism, characterized in that: The following steps are involved: Step S1: obtaining EEG signals; Step S2: filtering the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information; Step S3: re-referencing the multi-band EEG signal to eliminate interference information in the EEG signal; Step S4: splicing the re-referenced multi-band EEG signals and performing window slicing operations to input them into the neural network; Step S5: assigning different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to enhance the frequency band information that is useful for representing auditory attention; Step S6: fusing the weighted multi-band EEG signals into a two-dimensional EEG signal, extracting the frequency domain features of the EEG signal, and outputting the EEG signal with the extracted frequency domain features; Step S7: extracting time series features from the EEG signal whose frequency domain features have been extracted, and outputting the EEG signal whose time series features have been extracted; Step S8: Classify the EEG signal from which the time series features have been extracted.

2. The EEG auditory attention classification method based on the time-frequency attention mechanism according to claim 1 is characterized in that: In step S2, the EEG signal is filtered to extract a multi-band EEG signal for representing auditory attention information, specifically: Filters were used to extract the five frequency bands of EEG signals: δ (1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (12-30 Hz), and γ (30-50 Hz).

3. The EEG auditory attention classification method based on the time-frequency attention mechanism according to claim 2 is characterized in that: The filter adopts a Butterworth bandpass filter, and the transfer function of the Butterworth filter is shown as follows: Where n is the order of the filter, ω c is the cutoff frequency, ω p is the passband edge frequency; According to the characteristics of the bandpass filter, different upper and lower cutoff frequencies need to be set when extracting different frequency bands, so that only signals in specific frequency bands are allowed to pass.

4. The EEG auditory attention classification method based on the time-frequency attention mechanism according to claim 3 is characterized in that: In step S3, the multi-band EEG signal is re-referenced, specifically: performing average re-referencing on the multi-band EEG signals; Assume that each frequency band of the multi-band EEG signal is X i (t), where i is the i-th electrode and t is the sampling point of the current electrode at time t. The average re-referenced EEG signal is shown as follows: Where N is the number of electrodes, Y i (t) is the EEG signal after re-reference, X i (t) is the original EEG signal.

5. The EEG auditory attention classification method based on the time-frequency attention mechanism according to claim 4 is characterized in that: The step S4 specifically includes: The obtained EEG signals of the five frequency bands are spliced ​​along the third dimension to obtain the multi-band EEG data X∈R C ×T×B , where C is the number of electrodes, T is the sampling point, and B is the number of frequency bands; Then, the spliced ​​EEG data is segmented into segments of a set length, and a time length of 1 second is selected for segmentation, specifically: A 1-second sliding window is set to perform non-overlapping segmentation on the EEG data. The resulting sample format is: number of channels × sampling points × number of frequency bands, where the number of channels is 64, the number of sampling points is sampling rate × time, which is 128 sampling points here, and the number of frequency bands is 5.

6. The EEG auditory attention classification method based on the time-frequency attention mechanism according to claim 5 is characterized in that: The step S5 specifically includes: For the segmented multi-band EEG samples F∈R C×T×B , convolution is performed along the channel dimension, and the 2D feature map obtained after convolution is shown as follows: F s ∈R B×T F s Represents the feature map after convolution, whose output dimension is B×T; The obtained output is activated using the Elu function. The calculation formula of Elu is as follows: Among them, alpha is an optional parameter, usually the default value is 1.0; The activated feature map is average pooled along the time dimension. The feature map obtained by average pooling is shown in the following formula: F sp ∈R B×1 The pooled feature map is connected to two fully connected layers. Each fully connected layer is activated using the Tanh activation function to explore the relationship between different frequency bands and finally obtain the frequency band attention mask. The Tanh activation function formula is shown as follows: The attention mask obtained using full connection is shown as follows: Mask∈R 1×B Use the obtained band attention mask Mask∈R 1×B , assigning weights to different sub-bands of the EEG signal.

7. The EEG auditory attention classification method based on the time-frequency attention mechanism according to claim 6 is characterized in that: The step S6 specifically includes: The output feature map is convolved, and the feature map obtained after convolution is shown in the following formula: F t ∈R B×T The obtained feature map is then activated using the Relu function; The Relu activation function formula is as follows: Finally, the feature map is averaged along the time dimension. The feature map obtained after average pooling is shown in the following formula: F tp ∈R B×32 。 8. The EEG auditory attention classification method based on the time-frequency attention mechanism according to claim 7 is characterized in that: In step S7, a self-attention mechanism is used to extract temporal features from the EEG signal.

9. The EEG auditory attention classification method based on the time-frequency attention mechanism according to claim 8 is characterized in that: The self-attention calculation formula is as follows: Among them, Q, K, and V represent query, key, and value matrices, respectively, which are obtained by mapping EEG data three times. k Represents the dimension of the key.

10. An EEG auditory attention classification system based on time-frequency attention mechanism, characterized in that: It includes the following modules: signal acquisition module, filtering module, re-reference module, splicing and slicing module, frequency band attention module, multi-band EEG fusion module, time attention module, and classification module; The signal acquisition module is used to acquire EEG signals; The filtering module is used to filter the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information; The re-reference module is used to re-reference the multi-band EEG signal to eliminate interference information in the EEG signal; The splicing and slicing module is used to splice the re-referenced multi-band EEG signals and perform window slicing operations so as to input them into the neural network; The frequency band attention module is used to assign different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, thereby enhancing the frequency band information that is beneficial to the representation of auditory attention; The multi-band EEG fusion module is used to fuse the weighted multi-band EEG signals into a two-dimensional EEG signal, extract the frequency domain features of the EEG signal, and output the EEG signal with the extracted frequency domain features; The temporal attention module is used to extract time series features from the EEG signal from which frequency domain features have been extracted, and output the EEG signal from which time series features have been extracted; The classification module is used to classify the EEG signals from which time series features have been extracted.

Images