EEG auditory attention classification method and system based on time-frequency attention mechanism

Abstract

The invention discloses an electroencephalogram auditory attention classification method and system based on a time-frequency attention mechanism, and belongs to the field of pattern recognition and biomedical signal processing. Specifically, firstly, multi-band filtering, re-reference, multi-band electroencephalogram splicing and windowing slicing preprocessing are carried out on original electroencephalogram signals. Then different weights are distributed to different frequency bands in the multi-band electroencephalogram signals, the multi-band electroencephalogram signals are fused into two-dimensional electroencephalogram signals, frequency domain features of the electroencephalogram signals are extracted, and time sequence features of the electroencephalogram signals are extracted; and finally, extracting deep auditory attention features in the electroencephalogram signals from which the frequency domain features and the time domain features are extracted by using a lightweight residual neural network, and carrying out auditory attention classification. Experimental results show that the method can extract more discriminative features from the original electroencephalogram signals, in the execution of auditory attention classification, the average classification accuracy of 96.5% and 76.3% is obtained in a subject and a cross-subject experiment under a one-second decision window, and compared with other methods, the classification effect is obviously improved.

Description

Technical Field

[0001] The present invention belongs to the field of pattern recognition and biomedical signal processing, and in particular relates to an electroencephalogram (EEG) auditory attention classification method and system based on a time-frequency attention mechanism. Background Art

[0002] In a noisy environment with multiple sound sources, the brain's selective attention allows humans to keep tracking one of the sound sources. This phenomenon is described as the "cocktail party effect". Neuroscience research shows that the brain's selective attention can be decoded from human brain signals. In recent years, with the continuous development of the field of artificial intelligence, many researchers have begun to use deep learning methods to process EEG signals for selective attention decoding, which is called auditory attention detection. Auditory attention detection based on EEG signals can be divided into two categories: one is speaker recognition, which aims to identify the sound that the subject selectively pays attention to in a multi-sound source environment; the other is spatial attention detection, which aims to detect the spatial orientation of the subject's attention.

[0003] In the field of speaker recognition, the cross attention mechanism and spatial attention plus frequency band attention mechanism in neural networks were used to build classification models. The accuracy of the 2s decision window is about 90%-96%, and the accuracy of the 1s decision window is about 90%-94%. The above studies all use clean speech envelopes as model input, but this is unreasonable in practical applications because only mixed speech signals can be used in real scenarios.

[0004] In the field of related spatial attention detection, the convolutional neural network model was used to decode the spatial orientation of the subject's attention without using the speech envelope. The results showed that the accuracy of the 1s decision window was about 80.8%. Zhang et al. used a multi-task learning model to assist in direct auditory attention classification through the envelope reconstruction task. The results showed that the accuracy of the 2s decision window was about 82%. In these research methods, researchers tried to directly classify auditory attention without using the speech envelope. However, they did not take into account that EEG signals are complex signals with different information in different frequency bands and a large amount of temporal information, which cannot effectively improve the classification results. Secondly, the shallow convolutional neural network (CNN) they used has some limitations, which restricts the model from fully learning the effective information of EEG signals. Specifically, when processing EEG data, shallow CNN cannot deeply mine the complex features in the data due to its simple structure. In addition, the pooling layer may lose a lot of valuable information while reducing the feature dimension, which makes it difficult for the model to effectively represent the subtle differences between EEG signals that focus on different spatial orientations. These factors directly affect the overall performance of the model, especially in cross-subject experiments, because the EEG signal characteristics of different individuals may vary significantly, and shallow CNNs lack sufficient ability to adapt to such individual differences, resulting in decreased classification accuracy. Summary of the invention

[0005] The purpose of the present invention is to provide an EEG auditory attention classification method and system based on the time-frequency attention mechanism, so as to solve the problem that the original EEG signal has weak representation ability and cannot effectively improve the classification effect.

[0006] To achieve the purpose of the present invention, the technical solution provided by the present invention is specifically as follows:

[0007] First aspect

[0008] The present application provides an EEG auditory attention classification method based on a time-frequency attention mechanism, comprising the following steps:

[0009] Step S1: obtaining EEG signals;

[0010] Step S2: filtering the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information;

[0011] Step S3: re-referencing the multi-band EEG signal to eliminate interference information in the EEG signal;

[0012] Step S4: splicing the re-referenced multi-band EEG signals and performing window slicing operations so as to be input into the neural network;

[0013] Step S5: assigning different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to enhance the frequency band information that is useful for representing auditory attention;

[0014] Step S6: fusing the weighted multi-band EEG signals into a two-dimensional EEG signal, extracting the frequency domain features of the brain point signals, and outputting the EEG signal with the extracted frequency domain features;

[0015] Step S7: extracting time series features from the EEG signal whose frequency domain features have been extracted, and outputting the EEG signal whose time series features have been extracted;

[0016] Step S8: classifying the EEG signal from which the time series features have been extracted.

[0017] Second aspect

[0018] Corresponding to the above method, the present application provides an EEG auditory attention classification system based on the time-frequency attention mechanism, including the following modules: a signal acquisition module, a filtering module and a re-reference module, a splicing and slicing module, a frequency band attention module, a multi-band EEG fusion module, a time attention module, and a classification module;

[0019] The signal acquisition module is used to acquire EEG signals;

[0020] The filtering module is used to filter the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information;

[0021] The re-reference module is used to re-reference the multi-band EEG signal to eliminate interference information in the EEG signal;

[0022] The splicing and slicing module is used to splice the re-referenced multi-band EEG signals and perform window slicing operations so as to input them into the neural network;

[0023] The frequency band attention module is used to assign different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to strengthen the frequency band information that is beneficial to characterizing auditory attention;

[0024] The multi-band EEG fusion module is used to fuse the weighted multi-band EEG signals into a two-dimensional EEG signal, extract the frequency domain features of the brain point signal, and output the EEG signal with the extracted frequency domain features;

[0025] The temporal attention module is used to extract time series features from the EEG signal from which frequency domain features have been extracted, and output the EEG signal from which time series features have been extracted;

[0026] The classification module is used to classify the EEG signals from which the time series features have been extracted.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] Compared with simple feature selection, the frequency band attention module and the time attention module are introduced into the model, relying on the learning ability of the network to obtain the attention of different frequency bands and different times of the current sample. It does not require too much manual intervention and relies entirely on the generalization ability of the model. At the same time, the use of deep convolutional neural networks in the classification module can make the model have a stronger ability to capture auditory attention features, solving the problem that the existing technology is insufficient in extracting the time-frequency information of the original EEG signal, the original EEG information representation ability is not strong, and the auditory attention classification effect cannot be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a method flow chart provided by an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a system framework provided by an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of a frequency band attention module in an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of the temporal attention module in an embodiment of the present invention.

[0033] Figure 5 Schematic diagram of a classification module in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment provides an EEG auditory attention classification method based on a time-frequency attention mechanism, comprising the following steps:

[0037] Step S1: obtaining EEG signals;

[0038] Step S2: filtering the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information;

[0039] Step S3: re-referencing the multi-band EEG signal to eliminate interference information in the EEG signal;

[0040] Step S4: splicing the re-referenced multi-band EEG signals and performing window slicing operations so as to be input into the neural network;

[0041] Step S5: assigning different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to enhance the frequency band information that is useful for representing auditory attention;

[0042] Step S6: fusing the weighted multi-band EEG signals into a two-dimensional EEG signal, extracting the frequency domain features of the brain point signals, and outputting the EEG signal with the extracted frequency domain features;

[0043] Step S7: extracting time series features from the EEG signal whose frequency domain features have been extracted, and outputting the EEG signal whose time series features have been extracted;

[0044] Step S8: classifying the EEG signal from which the time series features have been extracted.

[0045] Preferably, in step S2, the EEG signal is filtered to extract a multi-band EEG signal for representing auditory attention information, specifically:

[0046] Use filters to extract five frequency bands of EEG signals: δ (1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (12-30 Hz), and γ (30-50 Hz).

[0047] Specifically, the filter adopts a Butterworth bandpass filter, and the transfer function of the Butterworth filter is shown in the following formula:

[0048]

[0049] Where n is the order of the filter, ω c is the cut-off frequency, ω p is the passband edge frequency;

[0050] According to the characteristics of the bandpass filter, different upper and lower cutoff frequencies need to be set when extracting different frequency bands, that is, only signals in specific frequency bands are allowed to pass.

[0051] Preferably, in step S3, the multi-band EEG signal is re-referenced, specifically:

[0052] performing average re-referencing on the multi-band EEG signals;

[0053] Assume that each frequency band of the multi-band EEG signal is X i (t), where i is the i-th electrode, t is the sampling point of the current electrode at time t, and the average re-referenced EEG signal is shown as follows:

[0054]

[0055] Where N is the number of electrodes, Y i(t) is the EEG signal after re-reference, X i (t) is the original EEG signal.

[0056] Preferably, the step S4 specifically includes:

[0057] The obtained five-band EEG signals are spliced ​​along the third dimension to obtain multi-band EEG data X∈R C×T×B , where C is the number of electrodes, T is the sampling point, and B is the number of frequency bands;

[0058] Then, the spliced ​​EEG data is segmented into segments of a set length. The time length of 1 second can be selected for segmentation, specifically:

[0059] A sliding window of 1 s is set to perform non-overlapping segmentation on the EEG data. The resulting sample format is: number of channels × sampling points × number of frequency bands, where the number of channels is 64, the number of sampling points is sampling rate × time, here 128 sampling points, and the number of frequency bands is 5.

[0060] Preferably, the step S5 specifically includes:

[0061] For the segmented multi-band EEG samples F∈R C×T×B , convolution is performed along the channel dimension, and the 2D feature map obtained after convolution is shown as follows:

[0062] F s ∈R B×T

[0063] F s Represents the feature map after convolution, and its output dimension is B×T;

[0064] The obtained output is activated using the Elu function. The calculation formula of Elu is as follows:

[0065]

[0066] Among them, alpha is an optional parameter, usually taking the default value of 1.0;

[0067] The activated feature map is average pooled along the time dimension. The feature map obtained by average pooling is shown in the following formula:

[0068] F sp ∈R B×1

[0069] The pooled feature map is connected to two fully connected layers. The Tanh activation function is used to activate each fully connected layer to explore the relationship between different frequency bands, and finally the frequency band attention mask is obtained. The Tanh activation function formula is shown as follows:

[0070]

[0071] The attention mask obtained using full connection is shown as follows:

[0072] Mask∈R 1×B

[0073] Use the obtained band attention mask Mask∈R 1×B , assigning weights to different sub-bands of the EEG signal.

[0074] Preferably, the step S6 specifically includes:

[0075] The output feature map is convolved, and the feature map obtained after convolution is shown in the following formula:

[0076] F t ∈R B×T

[0077] The obtained feature map is then activated using the Relu function;

[0078] The Relu activation function formula is as follows:

[0079]

[0080] Finally, the feature map is averaged and pooled along the time dimension. The feature map obtained after average pooling is shown in the following formula:

[0081] F tp ∈R B×32 .

[0082] Preferably, in step S7, a self-attention mechanism is used to extract the temporal features in the EEG signal. Specifically, the self-attention calculation formula is as follows:

[0083]

[0084] Among them, Q, K, and V represent query, key, and value matrices, respectively, which are obtained by mapping EEG data three times. k Represents the dimension of the key.

[0085] Preferably, the step S8 specifically includes:

[0086] The extracted frequency domain features are combined with the time series features to enhance the auditory attention detection effect and complete the auditory attention classification task. Specifically include:

[0087] For the EEG samples F∈R with extracted temporal features C×T , input into the lightweight residual neural network for auditory attention feature extraction, and the obtained 2D feature map is shown in the following formula:

[0088] F s ∈R 512×1

[0089] F s Represents the feature map after convolution, and its output dimension is 512×1;

[0090] Then, a fully connected layer is used to prepare for classification. In a fully connected layer, each neuron in each layer is connected to all neurons in the previous layer, but there is no connection between neurons in the same layer. The function of the fully connected layer is to integrate the information of all feature maps obtained by the network before the output layer and output it as a feature vector.

[0091] Finally, the Sigmoid classifier is used, and the output of the connected dense neural layer is set to 2. The Sigmoid function is used to map the final feature vector to the range of 0-1, that is, the probability of each category classification output by the neuron. The model takes the result with the largest probability value output by the neuron as the final recognition result. The calculation of Sigmoid is shown in the following formula:

[0092]

[0093] Example 2

[0094] like Figure 2 As shown, this embodiment provides an EEG auditory attention classification system based on the time-frequency attention mechanism, including the following modules: a signal acquisition module, a filtering module and a re-reference module, a splicing and slicing module, a frequency band attention module, a multi-band EEG fusion module, a time attention module, and a classification module;

[0095] The signal acquisition module is used to acquire EEG signals;

[0096] The filtering module is used to filter the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information;

[0097] The re-reference module is used to re-reference the multi-band EEG signal to eliminate interference information in the EEG signal;

[0098] The splicing and slicing module is used to splice the re-referenced multi-band EEG signals and perform window slicing operations so as to input them into the neural network;

[0099] The frequency band attention module is used to assign different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to strengthen the frequency band information that is beneficial to characterizing auditory attention;

[0100] The multi-band EEG fusion module is used to fuse the weighted multi-band EEG signals into a two-dimensional EEG signal, extract the frequency domain features of the brain point signal, and output the EEG signal with the extracted frequency domain features;

[0101] The temporal attention module is used to extract time series features from the EEG signal from which frequency domain features have been extracted, and output the EEG signal from which time series features have been extracted;

[0102] The classification module is used to classify the EEG signals from which the time series features have been extracted.

[0103] It should be noted that the overall framework of this embodiment is divided into two parts.

[0104] The first part is the EEG signal preprocessing part. The preprocessing part performs multi-band filtering, re-reference, multi-band EEG splicing, and window slicing on the EEG signal in sequence through the signal acquisition module, filtering module, re-reference module, and splicing and slicing module.

[0105] The second part is the EEG auditory attention classification network based on the time-frequency attention mechanism. This part uses the frequency band attention module to assign different weights to EEGs of different frequency bands to strengthen the weight of useful information; uses the multi-band EEG fusion module to extract the frequency domain features of EEG signals; uses the time attention module to obtain complex timing features; uses the classification module, first inputs it into a lightweight residual neural network for auditory attention feature extraction, and then inputs it into a Sigmoid classifier for auditory attention detection.

[0106] Preferably, the filtering module is used to filter the EEG signal to extract a multi-band EEG signal for characterizing auditory attention information, specifically: using a filter to extract the five frequency bands of the EEG signal: δ (1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (12-30 Hz), and γ (30-50 Hz).

[0107] Among them, a Butterworth bandpass filter is used, and the transfer function of the Butterworth filter is shown as follows:

[0108]

[0109] Where n is the order of the filter, ω c is the cut-off frequency, ω p is the passband edge frequency.

[0110] According to the characteristics of the bandpass filter, different upper and lower cutoff frequencies need to be set when extracting different frequency bands, that is, only signals in specific frequency bands are allowed to pass.

[0111] Preferably, the re-reference module is used to perform average re-reference on the multi-band EEG signals to eliminate interference information in the EEG signals.

[0112] Specifically, let each frequency band of the multi-band EEG signal be X i (t), where i is the i-th electrode and t is the sampling point of the current electrode at time t. The average re-referenced EEG signal is shown as follows:

[0113]

[0114] Where N is the number of electrodes, Y i (t) is the EEG signal after re-reference, X i (t) is the original EEG signal.

[0115] Preferably, the slicing splicing module is specifically used for:

[0116] The obtained five-band EEG signals are spliced ​​along the third dimension to obtain multi-band EEG data X∈R C×T×B , where C is the number of electrodes, T is the sampling points, and B is the number of frequency bands.

[0117] Then the EEG data is divided into segments of a set length. Here, a time length of 1 second is selected for illustration, specifically:

[0118] A sliding window of 1 s is set to perform non-overlapping segmentation on the EEG data. The resulting sample format is: number of channels × sampling points × number of frequency bands, where the number of channels is 64, the number of sampling points is sampling rate × time, here 128 sampling points, and the number of frequency bands is 5.

[0119] The EEG auditory attention classification network based on time-frequency attention mechanism consists of four modules, namely frequency band attention module, multi-band EEG fusion module, temporal attention module and classification module.

[0120] The band attention module contains 3D convolution, mean pooling, and two fully connected layers. The specific parameters of the band attention module are shown in Table 1.

[0121] Table 1 Specific parameters of the frequency band attention module

[0122]

[0123] The multi-band EEG fusion module consists of a convolution layer and a pooling layer. The specific parameters of the multi-band EEG fusion module are shown in Table 2.

[0124] Table 2 Specific parameters of multi-band EEG fusion module

[0125]

[0126] The classification module consists of multiple convolutional layers, pooling layers and residual connections. The specific parameters of the classification module are shown in Table 3.

[0127] Table 3 Specific parameters of auditory attention feature extraction module

[0128]

[0129]

[0130] The frequency band attention module is used to assign different weights to different frequency bands in the multi-band EEG signal, thereby enhancing the frequency band information that is useful for characterizing the auditory angle attention. Specifically,

[0131] The frequency band attention module can make up for the deficiency of convolutional neural network in extracting frequency domain features of EEG signals and improve the suppression of irrelevant frequency domain information. Figure 3 As shown in the figure, the principle of the frequency band attention mechanism is to generate a frequency band attention mask by analyzing the internal relationship of the feature map obtained by CNN. The frequency band attention focuses on "where" the effective information on the feature map is. The frequency band attention module enables the model to obtain more representative sub-bands from the EEG signal for the classification of auditory attention.

[0132] In the frequency band attention module, the samples are sequentially passed through a 3D convolution layer, an Elu activation function, a mean pooling layer, and two fully connected layers.

[0133] For the segmented multi-band EEG samples F∈R C×T×B , convolution is performed along the channel dimension, and the 2D feature map obtained after convolution is shown as follows:

[0134] F s ∈R B×T

[0135] F s Represents the feature map after convolution, and its output dimension is B×T;

[0136] The obtained output is activated using the Elu function. The calculation formula of Elu is as follows:

[0137]

[0138] Among them, alpha is an optional parameter and usually takes the default value of 1.0.

[0139] The activated feature map is average pooled along the time dimension. The feature map obtained by average pooling is shown in the following formula:

[0140] F sp ∈R B×1

[0141] The pooled feature map is connected to two fully connected layers. Each fully connected layer is activated using the Tanh activation function to explore the relationship between different frequency bands and finally obtain the frequency band attention mask. The Tanh activation function formula is shown as follows:

[0142]

[0143] The attention mask obtained using full connection is shown as follows:

[0144] Mask∈R 1×B

[0145] Use the obtained band attention mask Mask∈R 1×B , assigning weights to different sub-bands of the EEG signal.

[0146] The multi-band EEG fusion module is used to fuse the multi-band EEG into a two-dimensional EEG signal and extract the frequency domain features of the brain point signal, specifically including:

[0147] After the frequency band attention module, each sub-band with dynamically assigned weights is obtained, which will help this module to better extract the frequency domain features in the EEG signal and better complete the EEG auditory attention classification task.

[0148] In the multi-band EEG fusion module, the samples are sequentially passed through a 2D convolution layer, a Relu activation function, and a mean pooling layer.

[0149] The feature map output by the frequency band attention module is convolved, and the feature map obtained after convolution is shown in the following formula:

[0150] F t ∈R B×T

[0151] The obtained feature map is then activated using the Relu function.

[0152] The Relu activation function formula is as follows:

[0153]

[0154] Finally, the feature map is averaged and pooled along the time dimension. The feature map obtained after average pooling is shown in the following formula:

[0155] F tp ∈R B×32

[0156] The temporal attention module is used to extract the temporal features of EEG signals, including:

[0157] Psychoacoustic research provides sufficient evidence that human attention itself is a dynamic and time-based activity, and the auditory system is sensitive to time. Therefore, a temporal attention module is introduced in this scheme to effectively extract the complex timing features in EEG signals and focus on the auditory information expressed by the EEG signals at specific time nodes.

[0158] The EEG signal extracted by frequency domain features is further extracted with time series features in the time dimension. In this scheme, the self-attention mechanism is used to extract the time series features of the EEG signal. The implementation method is as follows: Figure 4 shown.

[0159] The self-attention mechanism mainly uses scaled dot product attention. Its input part consists of three matrices: Q (Queries), K (Key), and V (Value). The output is the weighted sum of the similarity and value of Q and K.

[0160] The self-attention calculation formula is as follows:

[0161]

[0162] Among them, Q, K, and V represent query, key, and value matrices, respectively, which are obtained by mapping EEG data three times. k Represents the dimension of the key.

[0163] The classification module is used to combine the extracted frequency domain features with the time series features to enhance the auditory attention detection effect and complete the auditory attention classification task. Specifically, it includes:

[0164] The lightweight residual network model provided in this embodiment is used to extract the features of EEG signals to realize auditory attention detection. Figure 5 As shown in the figure. This model is based on the ResNet architecture, but is simplified by using only two layers of residual connections, which reduces the complexity of the model and the number of parameters while retaining the advantages of ResNet in solving the gradient vanishing and gradient exploding problems. In this simplified structure, when the input is x, the learned feature is H(x), and we hope that the network will learn the residual F(x), whose formula is F(x) = H(x)-x, so that the actual output of the network is F(x)+x. Even if the residual is 0, the network still performs the identity mapping and does not affect the performance, but through non-zero residuals, the stacked layers can further learn new features based on the input features, improving the accuracy and efficiency of the model.

[0165] Then, a fully connected layer is used to prepare for classification. In a fully connected layer, each neuron in each layer is connected to all neurons in the previous layer, but there is no connection between neurons in the same layer. The function of the fully connected layer is to integrate the information of all feature maps obtained by the network before the output layer and output it as a feature vector.

[0166] Finally, the Sigmoid classifier is used, and the output of the connected dense neural layer is set to 2. The Sigmoid function is used to map the final feature vector to the range of 0-1, that is, the probability of each category classification output by the neuron. The model takes the result with the largest probability value output by the neuron as the final recognition result. The calculation of Sigmoid is shown in the following formula:

[0167]

[0168] It should be noted that the EEG auditory attention classification method based on the time-frequency attention mechanism of this application uses the EEG signals collected by the BioSemi ActiveTwo device to perform the auditory attention classification task in a laboratory scenario.

[0169] Here is an introduction to this dataset: The dataset used contains EEG data of 16 normal hearing subjects who were asked to selectively focus on one of two speakers speaking simultaneously. The speech stimuli consisted of four Dutch stories told by three male Flemish speakers. The stimuli were presented binaurally (one speaker in each ear) or filtered by the head-related transfer function (HRTF) to simulate speech coming from 90° to the left and 90° to the right. Throughout the experiment, the order of condition presentation was randomized for different subjects, and the number of experiments with left and right ear focus was equal. 64-channel EEG signals were recorded using the BioSemi ActiveTwo device with a sampling rate of 8192Hz. A total of 8×6 minutes of EEG data were collected for each subject, for a total of 12.8 hours of EEG data for the 16 subjects.

[0170] The neural network model needs to be trained with samples in advance before it can be used for auditory attention classification.

[0171] Specifically, the training process includes:

[0172] Within-subject experiment: The EEG data of each subject was divided using a four-fold cross-validation method. Specifically, each subject's 8 experiments (4 focusing on the left side and 4 focusing on the right side) were divided into 4 groups, each group containing one experiment focusing on the left side and one experiment focusing on the right side. One of the groups was selected as the test set, and the remaining 3 groups were used as training sets. The data of the 16 subjects were divided according to this method. During training, the training data of all subjects were mixed together, and during testing, the test data of all subjects were mixed together for testing.

[0173] Cross-subject experiment: In the cross-subject experiment, the 15-fold cross-validation method is used to divide the data. Each time, one of the 16 subjects is selected as the test set, and the data of the remaining 15 subjects are used as the training set. This cycle is repeated to ensure that the data of each subject is used as the test set for verification. This experiment is used to test the generalization ability of the model between different subjects.

[0174] The training process uses the stochastic gradient descent (SGD) optimizer for optimization. The model training is divided into 200 epochs. In the first 25 epochs, the learning rate is set to 0.01, and the learning rate is reduced to 0.001 in the subsequent 175 epochs. At the same time, batch_size is set to 256 to ensure that each training batch contains enough data samples for effective gradient updates.

[0175] In order to verify the effectiveness of the model, the method proposed in this application is compared with different models, including convolutional neural network (CNN) and STA-NET. The comparison results of different models are shown in Tables 4 and 5.

[0176] Table 4 shows the comparison results of different models within the subjects.

[0177]

[0178] Table 5 shows the comparison results of different models across subjects.

[0179]

[0180] It can be seen from Tables 4 and 5 that, compared with the CNN and STA-NET methods, the method proposed in this application has the highest accuracy in different time windows in both within-subject and across-subject experiments.

[0181] The optional implementation modes of the embodiments of the present invention are described in detail above in conjunction with the accompanying drawings. However, the embodiments of the present invention are not limited to the specific details in the above implementation modes. Within the technical concept of the embodiments of the present invention, various simple modifications can be made to the technical scheme of the embodiments of the present invention, and these simple modifications all belong to the protection scope of the embodiments of the present invention.

Claims

1. A method for classifying EEG auditory attention based on time-frequency attention mechanism, characterized in that: The following steps are involved: Step S1: obtaining EEG signals; Step S2: filtering the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information; Step S3: re-referencing the multi-band EEG signal to eliminate interference information in the EEG signal; Step S4: splicing the re-referenced multi-band EEG signals and performing window slicing operations so as to be input into the neural network; Step S5: assigning different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to enhance the frequency band information that is useful for representing auditory attention; Step S6: fusing the weighted multi-band EEG signals into a two-dimensional EEG signal, extracting the frequency domain features of the brain point signals, and outputting the EEG signal with the extracted frequency domain features; Step S7: extracting time series features from the EEG signal whose frequency domain features have been extracted, and outputting the EEG signal whose time series features have been extracted; Step S8: classifying the EEG signal from which the time series features have been extracted.

2. According to the method of claim 1, the method is characterized in that: In step S2, the EEG signal is filtered to extract a multi-band EEG signal for representing auditory attention information, specifically: Use filters to extract five frequency bands of EEG signals: δ (1-4 Hz), θ (4-8 Hz), α (8-12 Hz), β (12-30 Hz), and γ (30-50 Hz).

3. According to claim 2, the EEG auditory attention classification method based on the time-frequency attention mechanism is characterized in that: The filter adopts a Butterworth bandpass filter, and the transfer function of the Butterworth filter is shown in the following formula: Where n is the order of the filter, ω c is the cut-off frequency, ω p is the passband edge frequency; According to the characteristics of the bandpass filter, different upper and lower cutoff frequencies need to be set when extracting different frequency bands, that is, only signals in specific frequency bands are allowed to pass.

4. According to claim 3, the EEG auditory attention classification method based on the time-frequency attention mechanism is characterized in that: In step S3, the multi-band EEG signal is re-referenced, specifically: performing average re-referencing on the multi-band EEG signals; Assume that each frequency band of the multi-band EEG signal is X i (t), where i is the i-th electrode, t is the sampling point of the current electrode at time t, and the average re-referenced EEG signal is shown as follows: Where N is the number of electrodes, Y i (t) is the EEG signal after re-reference, X i (t) is the original EEG signal.

5. According to claim 4, the EEG auditory attention classification method based on the time-frequency attention mechanism is characterized in that: The step S4 specifically includes: The obtained five-band EEG signals are spliced ​​along the third dimension to obtain multi-band EEG data X∈R C ×T×B , where C is the number of electrodes, T is the sampling point, and B is the number of frequency bands; Then, the spliced ​​EEG data is segmented into segments of a set length, and a time length of 1 second is selected for segmentation, specifically: A sliding window of 1 s is set to perform non-overlapping segmentation on the EEG data. The resulting sample format is: number of channels × sampling points × number of frequency bands, where the number of channels is 64, the number of sampling points is sampling rate × time, here 128 sampling points, and the number of frequency bands is 5.

6. The method for classifying EEG auditory attention based on time-frequency attention mechanism according to claim 5, characterized in that: The step S5 specifically includes: For the segmented multi-band EEG samples F∈R C×T×B , convolution is performed along the channel dimension, and the 2D feature map obtained after convolution is shown as follows: F s ∈R B×T F s Represents the feature map after convolution, and its output dimension is B×T; The obtained output is activated using the Elu function. The calculation formula of Elu is as follows: Among them, alpha is an optional parameter, usually taking the default value of 1.0; The activated feature map is average pooled along the time dimension. The feature map obtained by average pooling is shown in the following formula: F sp ∈R B×1 The pooled feature map is connected to two fully connected layers. The Tanh activation function is used to activate each fully connected layer to explore the relationship between different frequency bands, and finally the frequency band attention mask is obtained. The Tanh activation function formula is shown as follows: The attention mask obtained using full connection is shown as follows: Mask∈R 1×B Use the obtained band attention mask Mask∈R 1×B , assigning weights to different sub-bands of the EEG signal.

7. The method for classifying EEG auditory attention based on time-frequency attention mechanism according to claim 6, characterized in that: The step S6 specifically includes: The output feature map is convolved, and the feature map obtained after convolution is shown in the following formula: F t ∈R B×T The obtained feature map is then activated using the Relu function; The Relu activation function formula is as follows: Finally, the feature map is averaged and pooled along the time dimension. The feature map obtained after average pooling is shown in the following formula: F tp ∈R B×32 。 8. The method for classifying EEG auditory attention based on time-frequency attention mechanism according to claim 7, characterized in that: In step S7, a self-attention mechanism is used to extract temporal features in the EEG signal.

9. The method for classifying EEG auditory attention based on time-frequency attention mechanism according to claim 8, characterized in that: The self-attention calculation formula is as follows: Among them, Q, K, and V represent query, key, and value matrices, respectively, which are obtained by mapping EEG data three times. k Represents the dimension of the key.

10. An EEG auditory attention classification system based on time-frequency attention mechanism, characterized in that: It includes the following modules: signal acquisition module, filtering module, re-reference module, splicing and slicing module, frequency band attention module, multi-frequency band EEG fusion module, time attention module, and classification module; The signal acquisition module is used to acquire EEG signals; The filtering module is used to filter the acquired EEG signal to extract a multi-band EEG signal for representing auditory attention information; The re-reference module is used to re-reference the multi-band EEG signal to eliminate interference information in the EEG signal; The splicing and slicing module is used to splice the re-referenced multi-band EEG signals and perform window slicing operations so as to input them into the neural network; The frequency band attention module is used to assign different weights to different frequency bands in the multi-band EEG signal after the windowing and slicing operation, so as to strengthen the frequency band information that is beneficial to characterizing auditory attention; The multi-band EEG fusion module is used to fuse the weighted multi-band EEG signals into a two-dimensional EEG signal, extract the frequency domain features of the brain point signal, and output the EEG signal with the extracted frequency domain features; The temporal attention module is used to extract time series features from the EEG signal from which frequency domain features have been extracted, and output the EEG signal from which time series features have been extracted; The classification module is used to classify the EEG signals from which the time series features have been extracted.

Images