Auditory Attention Decoding Method and System Based on Multi-Scale Frequency-Spatial Attention Network

Abstract

The present invention discloses an auditory attention decoding method and system based on a multi-scale spatio-temporal attention network. The method includes the following steps: acquiring electroencephalogram (EEG) data when attentively listening to voices in different directions, and dividing the EEG data using a sliding window to generate a series of decision windows, where each decision window contains a segment of EEG signal; inputting the EEG signal of the decision window into the multi-scale spatio-temporal attention network: the multi-scale time-domain convolution module processes the input EEG signal to extract time-domain features in different frequency ranges, and the spatio-temporal attention module converts the time-domain features into a spatio-temporal feature map to further capture the global dependencies between different brain regions and extract the spectral-spatial information related to auditory attention; the fully connected layer classification module outputs the probabilities of predicting that the speech direction is the left or right direction based on the spectral-spatial information. The present invention significantly improves the accuracy of auditory attention decoding.

Description

Technical Field

[0001] The present invention belongs to the field of auditory brain-computer interfaces, and particularly relates to an auditory attention decoding method and system based on a multi-scale spatio-temporal attention network, which can be applied to the development of neuro-driven auditory assistive devices (cochlear implants or hearing aids), etc. Background Art

[0002] In a complex acoustic environment, humans can exhibit the "cocktail party effect", that is, focus on the target speech of interest in an environment where multiple sounds are mixed, while ignoring the interference of other sounds. However, for people with hearing loss, the masking effect of background noise on the target speech is significantly enhanced, resulting in impaired auditory selective attention ability and difficulty in effectively focusing on the target speech. Neuroscience research shows that compared with non-target speech, the neural activity in the cerebral cortex shows a stronger similarity with the amplitude envelope of the target speech. Therefore, decoding the direction of auditory attention (i.e., auditory attention decoding) based on electroencephalogram (EEG) signals can provide key technical support for neuro-driven auditory assistive devices.

[0003] At present, existing research has confirmed the feasibility of decoding auditory attention from EEG, and EEG-based auditory attention decoding algorithms are mainly divided into two types: stimulus reconstruction and direct classification. The stimulus reconstruction algorithm faces huge challenges in practical applications because it needs to separate pure speech from mixed speech, which is extremely difficult in real-world scenarios. Although the direct classification method has more potential in practical applications, traditional linear decoders have obvious defects. Due to the non-linear characteristics of the brain's auditory system, traditional linear decoders are difficult to capture the non-linear mapping relationship in EEG signals, resulting in a longer decision time window, and the decoding accuracy will rapidly decline as the decoding window length shortens. In recent years, deep learning techniques have been widely used in auditory attention decoding research, but existing methods still have many problems. 1) Complex and suboptimal frequency-domain feature extraction: When extracting EEG frequency-domain features, existing methods usually need to pre-filter the EEG signal and then manually extract differential entropy features in fixed frequency bands. This operation not only increases the complexity of data preprocessing, but also due to the fixed frequency band division method, it is difficult to obtain the optimal frequency-domain decoding range that best matches auditory attention decoding, resulting in the inability to fully exploit the frequency-domain information in EEG signals. 2) Poor adaptability of convolutional kernels: Most methods based on convolutional neural networks use fixed-size convolutional kernels to learn local features. However, in actual situations, the optimal convolutional kernel size varies among different subjects and at different time points. The method with a fixed convolutional kernel size cannot adapt to this variation, limiting the effective extraction of features from different EEG data. 3) Limitations of 3D convolution: Some studies have attempted to map two-dimensional EEG into three-dimensional data and use 3D convolution to process spatio-temporal or frequency-spatial features to utilize the spatial distribution features of EEG signals. However, 3D convolution faces many difficulties during training, has a high computational complexity, and the model is difficult to optimize. At the same time, it is also difficult to capture the long-range dependence relationships and dynamic change information between different brain regions in EEG signals, and cannot comprehensively reflect the activity characteristics of the brain during the auditory attention process. 4) Lack of an effective attention mechanism: Currently, in the field of auditory attention decoding, there has been insufficient research on attention mechanisms based on spatial and frequency-domain features. Due to the lack of such a mechanism that can comprehensively integrate and analyze spatial and frequency-domain information in EEG data, existing methods cannot fully capture the key information related to auditory attention, resulting in limitations in the performance of the model.

[0004] In summary, existing auditory attention decoding technologies still have bottlenecks in feature extraction, model training, and information capture. There is an urgent need for an innovative method to optimize the decoding strategy, improve the decoding accuracy in a short time window, and enhance the real-time performance and adaptability of the system, so as to provide more accurate and efficient technical support for neuro-driven auditory assistive devices. Summary of the Invention

[0005] In view of the deficiencies of existing auditory attention decoding technologies, the present invention proposes an auditory attention decoding method and system based on a multi-scale spatio-temporal attention network, aiming to improve the decoding accuracy and computational efficiency, especially the decoding performance in complex acoustic environments and short decision windows.

[0006] The technical solution adopted to achieve the object of the present invention is as follows:

[0007] An auditory attention decoding method based on a multi-scale spatio-temporal attention network, comprising the following steps:

[0008] Step 1, obtain electroencephalogram (EEG) data when attentively listening to speech in different directions, and use a sliding window to divide the EEG data to generate a series of decision windows, each decision window containing a segment of EEG signal;

[0009] Step 2, input the EEG signal of the decision window into a multi-scale spatio-temporal attention network (MSSANet model), and the multi-scale spatio-temporal attention network includes a multi-scale time-domain convolution module, a spatio-temporal attention module, and a fully connected layer classification module:

[0010] Step 2.1, the multi-scale time-domain convolution module processes the input EEG signal to extract time-domain features in different frequency ranges: the multi-scale time-domain convolution module includes a multi-scale residual convolution unit and a time-domain logarithmic variance calculation unit. In the multi-scale residual convolution unit, use N convolution kernels of size 1×1 to perform a dimensionality increase operation on a single input sample R , and then divide the output Y after dimensionality increase into K groups according to the channel dimension. For each group of output Y b perform depth convolution using different convolution kernels to obtain , and splice the results after grouped depth convolution along the output channel dimension to obtain a multi-scale convolution output , and the multi-scale convolution output is obtained through time-domain logarithmic variance calculation to get ;

[0011] At the same time, for each group of output Y b obtain a residual convolution output res through fixed convolution operation Conv , and the residual convolution output is obtained through time-domain logarithmic variance calculation to get ;

[0012] and are added together to obtain the output of the multi-scale time-domain convolution module, that is, the time-domain features in different frequency ranges ;

[0013] Step 2.2, the spatio-temporal frequency attention module converts the temporal domain features into spatio-temporal frequency feature maps, and further captures the global dependencies between different brain regions through the self-attention mechanism and learnable position encoding, extracting the spectral-spatial information related to auditory attention;

[0014] Step 2.3, the fully connected layer classification module outputs the probabilities of predicting the speech direction as the left or right direction based on the spectral-spatial information.

[0015] In the above technical solution, in Step 1, before dividing the EEG data, a data preprocessing step is further included, and the data preprocessing step includes downsampling, filtering, artifact removal, and / or channel normalization.

[0016] If the EEG data comes from the KUL dataset, first downsample the EEG data to 128 Hz, then perform band-pass filtering with an 8th-order Butterworth filter in the range of 0.1 - 50 Hz, and finally perform channel normalization;

[0017] If the EEG data comes from the DTU dataset, first filter out 50 Hz line noise and artifacts, remove eye artifacts through joint decorrelation analysis, perform whole-brain average rereference, then downsample to 128 Hz and perform channel normalization.

[0018] In the above technical solution, , where is a convolution with a kernel of 1, R is the EEG signal within each decision window, W1 is the weight matrix of the 1×1 convolution kernel, b1 is the bias vector, Y ∈ R N×C×T is the output after dimensionality increase, N is the number of convolutions, C is the number of channels of the EEG signal, and T is the number of samples within each decision window.

[0019] In the above technical solution, Y = [Y1, Y2,..., Y K , where Y b ∈ R N / K×C×T , b = 1, 2,..., K, and the b-th group uses a convolution kernel of size (1, k b ) for depth convolution, , where is the depth convolution, k b are different convolution kernel sizes, W 2b is the weight matrix of the b -th group of convolution kernels, b 2b is the bias vector, Z b is the EEG data after depth convolution, Z b ∈ RN / K×C×T , .

[0020] In the above technical solution, , where is a residual convolution, W3 is the weight matrix of the residual convolution kernel, and b3 is the bias vector.

[0021] In the above technical solution, the calculation formula of the time-domain logarithmic variance is , where ∈ represents the i th sample point of the lead, is the stride, represents the variance of the sample points, and the sample points are or .

[0022] In the above technical solution, in step 2.2, the output obtained by the multi-scale time-domain convolution module is converted into a frequency-time spatial feature map F of size N×M ∈ R N×M , where M = C×D, D represents the number of divisions of the EEG time-domain length according to the T' step size. The frequency-time spatial feature map F is passed through a learnable position encoding to retain the spatial position information of the EEG signal and output the feature P , and a Transformer Encoder is used to perform cross-frequency domain processing on the feature P to obtain , , is the spectral spatial information related to auditory attention.

[0023] In the above technical solution, in step 2.3, the frequency-time spatial attention feature is first flattened, and then passed through two fully connected layers to predict the probability of the auditory attention decoding direction to obtain , , where is the predicted probability output by the model, W4 is the weight matrix of the transformation, and b4 is the bias vector.

[0024] In the above technical solution, in step 2, the cross-entropy loss function is used to evaluate the multi-scale frequency-time spatial attention network, , where represents the number of samples, is the number of classifications, is the true value, is the predicted value, i is the ia sample, c is the number of categories, corresponding to left or right.

[0025] On the other hand, the present invention further includes a system capable of implementing the above-mentioned auditory attention decoding method based on a multi-scale spatio-frequency attention network, including a data import module, a data preprocessing module, the above-mentioned multi-scale spatio-frequency attention network, a model training module, and a result visualization module;

[0026] The data import module is used to select different types of data sets and import data. The data preprocessing module preprocesses the imported data, and the preprocessing includes downsampling, high-pass filtering, low-pass filtering, and / or normalization;

[0027] The model training module is used to set the length of the time window and the data set division ratio to optimize the training effect of the multi-scale spatio-frequency attention network to meet the requirements of different auditory attention decoding tasks;

[0028] The result visualization module visually displays the accuracy of the model prediction results after the multi-scale spatio-frequency attention network is trained. When the probability of the predicted speech direction being high on the left side in step 2.3, the model prediction result is "left side", and when the probability of the predicted speech direction being high on the right side in step 2.3, the model prediction result is "right side". The accuracy rate is the percentage of the correctly predicted ones in all samples accounting for the total samples.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] (1) Efficient feature extraction and adaptive frequency domain information acquisition: The MSSANet model proposed by the present invention extracts EEG local features in different frequency domain ranges through multi-scale time domain convolution, can simulate the filtering process, avoids the complex preprocessing of manually extracting frequency domain features, and can adaptively obtain frequency domain information related to auditory attention decoding. When the time domain logarithmic variance calculation unit processes the EEG signal, on the one hand, it can efficiently extract time domain information, and on the other hand, it cleverly retains the spatial information in the signal. Through experiments, it is verified that compared with traditional convolution and pooling operations, the auditory attention decoding accuracy is significantly improved;

[0031] (2) Spatio-frequency attention module for enhancing model performance: This module effectively captures long-range dependencies and global dependencies between different brain regions with the help of self-attention mechanism and learnable position encoding, comprehensively obtains frequency domain and spatial information related to auditory attention, and improves model performance. The self-attention mechanism can suppress the interference of irrelevant frequency bands / brain regions by enhancing the interaction weights of brain region pairs related to auditory attention. This module realizes the explicit modeling of cross-band spatial dependence relationships in EEG signals, providing a more global perspective feature representation for auditory attention decoding;

[0032] (3)Excellent experimental performance and high practical value: In the experiments on the KUL and DTU public datasets, MSSANet demonstrated its superior strength. It achieved the highest classification accuracy under extremely short decision windows of 0.1 second, 0.5 second, and the conventional 1-second window. This excellent performance under short decoding time windows lays a solid foundation for real-time auditory attention decoding, greatly improving the response speed and accuracy. With this advantage, it has great practical value in practical application scenarios such as neurally-driven auditory assistive devices (cochlear implants or hearing aids), can effectively meet real-world needs, and injects strong impetus into the development of related fields;

[0033] (4)Constructing an efficient automated decoding system: The present invention develops an auditory attention decoding system based on the multi-scale spatio-temporal attention network framework, integrating modules such as data import, preprocessing, model training, and result visualization. It supports the import of multiple datasets and formats. After preprocessing such as downsampling, the model is optimized through training parameters such as adjustable time windows, and the results are presented visually in terms of accuracy, realizing full-process automated analysis. It has the ability to flexibly adjust parameters and efficiently process data, reducing human errors, and providing a powerful tool for research and application related to auditory attention. Description of the Drawings

[0034] Figure 1 It is a flowchart of an auditory attention decoding method based on a multi-scale spatio-temporal attention network.

[0035] Figure 2 It is a schematic diagram of a multi-scale spatio-temporal attention network.

[0036] Figure 3 It is the decoding effect of the multi-scale spatio-temporal attention network in two public datasets.

[0037] Figure 4 It is the decoding effect of the ablation experiment of the multi-scale spatio-temporal attention network.

[0038] Figure 5 It is the visualization result of the ablation experiment features.

[0039] Figure 6 It is an auditory attention decoding system based on a multi-scale spatio-temporal attention network. Detailed Embodiments

[0040] The following further elaborates on the present invention in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] Embodiment 1

[0042] An auditory attention decoding method based on a multi-scale spatio-temporal attention network includes the following steps:

[0043] Step 1: Obtain EEG data, preprocess the EEG data, and then divide it to generate a series of decision windows:

[0044] Step 1.1: EEG data preprocessing step. The data preprocessing step includes downsampling, filtering, artifact removal, and normalization. If the EEG data is from the KUL dataset, first downsample the EEG data to 128 Hz, then perform band-pass filtering from 0.1 - 50 Hz using an 8th-order Butterworth filter, and finally perform channel normalization. If the EEG data is from the DTU dataset, first filter to remove 50 Hz line noise and artifacts, remove eye artifacts through joint decorrelation analysis, perform whole-brain average re-reference, then downsample to 128 Hz and perform channel normalization.

[0045] Step 1.2: Use a sliding window to divide the preprocessed EEG data to generate a series of decision windows, where each decision window contains a segment of EEG signal (such as 0.1 second, 0.5 second, and 1 second).

[0046] Step 2: Input the EEG signal of the decision window into a multi-scale frequency-space attention network (MSSANet model). The multi-scale frequency-space attention network includes a multi-scale time-domain convolution module, a frequency-space attention module, and a fully-connected layer classification module.

[0047] Step 2.1: The multi-scale time-domain convolution module processes the input EEG signal to extract time-domain features in different frequency ranges.

[0048] In the multi-scale residual convolution unit, use N convolution kernels of size 1×1 for dimension elevation operation, specifically expressed as:

[0049] (1);

[0050] where, R is the EEG signal within each decision window, is the convolution with a kernel of 1, W1 is the weight matrix of the 1×1 convolution kernel, b1 is the bias vector, Y ∈ R N×C×T is the output after dimension elevation, N is the number of convolutions, C is the number of channels of the EEG signal, T is the number of samples within each decision window. After the above steps, the dimension of the input data R can be changed to N×C×T;

[0051] Then divide the output Y after dimension elevation into K groups along the channel dimension, that is, Y = [Y1, Y2,..., Y K , where Y b ∈ R N / K×C×T , b = 1, 2,..., K. The b-th group uses a size of (1, k b) ( k b Perform depth convolution on convolution kernels (for different convolution kernel sizes), specifically expressed as:

[0052] (2);

[0053] Among them, is depth convolution, W 2b is the b weight matrix of the convolution kernel in the 2b group, b is the bias vector, b ∈ R N / K×C×T is the feature after depth convolution, Z

[0054] (3);

[0055] is the multi-scale convolution output.

[0056] Since time-domain convolution has a filtering-like effect, the model can dynamically extract multi-scale features in EEG signals during operation. In addition, introduce a residual path with a fixed convolution kernel, and use the residual connection mechanism to enhance the feature extraction ability and improve the network's ability to capture long-term dependence relationships. Assume that through a fixed convolution operation Conv res process the input to obtain the residual convolution output :

[0057] (4);

[0058] Among them, is the residual convolution, W3 is the weight matrix of the residual convolution kernel, and b3 is the bias vector.

[0059] In the time-domain logarithmic variance calculation unit: To efficiently extract time-domain features and retain spatial information in EEG signals, calculate the variance of the time series of each lead of EEG using logarithmic variance and take the logarithm. The calculation formula of the logarithmic variance operation is as follows:

[0060] (5);

[0061] Among them, ∈ represents the i sample point of the lead, is the stride, represents the variance of the sample point, and the sample point is or , based on Equation (5), the time-domain logarithmic variance calculation unit is applied respectively after multi-scale convolution ( ) and residual convolution ( ) to perform the logarithmic variance operation with a stride of .

[0062] Thus, after the multi-scale convolution output ( ) passes through the logarithmic variance operation, we get ∈ R N×C×D , and after the residual convolution output (O) passes through the logarithmic variance operation, we get ∈ R N×C×D , where , D represents the number of divisions of the EEG time-domain length according to the T' step size, and is the number of samples in each decision window. After passing through the logarithmic variance operation module, the results of the two paths are finally added element-wise:

[0063] (6);

[0064] is the output of the multi-scale time-domain convolution module;

[0065] Step 2.2, the frequency-space attention module processes the frequency-space feature map, captures the global dependencies between different brain regions, and extracts the spectral-spatial information related to auditory attention;

[0066] The processing process of the frequency-space attention module includes the following steps:

[0067] The output obtained after passing through the multi-size time-domain convolution module is converted into a frequency-space feature map F ∈ R N ×M of size N × M, where M = C × D. Further, the frequency-space feature map F is passed through a learnable position encoding to retain the spatial position information of the EEG signal and output the feature P , and the Transformer Encoder is used to perform cross-frequency domain processing on the feature P to enhance the model's ability to extract frequency-space distribution information. Finally, the output of the model can be obtained through the above steps:

[0068] (7);

[0069] is the spectral-spatial information related to auditory attention;

[0070] Step 2.3, the fully connected layer classification module outputs the probabilities of predicting the left or right direction based on the spectral-spatial information, and then defines the one with the higher probability as the result predicted by the model. Further, the percentage of the correctly predicted ones among all samples in the total samples is the final accuracy rate.

[0071] The fully connected layer classification module uses two fully connected layers as classifiers. First, the spectral-spatial information related to auditory attention in the frequency domain is flattened, and then, the probabilities of the decoding directions of auditory attention are predicted through two fully connected layers:

[0072] (8);

[0073] where, is the predicted probability output by the model, W4 is the transformed weight matrix, and b4 is the bias vector;

[0074] Further, the cross-entropy loss function is adopted to measure the difference between the prediction result and the real result of the MSSANet model, so as to guide the training of the MSSANet model:

[0075] (9);

[0076] where, represents the number of samples, is the number of classifications (in this invention, it is 2: left or right), is the real value, is the predicted value, i is for the i th sample, c is the number of categories, corresponding to left or right.

[0077] Embodiment 2

[0078] This embodiment verifies a method for decoding auditory attention based on a multi-scale frequency-spatial attention network in Embodiment 1.

[0079] This embodiment conducts experimental verification based on two publicly available datasets (KUL and DTU) widely used in the field of auditory attention research.

[0080] The KUL dataset contains EEG data of 16 normal-hearing subjects. The 64-channel EEG data is recorded using the BioSemi ActivateTwo System with a sampling rate of 8196 Hz. The experiment is carried out in a shielded room. The speech materials are read by 3 male native Dutch speakers, filtered at 4 kHz and set to an intensity of 60 dB, and then played through in-ear headphones. The experiment includes two auditory conditions. Each subject listens to 8 trials, and each trial lasts for 6 minutes.

[0081] The DTU dataset contains data from 18 normal-hearing subjects, which were collected at a sampling rate of 512 Hz using a 64-channel BioSemi ActiveTwo device. The subjects were required to listen to one of two simultaneously presented speakers, and the speaker's voice was played at an angle of 60° relative to the subject. The auditory material was played through ER-2 in-ear headphones at an intensity of 60 dB and included Danish audiobooks read by 3 male and 3 female readers. Each subject performed an experiment with 60 trials, and each trial lasted for 50 seconds.

[0082] Data preprocessing: To ensure the fairness and reliability of the experimental results, specific data preprocessing strategies were implemented for different datasets. For the KUL dataset, first, the EEG data was downsampled to 128 Hz to reduce the data dimension while retaining key information. Then, an 8th-order Butterworth filter was used for band-pass filtering in the range of 0.1 - 50 Hz to effectively remove noise while maximizing the retention of EEG information. Finally, channel normalization was performed to make the mean of each lead sample point 0 and the standard deviation 1, ensuring that the data of each lead was analyzed on the same scale. For the DTU dataset, first, the data was filtered to remove 50 Hz linear noise and other artifacts to ensure data quality. Then, eye artifacts were removed through joint decorrelation analysis to reduce the interference of eye movements on the EEG signal. Whole-brain average re-reference was performed to optimize the signal reference standard. After that, the data was downsampled to 128 Hz and channel normalization was completed to make the data meet the requirements of subsequent experiments.

[0083] Dataset division: To ensure the effectiveness and generalization ability of model training, the dataset was divided as follows. The first 90% of the data was intercepted from each trial as the training set for model training and parameter learning; the last 10% of the data was used as the test set to evaluate the generalization ability of the model on unknown data. Using the sliding window technique, samples were extracted from the training set and the test set with an overlap rate of 50% to increase the utilization rate and diversity of the data. The training set samples were further randomly divided into a training set (90% of the total) and a validation set (10% of the total). Taking the KUL dataset as an example, after the above operations, each subject could obtain 4658 training samples, 518 validation samples, and 568 test samples. The present invention strictly follows the dataset division principle to ensure data independence during the experimental process, avoid data leakage, and improve the reliability of the results.

[0084] To improve the training effect of the model, this embodiment adopts an optimization strategy. Specifically, the cross-entropy loss function is selected as an index to measure the difference between the model prediction result and the true label, guiding the training direction of the model. The batch size is set to 32 to balance memory usage and training efficiency; the epochs of the model are set to 100 to ensure that the model has enough training times to learn data features. An early stopping strategy is adopted. When the loss function value on the validation set does not decrease for 10 consecutive epochs, the training is stopped to prevent the model from overfitting. The Adam optimizer is used to train the model, and the learning rate is set to 0.0005 and the weight decay is set to 0.0003 to adjust the model parameters and make the model converge towards the optimal solution. The MSSANet model of the present invention is implemented in the Python 3.10 environment with the help of the PyTorch framework and accelerated by the GeForce RTX 3090 GPU to improve the training efficiency. In addition, the present invention uses the grid search method for hyperparameter tuning to ensure that the model reaches the best configuration. The parameter settings of the MSSANet model are shown in Table 1 in detail;

[0085] ;

[0086] Model effect comparison and analysis: The present invention conducts comprehensive experiments under decision windows of 0.1 s, 0.5 s, 1 s, etc. with important practical significance, and focuses on analyzing the decoding accuracy of the MSSANet model (presented in the form of mean ± standard deviation). The results are shown in Figure 3. The accuracies of the MSSANet model of the present invention on the KUL dataset are 92.4% ± 6.90 (0.1 s time window), 95.0% ± 5.09 (0.5 s time window), and 95.3% ± 4.94 (1 s time window) respectively; the accuracies on the DTU dataset are 78.5% ± 6.51 (0.1 s time window), 84.5% ± 5.75 (0.5 s time window), and 83.4% ± 6.80 (0.1 s time window) respectively. Further, to verify the superiority of the MSSANet model of the present invention, it is compared with a variety of advanced models, including STAnet, XAnet, MBSSFCC, BSAnet, DenseNet-3D, DBPNet, DARnet, etc. The results are shown in Table 2 in detail. The model of the present invention shows superiority under each decision window and significantly improves the decoding performance;

[0087] ;

[0088] Ablation experiment verification: To deeply explore the specific contributions of each module of the model to the overall performance, ablation experiments are carried out to comprehensively analyze the influence of different modules on the model performance. The specific experimental operations and analysis processes are as follows:

[0089] 1) Key module adjustment experiment: By replacing specific modules in the model or modifying their parameters, here we focus on adjusting the multi-scale time-domain convolution module and the size of the convolutional kernel. The results are as described in Table 3;

[0090] 2) Verification of the effectiveness of the frequency-space attention module: To prove the effectiveness of the frequency-space attention module in the model, we removed this module from the model while keeping other experimental settings the same as the original MSSANet model, and analyzed the changes in the auditory attention decoding performance of this module. The experiment is as Figure 4 shown. It can be seen from the figure that after removing the frequency-space attention module, the average accuracy rates in the decision windows of 0.1 s, 0.5 s, and 1 s decreased by 0.25%, 0.51%, and 0.74% respectively;

[0091] 3) Feature visualization analysis: To intuitively compare the capabilities of different modules in feature extraction and information representation, the t-SNE technique is used to visualize the features before the fully connected layer classification module (raw data representation refers to the data without being processed by the model; convolution: refers to using a one-dimensional convolutional kernel to replace the time-domain logarithmic variance calculation unit; average: refers to using the mean to replace the time-domain logarithmic variance calculation unit; kernel is 17: refers to using a one-dimensional convolution (kernel of size 1*17) to replace the multi-scale time-domain convolution module; without the frequency-space attention module means removing the frequency-space attention module from the model), as Figure 5 shown. It can be better understood the differences in feature representation of different modules, and the model of the present invention has stronger feature separability (the red and blue feature points are more separated). The results of this experiment show that the frequency-space attention module and the time-domain logarithmic variance calculation unit play a key role in improving the decoding performance, further verifying the effectiveness of the present invention;

[0092] ;

[0093] Example 3

[0094] A system for an auditory attention decoding method based on a multi-scale frequency-space attention network according to Example 1, as Figure 6 shown. This system can realize the full process operation from auditory attention-related data processing to model training and result presentation, effectively improving the auditory attention decoding efficiency. The specific component modules are as follows:

[0095] Data Import Module: Used to select different types of data sets and import data. It supports KUI data sets, DTU data sets, etc., and can also import newly collected data stored in formats such as.mff,.cnt, curry, etc. Through this module, users can select appropriate data set sources according to actual needs, providing a data basis for subsequent analysis. This system is applicable to the diverse needs of data sources for auditory attention decoding in different research or application scenarios, such as using specific publicly available data sets in scientific research experiments or importing newly collected data on-site in practical applications.

[0096] Data Preprocessing Module: Performs a series of preprocessing operations on the imported data. These include downsampling (which can be set to 128), high-pass filtering (with a parameter set to 48 Hz), low-pass filtering (with a parameter of 0.5 Hz), and normalization range setting (set to 0 - 1), etc. Through these preprocessing operations, the module removes interference factors, ensures the accuracy and consistency of the data, and improves the reliability of subsequent analysis, making it meet the requirements of subsequent model training.

[0097] Model Training Module: Users can select training parameters for model training. The time window (such as 0.1 seconds, 0.5 seconds, 1 second, etc.) and the data set division ratio (such as training set 0.9, training set 0.8, or other ratios) can be set. By adjusting these parameters, the training effect of the model can be optimized to meet the requirements of different auditory attention decoding tasks. This module can flexibly adjust training parameters according to specific tasks, improving the adaptability and accuracy of the model for different auditory attention decoding tasks.

[0098] Result Visualization Module: Visualizes the results after model training in the form of accuracy. For example, the accuracy corresponding to a 0.1 - second time window is 92% (green light), the accuracy corresponding to a 0.5 - second time window is 94% (green light), and the accuracy corresponding to a 1 - second time window is being calculated (red light), which facilitates users to intuitively understand the performance of the model under different parameter settings. Researchers or users can quickly evaluate the training effect of the model through this module, compare the accuracies of the model under different parameter settings such as different time windows, and thus select the optimal parameter configuration to guide subsequent auditory attention decoding applications.

[0099] The above - mentioned is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An auditory attention decoding method based on a multi-scale spatio-temporal attention network, characterized in that It includes the following steps: Step 1: Obtain the electroencephalogram (EEG) data when listening attentively to voices in different directions, and use a sliding window to divide the EEG data to generate a series of decision windows, where each decision window contains a segment of EEG signal; Step 2: Input the EEG signal of the decision window into a multi-scale spatio-temporal attention network, which includes a multi-scale time-domain convolution module, a spatio-temporal attention module, and a fully-connected layer classification module: Step 2.1, the multi-scale time-domain convolution module processes the input EEG signals to extract time-domain features in different frequency ranges: the multi-scale time-domain convolution module includes a multi-scale residual convolution unit and a time-domain logarithmic variance calculation unit. In the multi-scale residual convolution unit, N convolution kernels of size 1×1 are used to perform a dimensionality increase operation on a single input sample R , and then the output Y after dimensionality increase is divided into K groups according to the channel dimension. For each group of output Y b , depth convolution is performed using different convolution kernels to obtain . The results after grouped depth convolution are concatenated according to the output channel dimension to obtain a multi-scale convolution output . The multi-scale convolution output is obtained through time-domain logarithmic variance calculation to get ; Meanwhile, for each set of outputs Y b Through the fixed convolution operation Conv res The residual convolution output is obtained through processing , the residual convolution output Is obtained through time-domain logarithmic variance calculation ; and are added to obtain the output of the multi-scale time-domain convolution module, i.e., the time-domain features in different frequency ranges ; Step 2.2, the frequency-space attention module converts the time-domain features into a frequency-space feature map, and further captures the global dependencies between different brain regions through the self-attention mechanism and learnable position encoding, extracting the spectral-spatial information related to auditory attention; Step 2.3: The fully-connected layer classification module outputs the probability of predicting that the speech direction is the left or right direction based on the spectral-spatial information.

2. The auditory attention decoding method based on a multi-scale spatio-temporal attention network according to claim 1, wherein In Step 1, before dividing the EEG data, it further includes a data preprocessing step, and the data preprocessing step includes downsampling, filtering, artifact removal, and / or channel normalization.

3. The auditory attention decoding method based on a multi-scale spatio-temporal attention network according to claim 1, wherein , where is a convolution with a kernel of 1, R is the EEG signal within each decision window, W1 is the weight matrix of the 1×1 convolution kernel, and b1 is the bias vector. is the output after dimensionality increase, N is the number of convolutions, C is the number of channels of the EEG signal, and T is the number of samples within each decision window.

4. The auditory attention decoding method based on the multi-scale spatio-temporal attention network according to claim 3, wherein , where Y b ∈R N / K×C×T , b = 1, 2, …, K, the b-th group performs depth convolution using a convolution kernel of size (1, k b ). , where, is depth convolution, k b are different convolution kernel sizes, W 2b is the weight matrix of the b -th group of convolution kernels, b 2b is the bias vector, Z b is the EEG data after depth convolution, Z b ∈R N / K×C×T , .

5. The auditory attention decoding method based on a multi-scale spatio-temporal attention network according to claim 1, wherein , where is a residual convolution, W3 is the weight matrix of the residual convolution kernel, and b3 is the bias vector.

6. The auditory attention decoding method based on the multi-scale spatio-temporal attention network according to claim 1, wherein The calculation formula for the time-domain logarithmic variance is , where represents the i th sample point of the lead, is the stride, represents the variance of the sample points, and the sample points are or .

7. The auditory attention decoding method based on a multi-scale spatio-temporal attention network according to claim 1, characterized in that, In step 2.2, the output obtained from the multi-scale time-domain convolution module is converted into a spatio-frequency feature map F ∈ R of size N×M N×M , where M = C×D, D denotes the number of divisions of the EEG time-domain length according to T' the step size. The spatio-frequency feature map F retains the spatial position information of the EEG signal through learnable positional encoding to output a feature P , and the Transformer Encoder is used to perform cross-frequency domain processing on the feature P to obtain , , which is the spectral-spatial information related to auditory attention.

8. The auditory attention decoding method based on the multi-scale spatio-temporal attention network according to claim 1, characterized in that In step 2.3, first flatten the frequency-spatial attention feature and then pass it through two fully connected layers to predict the probability of the auditory attention decoding direction to obtain , , where is the predicted probability output by the model, W4 is the transformed weight matrix, and b4 is the bias vector.

9. The auditory attention decoding method based on a multi-scale spatio-temporal attention network according to claim 1, characterized in that In step 2, the cross-entropy loss function is used to evaluate the multi-scale spatio-temporal attention network, , where represents the number of samples, is the number of classifications, is the true value, is the predicted value, i is the i th sample, c is the number of classes, corresponding to left or right.

10. A system based on the auditory attention decoding method of the multi-scale spatio-temporal attention network as described in claim 1, characterized in that, It includes a data import module, a data preprocessing module, the multi-scale spatio-temporal attention network, a model training module, and a result visualization module; The data import module is used to select different types of data sets and import data, and the data preprocessing module preprocesses the imported data, and the preprocessing includes downsampling, high-pass filtering, low-pass filtering, and / or normalization; The model training module is used to set the length of the time window and the data set division ratio to optimize the training effect of the multi-scale spatio-temporal attention network to meet the requirements of different auditory attention decoding tasks; The result visualization module visually displays the accuracy rate of the model prediction result after the multi-scale spatio-temporal attention network is trained. When the probability of predicting the speech direction as the left is high in Step 2.3, the model prediction result is "left", and when the probability of predicting the speech direction as the right is high in Step 2.3, the model prediction result is "right". The accuracy rate is the percentage of the correctly predicted ones in all samples accounting for the total samples.

Images