A multi-task auditory attention detection method and device based on electroencephalogram signals

Abstract

The present invention proposes a multi-task auditory attention detection method and device based on electroencephalogram signals, comprising the following steps: collecting sound source position data of a user, and simultaneously collecting voice signals and electroencephalogram signals of the user, and preprocessing the voice signals and the electroencephalogram signals; establishing a multi-task decoding model, using the sound source position data of the user, the preprocessed voice signals and the electroencephalogram signals as training data of the multi-task decoding model, and training the multi-task decoding model with the training data; using the trained multi-task decoding model to identify the user's auditory attention object; compared with the prior art, the method solves the technical problem that the prior art cannot simultaneously process spatial positioning and voice envelope, shares resources and information between multiple tasks, and improves processing efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of physiological signal processing technology, and in particular to a multi-task auditory attention detection method and device based on electroencephalogram (EEG) signals. Background Technology

[0002] In multi-sound-source scenarios, humans can focus their auditory attention on a specific speaker, a phenomenon known as the "cocktail party effect." However, people with hearing impairments often struggle in such situations, finding it difficult to separate meaningful speech signals from background noise, which creates communication barriers in daily life.

[0003] Recent neuroscience research shows that auditory attention can be detected directly from neural activity; this technology is known as auditory attention detection. This technology has broad application prospects, particularly in the field of brain-computer interfaces, such as hearing aids and cochlear implants. By integrating advanced auditory attention detection algorithms into cognitively controlled hearing aids and rehabilitation devices, it is possible to help hearing-impaired individuals more accurately capture target sounds while suppressing background noise, thereby improving their auditory comprehension in multi-sound-source scenarios.

[0004] Existing auditory attention detection methods based on EEG signal spatial localization can achieve high decoding accuracy within a short decision window (1 second) in left-right binary classification scenarios. However, the performance of auditory attention detection based on sound source spatial localization significantly degrades when sound sources are close together or there are too many sound sources. Another approach is speaker identification, which requires reconstructing the speech envelope from EEG signals and performing correlation analysis between it and candidate envelopes. Although this method has poor decoding performance within 1 second, it addresses the lack of spatial information in auditory attention detection scenarios.

[0005] Currently, there is no effective solution to the limitations of existing auditory attention detection systems in multi-source environments in terms of decoding performance. Designing a high-performance auditory decoding and detection system that can adapt to diverse scenarios and provide better auditory assistance services for individuals with hearing impairments is an urgent challenge for researchers in the field of hearing.

[0006] Therefore, the relevant technologies for multi-task auditory attention detection systems based on electroencephalogram (EEG) signals still need further improvement. Summary of the Invention

[0007] In view of this, the present invention proposes a multi-task auditory attention detection method and device based on electroencephalogram (EEG) signals, which solves the technical problem that the prior art cannot simultaneously process spatial localization and speech envelope, and shares resources and information among multiple tasks, thereby improving processing efficiency and accuracy.

[0008] The technical solution of this invention is implemented as follows:

[0009] On the one hand, the present invention provides a multi-task auditory attention detection method based on electroencephalogram (EEG) signals, comprising the following steps:

[0010] Collect user's sound source location data, and simultaneously acquire user's voice signal and EEG signal, and preprocess the voice signal and EEG signal;

[0011] A multi-task decoding model is established, and the user's sound source location data, the preprocessed speech signal, and the EEG signal are used as training data for the multi-task decoding model. The multi-task decoding model is trained using the training data.

[0012] The trained multi-task decoding model is used to identify the user's auditory attention objects.

[0013] Based on this scheme, a further preferred embodiment is that the training data is divided into a training set and a validation set to evaluate the performance of the multi-task decoding model and avoid overfitting;

[0014] The multi-task decoding model is evaluated using the validation set, and the evaluation includes the loss function value, the accuracy of reconstructing the speech envelope, and the classification accuracy of spatial localization.

[0015] Based on the evaluation results, the parameters and architecture of the multi-task decoding model are adjusted, and the feature extraction and fusion processes are optimized to improve overall performance.

[0016] Based on this scheme, and further preferably, the preprocessing of the speech signal and the EEG signal, wherein the preprocessing of the speech signal includes bandpass filtering, downsampling, and normalization, specifically including the following steps:

[0017] The speech signal is bandpass filtered to extract the characteristic frequency bands;

[0018] Calculate the envelope of the speech signal after the bandpass filtering; the envelope of the speech signal is the speech envelope.

[0019] The calculated speech envelope is sampled at a rate consistent with that of the EEG signal to obtain the preprocessed speech signal.

[0020] Based on this scheme, and further preferably, the preprocessing of the EEG signal specifically includes the following steps:

[0021] The EEG signal is rereferenced and then bandpass filtered.

[0022] The EEG signal after bandpass filtering is sampled at the same sampling rate as the speech signal.

[0023] Finally, the sampled EEG signals were normalized.

[0024] Based on this scheme, and more preferably, the multi-task decoding model includes an input layer, a sharing layer, a spatial localization module, a feature fusion module, a reconstructed speech envelope module, and a computation module;

[0025] The input layer is used to input the electroencephalogram (EEG) signals.

[0026] The shared layer includes a convolutional neural network, which is used to extract common features of the EEG signals;

[0027] The spatial localization module includes a deep learning model, which is used to capture the spatial relationships of the EEG signals and extract spatial information features.

[0028] The feature fusion module is used to splice and fuse the general features extracted by the shared layer and the spatial information features extracted by the spatial positioning module;

[0029] The reconstructed speech envelope module includes a deep learning decoder. The reconstructed speech envelope module is used to reconstruct the speech envelope using the fused features to obtain reconstructed features.

[0030] The calculation module is used to calculate the correlation coefficient of the reconstructed features, the correlation coefficient including a combination of Pearson correlation coefficient and mean squared error.

[0031] Based on this scheme, and more preferably, the process of training the user's auditory attention object using the trained multi-task decoding model specifically includes the following:

[0032] The preprocessed EEG signal is input into the multi-task decoding model to extract general features and spatial information features. The general features and spatial information features are then spliced ​​and fused together, and the speech envelope is reconstructed based on the fused features.

[0033] For each candidate speech envelope, calculate its Pearson correlation coefficient with the reconstructed speech envelope, and take the candidate speech envelope with the largest Pearson correlation coefficient as the user's auditory attention object;

[0034] The positioning results from the spatial positioning module are used to identify and verify the user's auditory attention target;

[0035] Based on the calculated Pearson correlation coefficient and spatial localization results, the output is used for further processing or application to select the best matching candidate speech envelope or auditory attention direction as the user's auditory attention object.

[0036] Based on this scheme, and further preferably, the convolutional neural network is a one-dimensional convolutional neural network with a kernel size of 3, a stride of 1, and the number of kernels equal to the number of channels of the EEG signal.

[0037] Based on this scheme, and more preferably, the spatial positioning module further includes a global average pooling layer and a classifier;

[0038] The global average pooling layer is used to perform dimensionality reduction processing on the spatial information features.

[0039] The classifier includes two fully connected layers for processing and classifying the spatial information features.

[0040] Based on this scheme, and further preferably, the feature fusion module is used to splice and fuse the general features extracted by the shared layer and the spatial information features extracted by the spatial positioning module, specifically including the following:

[0041] The general features and the spatial information features are concatenated along the feature dimension to form a new feature matrix, thus obtaining the initial feature matrix dimension;

[0042] The concatenated features are transformed using the fully connected layer, and the features are then normalized using a multi-head attention mechanism. The feature fusion module outputs the normalized feature matrix.

[0043] Based on this scheme, and even more preferably, the correlation coefficient specifically includes a combination of Pearson correlation coefficient and mean squared error.

[0044] Secondly, the present invention also provides a multi-task auditory attention detection device based on electroencephalogram (EEG) signals, as described in one aspect, comprising:

[0045] The preprocessing module is used to collect the user's sound source location data, and simultaneously acquire the user's speech signal and EEG signal, and preprocess the speech signal and the EEG signal.

[0046] A multi-task decoding module is used to establish a multi-task decoding model, using the user's sound source location data, the preprocessed speech signal, and the EEG signal as training data for the multi-task decoding model, and training the multi-task decoding model with the training data.

[0047] The recognition module uses the trained multi-task decoding model to identify the user's auditory attention objects.

[0048] Thirdly, the present invention also provides a computer storage medium comprising multiple programs for implementing the multi-task auditory attention detection method based on electroencephalogram (EEG) signals described in one aspect.

[0049] The present invention provides a multi-task auditory attention detection method and device based on electroencephalogram (EEG) signals, which has the following advantages over the prior art:

[0050] This invention utilizes a deep learning-based multi-task learning model to simultaneously handle two tasks: spatial localization and speech envelope reconstruction. This differs from existing methods that typically handle these two tasks separately or focus on only one of them. This strategy enables the system to share resources and information across multiple tasks, improving processing efficiency and accuracy. Furthermore, by leveraging common features extracted through shared layers, this invention can utilize computational resources more efficiently and improve overall decoding performance and robustness by integrating information from different tasks.

[0051] Furthermore, this invention optimizes speech envelope reconstruction by extracting spatial information from electroencephalogram (EEG) signals. Existing technologies often overlook the potential of spatial information in speech envelope reconstruction, processing either the speech signal or spatial information separately. This design, by combining spatial information features with speech envelope reconstruction features, improves the accuracy and anti-interference capability of the system in auditory attention detection in complex environments. When users are in talkative or noisy environments, spatial information can help the system more accurately separate and identify target speech signals, enabling hearing-impaired individuals to understand speech information more clearly.

[0052] By designing a multi-task learning model, this method and device can be adjusted according to different application requirements, adapting to variations in the number and location of sound sources. For example, under different noise environments and speaker position changes, the system can dynamically adjust its decoding strategy through the feature fusion module, ensuring the efficiency and accuracy of auditory attention detection. This adaptability and flexibility make the system more versatile and able to meet a wider range of application needs. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1This is a flowchart of the multi-task auditory attention detection method based on electroencephalogram (EEG) signals described in this invention;

[0055] Figure 2 This is a flowchart of the preprocessing of EEG signals according to the present invention;

[0056] Figure 3 This is a flowchart of the preprocessing of the speech signal according to the present invention;

[0057] Figure 4 This is a framework diagram of the multi-task decoding model described in this invention;

[0058] Figure 5 This is a framework diagram of the spatial positioning module described in this invention;

[0059] Figure 6 This is a framework diagram of the multi-task auditory attention detection device based on electroencephalogram (EEG) signals described in this invention.

[0060] Figure 7 This is a structural diagram of the computer storage medium described in this invention. Detailed Implementation

[0061] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0062] It should be understood that the terminology used in this specification of embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of the invention. As used in this specification of embodiments of the invention and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0063] This invention provides a multi-task auditory attention detection method and apparatus based on electroencephalogram (EEG) signals. First, real-time acquired EEG and speech signals are preprocessed, and then input into a deep learning-based multi-task auditory attention decoding model. This model can efficiently extract key spatial information from the EEG signals and use this information to optimize the speech envelope reconstruction process. After reconstruction, the system calculates the correlation coefficient between the reconstructed speech envelope and multiple candidate speech envelopes to select the speech stream that best matches the subject's auditory attention, thus completing the auditory attention detection.

[0064] Example 1

[0065] A multi-task auditory attention detection method based on electroencephalogram (EEG) signals, the specific process of which is as follows: Figure 1 As shown, it includes the following steps:

[0066] S1. Collect the user's sound source location data, and simultaneously collect the user's voice signal and EEG signal, and preprocess the voice signal and the EEG signal;

[0067] S2. Establish a multi-task decoding model, using the user's sound source location data, the preprocessed speech signal, and the EEG signal as training data for the multi-task decoding model, and train the multi-task decoding model with the training data.

[0068] S3. Use the trained multi-task decoding model to identify the user's auditory attention target.

[0069] Specifically, the user's sound source location data refers to information related to the location where the user's voice is emitted, which helps identify the source of the sound. The sound source location data collected by the microphone is analyzed by deep learning combined with algorithms, and the sound source location data is used as training data for the multi-task decoding model.

[0070] Specifically, the training data is divided into a training set and a validation set to evaluate the performance of the multi-task decoding model and avoid overfitting.

[0071] The multi-task decoding model is evaluated using the validation set, and the evaluation includes the loss function value, the accuracy of reconstructing the speech envelope, and the classification accuracy of spatial localization.

[0072] Based on the evaluation results, the parameters and architecture of the multi-task decoding model are adjusted, and the feature extraction and fusion processes are optimized to improve overall performance.

[0073] Specifically, the accuracy of the reconstructed speech envelope is evaluated using the Pearson correlation coefficient.

[0074] In practical applications, the trained multi-task decoding model is used to identify the user's auditory attention objects, specifically including the following:

[0075] The preprocessed EEG signals acquired in real time are input into the model. Common features are extracted through the shared layer, spatial information features are extracted through the spatial localization module, and splicing and fusion are performed in the feature fusion module. Then, in the speech envelope reconstruction module, the speech envelope is reconstructed based on the fused features.

[0076] Next, the Pearson correlation coefficient between each candidate speech envelope and the reconstructed speech envelope is calculated. This coefficient reflects the linear correlation between the two signals; a larger value indicates greater similarity. The candidate speech envelope with the highest correlation coefficient is selected as the user's auditory attention target. Furthermore, the results from the spatial localization module can provide additional information for identifying and verifying the user's auditory attention target.

[0077] Finally, based on the calculated correlation coefficient and spatial localization results, the best-matching candidate speech envelope or auditory attention direction is selected as the user's auditory attention object, and this result is output for further processing or application. Through the above process, the multi-task decoding model can effectively identify the user's auditory attention object, ensuring accuracy and real-time performance in various application scenarios.

[0078] Specifically, the preprocessing of the EEG signals is illustrated in the flowchart below. Figure 2 As shown, the specific steps include the following:

[0079] The EEG signal is rereferenced and then bandpass filtered, optionally using a 1-50Hz bandpass filter.

[0080] The EEG signal after bandpass filtering is sampled at the same sampling rate as the speech signal, and optionally, downsampled to 128Hz.

[0081] Finally, the sampled EEG signals were normalized.

[0082] Specifically, the preprocessing of the speech signal is illustrated in the flowchart below. Figure 3 As shown, the specific steps include the following:

[0083] The speech signal is bandpass filtered to extract the characteristic frequency bands. Optionally, the Mel filter bank bandpass filtering can be used.

[0084] Calculate the envelope of the speech signal after the bandpass filtering, i.e., the speech envelope, and extract amplitude information;

[0085] The calculated speech envelope is sampled at a rate consistent with that of the EEG signal; optionally, it is downsampled to 128 Hz.

[0086] Specifically, the multi-task decoding model includes an input layer, a shared layer, a spatial localization module, a feature fusion module, a speech envelope reconstruction module, and a computation module; the framework diagram is as follows. Figure 4 As shown.

[0087] The input layer is used to input the electroencephalogram (EEG) signals.

[0088] The shared layer includes a convolutional neural network, which is used to receive the EEG signal from the input layer, extract the general features of the EEG signal, and output the general features to the feature fusion module.

[0089] The spatial positioning module includes a deep learning model. The spatial positioning module is used to receive the EEG signal from the shared layer, capture the spatial relationship of the EEG signal, extract spatial information features, and output the spatial information features to the feature fusion module.

[0090] The feature fusion module is used to receive, splice, and fuse the general features extracted by the shared layer and the spatial information features extracted by the spatial positioning module;

[0091] The reconstructed speech envelope module includes a deep learning decoder, which is used to reconstruct the speech envelope using the fused features to obtain reconstructed features.

[0092] The calculation module is used to receive the reconstructed features after reconstruction by the reconstructed speech envelope module, and to calculate the correlation coefficient of the reconstructed features.

[0093] Specifically, the convolutional neural network is a one-dimensional convolutional neural network with a kernel size of 3, a stride of 1, and the number of kernels equal to the number of channels of the EEG signal.

[0094] Specifically, the spatial localization module further includes a global average pooling layer and a classifier.

[0095] The global average pooling layer is used to perform dimensionality reduction processing on the spatial information features.

[0096] The classifier includes two fully connected layers for processing and classifying the spatial information features.

[0097] Specifically, the feature fusion module is used to splice and fuse the general features extracted by the shared layer and the spatial information features extracted by the spatial positioning module, specifically including the following:

[0098] The general features and the spatial information features are concatenated along the feature dimension to form a new feature matrix, thus obtaining the initial feature matrix dimension;

[0099] The concatenated features are transformed using the fully connected layer, and the features are then normalized using a multi-head attention mechanism. The feature fusion module outputs the normalized feature matrix.

[0100] Specifically, the correlation coefficient includes a combination of the Pearson correlation coefficient and the mean squared error.

[0101] Specifically, in the training data of the multi-task decoding model, the Adam optimizer is used for model training, the learning rate is set to 0.001, the total training time is 200 iterations (epochs), and an early stopping mechanism is introduced to prevent overfitting, wherein the patience parameter is set to 20 iterations (epochs).

[0102] In a preferred embodiment, specifically, the preprocessing of the speech signal, such as... Figure 3 As shown, it includes the following steps:

[0103] The Mel filter bank is used to perform bandpass filtering on the speech signal in order to extract the characteristic frequency bands in the speech signal;

[0104] The envelope of the filtered speech signal is calculated using the Hilbert-Huang transform to extract the amplitude information of the speech.

[0105] The calculated speech envelope was downsampled to 128Hz to reduce the amount of data and maintain a sampling rate consistent with the EEG signal.

[0106] In a preferred embodiment, specifically, the preprocessing of the EEG signals, such as... Figure 2 As shown, it includes the following steps:

[0107] By re-referencing to the average response of all electrodes, signal quality can be effectively improved.

[0108] Applying a 1-50 Hz bandpass filter to the EEG signal is sufficient to remove low-frequency drift and high-frequency noise;

[0109] The filtered EEG signal was downsampled to 128 Hz to reduce the amount of data and speed up subsequent processing.

[0110] The downsampled EEG signals were normalized to ensure data consistency and comparability.

[0111] In a preferred embodiment, specifically, the input layer is used to input EEG signals, transforming their dimension to E∈R. T×C T represents the number of sampling points within a decision window, and C represents the number of channels in the EEG signal.

[0112] In a preferred embodiment, the shared layer specifically includes a one-dimensional convolutional neural network (CNN) with a kernel size of 3, a stride of 1, and the number of kernels equal to the number of channels C. During the convolution process, the input signal is padded to ensure that the length of the output is the same as that of the input.

[0113] The main purpose of the shared layer is to extract common features of EEG signals. F shared ∈R T×CWhere T is the time step and C is the dimension of the general features, which will be shared in subsequent processing. The shared layer finally uses the ReLU activation function to enhance the non-linear expressive power of the model, resulting in richer extracted features.

[0114] In a preferred embodiment, specifically, such as Figure 5 As shown, the spatial localization module includes a Transformer model based on a self-attention mechanism. This Transformer model uses self-attention to capture spatial relationships in EEG signals and extract spatial information features. F spatial ∈R T×P , where T is the time step and P is the dimension of the spatial information feature.

[0115] In a preferred embodiment, to further process and classify these spatial features, the spatial localization module further includes a classifier consisting of a global average pooling layer and two fully connected layers. The global average pooling layer performs dimensionality reduction on the features, while the two fully connected layers use the sigmoid activation function to process and classify the spatial features, thus completing the spatial localization task.

[0116] In a preferred embodiment, the feature fusion module is used to splice and fuse the general features extracted from the shared layer. F shared and the spatial information features extracted by the spatial positioning module F spatial The general features and spatial information features are concatenated along the feature dimension to form a new feature matrix. F concat The general features F shared The dimension is R T×C The spatial information features F spatial The dimension is R T×P Therefore, the concatenated feature matrix F concat The dimension is R T ×(C+P) ,in,

[0117] ;

[0118] Then, the concatenated feature matrix F concat The input is fed into a fully connected layer for feature transformation to improve the representational power of the features, thereby obtaining... F transformed The operations of a fully connected network include linear transformations and activation function processing, as detailed in the following formulas:

[0119] ;

[0120] in, W It is a weight matrix. b This is the bias vector, and ReLU is the activation function. The transformed feature matrix... F transformed The dimension is R T×F , where F is the dimension of the output feature;

[0121] Furthermore, in order to stabilize the training process and improve the convergence speed of the model, the transformed features... F transformed Perform batch normalization. Batch normalization can make the mean of the features 0 and the variance 1, thereby reducing the internal covariate bias.

[0122] ;

[0123] Meanwhile, in order to capture the correlation between different features, a multi-head attention mechanism is used to process the normalized features. F normalized Processing:

[0124] ;

[0125] The advantage of this is that the multi-head attention mechanism can better capture the complex relationships between features, improving the model's expressive power. The final output of the feature fusion module is the processed feature matrix. F fused ,in:

[0126] ,

[0127] F fused This will serve as the input for the subsequent speech envelope reconstruction module. By fusing general and spatial information features, the model's ability to reconstruct the speech envelope can be improved.

[0128] In a preferred embodiment, the reconstructed speech envelope module includes a Transformer decoder composed of multi-head attention and masked multi-head attention mechanisms. The main function of this module is to reconstruct the speech envelope using the fused features. The multi-head attention mechanism can better capture the correlation between different features, enabling the model to reconstruct the speech envelope more accurately. The masked multi-head attention mechanism helps to effectively process sequence data, ensuring the causal consistency of information processing, thereby further improving the accuracy of speech envelope reconstruction.

[0129] For spatial localization tasks, binary cross-entropy is chosen as the loss function because the task is essentially a classification problem, requiring the model to distinguish and predict the location of the sound source from the EEG signal.

[0130] For the task of reconstructing the speech envelope, a combination of Pearson correlation coefficient and mean squared error was used as the loss function. This combination considers both the correlation between predicted and actual values ​​and the absolute error between them, thus comprehensively evaluating the model's performance.

[0131] In a preferred embodiment, this embodiment also provides a multi-task auditory attention detection device based on electroencephalogram (EEG) signals, such as... Figure 6 As shown, it includes:

[0132] Preprocessing module 10 is used to collect the user's sound source location data, and simultaneously collect the user's speech signal and EEG signal, and preprocess the speech signal and the EEG signal;

[0133] The multi-task decoding module 20 is used to establish a multi-task decoding model, using the user's sound source location data, the preprocessed speech signal and the EEG signal as training data for the multi-task decoding model, and using the trained multi-task decoding model to identify the user's auditory attention object.

[0134] The recognition module 30 uses the trained multi-task decoding model to identify the user's auditory attention objects.

[0135] In a preferred embodiment, this embodiment also provides a computer-readable storage medium, such as... Figure 7 As shown, the computer storage medium 400 has a computer program 41. When the computer program 41 is executed, it is used to implement the multi-task auditory attention detection method based on electroencephalogram (EEG) signals described above.

[0136] In summary, this invention proposes a multi-task auditory attention detection method and device based on electroencephalogram (EEG) signals, which can simultaneously achieve two tasks: spatial localization and speech envelope reconstruction. It can effectively extract spatial information from EEG signals and use it to improve the accuracy of speech envelope reconstruction, adapting to different application scenarios and needs.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-task auditory attention detection method based on EEG signals, characterized in that: The steps include: Collecting the user's sound source location data, and simultaneously collecting the user's voice signal and EEG signal, and preprocessing the voice signal and the EEG signal; Establishing a multi-task decoding model, using the sound source position data of the user, the pre-processed speech signal and the EEG signal as training data of the multi-task decoding model, and training the multi-task decoding model with the training data; Using the trained multi-task decoding model to identify the user's auditory attention object; The process of using the trained multi-task decoding model to identify the user's auditory attention object specifically includes the following steps: Inputting the preprocessed EEG signal into the multi-task decoding model, extracting common features and spatial information features, splicing and fusing the common features and the spatial information features, and reconstructing the speech envelope according to the fused features; Calculating the Pearson correlation coefficient between each candidate speech envelope and the reconstructed speech envelope, and taking the candidate speech envelope with the largest Pearson correlation coefficient as the user's auditory attention object; Identify and verify the user's auditory attention object according to the positioning result of the spatial positioning module; According to the calculated Pearson correlation coefficient and spatial positioning result, the best matching candidate speech envelope or auditory attention direction is selected as the user's auditory attention object, and the result is output for further processing or application.

2. The multi-task auditory attention detection method based on EEG signals as claimed in claim 1, characterized in that: The training data is divided into a training set and a validation set; Using the validation set to evaluate the multi-task decoding model, the evaluation content includes the loss function value, the accuracy of reconstructed speech envelope and the classification accuracy of spatial positioning; The parameters and architecture of the multi-task decoding model are adjusted according to the evaluation results to optimize the feature extraction and fusion process.

3. The multi-task auditory attention detection method based on EEG signals as claimed in claim 1, characterized in that: The preprocessing of the speech signal and the EEG signal, wherein the preprocessing of the speech signal includes bandpass filtering and downsampling, specifically includes the following steps: Performing bandpass filtering on the speech signal to extract characteristic frequency bands; Calculating the envelope of the speech signal after the bandpass filtering, the envelope of the speech signal is the speech envelope; The calculated speech envelope is sampled, and the sampling rate is kept consistent with the sampling rate of the electroencephalogram signal to obtain the preprocessed speech signal.

4. The multi-task auditory attention detection method based on EEG signals as claimed in claim 1, characterized in that: The preprocessing of the EEG signal specifically includes the following steps: re-referencing the EEG signal and then performing band-pass filtering; Sampling the EEG signal after bandpass filtering, with the sampling rate being consistent with the sampling rate of the speech signal; Finally, the sampled EEG signal is normalized.

5. The multi-task auditory attention detection method based on EEG signals as claimed in claim 3, characterized in that: The multi-task decoding model includes an input layer, a sharing layer, a spatial positioning module, a feature fusion module, a speech envelope reconstruction module and a calculation module; Wherein, the input layer is used to input the EEG signal; The shared layer includes a convolutional neural network, and the shared layer is used to extract common features of the EEG signal; The spatial positioning module includes a deep learning model, and the spatial positioning module is used to capture the spatial relationship of the EEG signal and extract spatial information features; The feature fusion module is used to splice and fuse the common features extracted by the shared layer and the spatial information features extracted by the spatial positioning module; The speech envelope reconstruction module includes a deep learning decoder, and the speech envelope reconstruction module is used to reconstruct the speech envelope using the fused features to obtain the reconstructed features; The calculation module is used to calculate the correlation coefficient of the reconstructed feature, where the correlation coefficient includes a combination of a Pearson correlation coefficient and a mean square error.

6. The multi-task auditory attention detection method based on EEG signals as claimed in claim 5, characterized in that: The convolutional neural network is a one-dimensional convolutional neural network with a convolution kernel size of 3 and a step size of 1. The number of convolution kernels is equal to the number of channels of the EEG signal.

7. The multi-task auditory attention detection method based on EEG signals as claimed in claim 5, characterized in that: The spatial positioning module also includes a classifier, which includes a global average pooling layer and two fully connected layers; Wherein, the global average pooling layer is used to perform dimensionality reduction processing on the spatial information features; The two fully connected layers are used to process and classify the spatial information features.

8. The multi-task auditory attention detection method based on EEG signals as claimed in claim 7, characterized in that: The feature fusion module is used to splice and fuse the common features extracted by the shared layer and the spatial information features extracted by the spatial positioning module, and specifically includes the following: The general features and the spatial information features are concatenated in the feature dimension to form a new feature matrix, thereby obtaining an initial feature matrix dimension; The concatenated features are transformed using a fully connected layer, and the concatenated features are normalized using a multi-head attention mechanism. The feature fusion module outputs the normalized feature matrix.

9. A multi-task auditory attention detection device based on EEG signals, characterized in that: include: A preprocessing module is used to collect the user's sound source location data, collect the user's voice signal and EEG signal, and preprocess the voice signal and the EEG signal; A multi-task decoding module, used to establish a multi-task decoding model, use the user's sound source position data, the pre-processed speech signal and the EEG signal as training data for the multi-task decoding model, and train the multi-task decoding model with the training data; A recognition module, using the trained multi-task decoding model to recognize the user's auditory attention object; The recognition module uses the trained multi-task decoding model to recognize the user's auditory attention object, and the recognition process specifically includes the following: Inputting the preprocessed EEG signal into the multi-task decoding model, extracting common features and spatial information features, splicing and fusing the common features and the spatial information features, and reconstructing the speech envelope according to the fused features; Calculating the Pearson correlation coefficient between each candidate speech envelope and the reconstructed speech envelope, and taking the candidate speech envelope with the largest Pearson correlation coefficient as the user's auditory attention object; Identify and verify the user's auditory attention object according to the positioning result of the spatial positioning module; According to the calculated Pearson correlation coefficient and spatial positioning result, the best matching candidate speech envelope or auditory attention direction is selected as the user's auditory attention object, and the result is output for further processing or application.

Images