Auditory attention decoding method, device, equipment and medium based on multi-sound source scene

Abstract

The embodiment of the present invention provides an auditory attention decoding method, device, equipment and medium based on a multi-sound source scenario, which belongs to the field of EEG signal technology. The method includes: obtaining an initial EEG signal to which an initial user responds under a first sound stimulus, a multi-sound source speech signal of the first sound stimulus and a sound source label of the multi-sound source speech signal; preprocessing the initial EEG signal and the multi-sound source speech signal to obtain an EEG signal to be processed of the initial EEG signal and a speech signal to be processed of the multi-sound source speech signal; training an initial auditory attention decoding model according to the EEG signal to be processed, the speech signal to be processed and the sound source label to obtain a target auditory attention decoding model; obtaining a target EEG signal to which a target user responds under a second sound stimulus and a target speech signal of the second sound stimulus; using the target auditory attention decoding model to perform auditory attention decoding on the target EEG signal and the target speech signal to obtain an attention target.

Description

Technical Field

[0001] The present invention relates to the field of electroencephalogram signal technology, and in particular to an auditory attention decoding method, device, equipment and medium based on a multi-sound source scenario. Background Art

[0002] In a multi-sound source environment, the human brain can relatively easily focus on a certain speaker while effectively suppressing the interference of other sounds. However, for people wearing hearing aids or cochlear implants, although these devices have certain background noise reduction functions, they cannot accurately identify the auditory object that the wearer is paying attention to in complex multi-person conversation scenarios, making it difficult for users to focus on specific voice content, which can easily lead to social isolation and reduced quality of life. In response to the above problems, the relevant technology can decode auditory attention through EEG data to accurately identify the auditory object that the wearer is paying attention to, but the current technology usually requires a longer time window when decoding auditory attention, ranging from a few seconds to tens of seconds, which leads to poor real-time performance and inability to timely feedback the user's attention status. In addition, the existing algorithms do not support scenarios where users frequently switch their attention objects, especially in noisy environments or multi-person conversation scenarios, and it is difficult to quickly respond to the user's attention shift. More importantly, the existing decoding auditory attention algorithm fails to effectively integrate voice information, resulting in insufficient robustness of the algorithm when facing different audio stimuli, thereby further reducing the accuracy of attention decoding in noisy environments. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a method, device, equipment and medium for auditory attention decoding based on a multi-sound source scenario, aiming to solve the problem that the existing decoding auditory attention algorithm fails to effectively integrate speech information, resulting in insufficient robustness of the algorithm when facing different audio stimuli, thereby reducing the accuracy of attention decoding in noisy environments.

[0004] In a first aspect, an embodiment of the present invention provides an auditory attention decoding method based on a multi-sound source scenario, comprising:

[0005] Obtaining target object data corresponding to multiple initial users in a multi-sound source scenario, the target object data including an initial EEG signal to which the initial user responds under a first sound stimulus, a multi-sound source speech signal corresponding to the first sound stimulus, and a sound source label corresponding to the multi-sound source speech signal, the sound source label being used to characterize position information of a speaking object that the initial user needs to pay attention to;

[0006] Preprocessing the initial EEG signal and the multi-source speech signal respectively to obtain an EEG signal to be processed corresponding to the initial EEG signal and a speech signal to be processed corresponding to the multi-source speech signal;

[0007] Performing model training on an initial auditory attention decoding model according to the to-be-processed electroencephalogram signal, the to-be-processed speech signal, and the sound source label to obtain a target auditory attention decoding model;

[0008] Obtaining user data corresponding to a target user in a multi-sound source scenario, the user data including a target EEG signal to which the target user responds under a second sound stimulus and a target speech signal corresponding to the second sound stimulus;

[0009] The target auditory attention decoding model is used to perform auditory attention decoding on the target EEG signal and the target speech signal to obtain the attention target corresponding to the target user.

[0010] In a second aspect, an embodiment of the present invention provides an auditory attention decoding device based on a multi-sound source scenario, comprising:

[0011] A data acquisition module, used to obtain target object data corresponding to multiple initial users in a multi-sound source scenario, wherein the target object data includes an initial EEG signal to which the initial user responds under a first sound stimulus, a multi-sound source speech signal corresponding to the first sound stimulus, and a sound source label corresponding to the multi-sound source speech signal, wherein the sound source label is used to characterize the position information of the speaking object that the initial user needs to pay attention to;

[0012] A data processing module, used for preprocessing the initial EEG signal and the multi-source speech signal to obtain the EEG signal to be processed corresponding to the initial EEG signal and the speech signal to be processed corresponding to the multi-source speech signal;

[0013] A model training module, used for training an initial auditory attention decoding model according to the to-be-processed electroencephalogram signal, the to-be-processed speech signal and the sound source label, to obtain a target auditory attention decoding model;

[0014] A data acquisition module, used to obtain user data corresponding to a target user in a multi-sound source scenario, wherein the user data includes a target EEG signal of the target user in response to a second sound stimulus and a target speech signal corresponding to the second sound stimulus;

[0015] The target recognition module is used to use the target auditory attention decoding model to perform auditory attention decoding on the target EEG signal and the target speech signal to obtain the attention target corresponding to the target user.

[0016] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any one of the auditory attention decoding methods based on a multi-sound source scenario provided in the specification of the present invention are implemented.

[0017] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any step of the auditory attention decoding method based on a multi-sound source scenario provided in the specification of the present invention.

[0018] The embodiment of the present invention provides an auditory attention decoding method, device, equipment and medium based on a multi-sound source scenario, the method comprising: obtaining target object data corresponding to multiple initial users in a multi-sound source scenario, the target object data comprising an initial EEG signal to which the initial user responds under a first sound stimulus, a multi-sound source voice signal corresponding to the first sound stimulus and a sound source label corresponding to the multi-sound source voice signal, the sound source label being used to characterize the position information of the speaking object to which the initial user needs to pay attention; preprocessing the initial EEG signal and the multi-sound source voice signal respectively to obtain an EEG signal to be processed corresponding to the initial EEG signal and a voice signal to be processed corresponding to the multi-sound source voice signal; training an initial auditory attention decoding model according to the EEG signal to be processed, the voice signal to be processed and the sound source label to obtain a target auditory attention decoding model; obtaining user data corresponding to a target user in a multi-sound source scenario, the user data comprising a target EEG signal to which the target user responds under a second sound stimulus and a target voice signal corresponding to the second sound stimulus; performing auditory attention decoding on the target EEG signal and the target voice signal using the target auditory attention decoding model to obtain an attention target corresponding to the target user. In multi-sound source scenarios, this method can better adapt to the real auditory environment by training the target auditory attention decoding model to identify and distinguish different sound sources, improve the robustness in complex scenarios when facing different audio stimuli, and can more accurately decode the target user's attention target by training the target auditory attention decoding model, thereby improving the accuracy and reliability of auditory attention analysis and further improving user satisfaction. This method also solves the problem that the existing decoding auditory attention algorithm fails to effectively integrate voice information, resulting in insufficient robustness of the algorithm when facing different audio stimuli, thereby reducing the accuracy of attention decoding in noisy environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of an auditory attention decoding method based on a multi-sound source scenario provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the model structure of an initial auditory attention decoding model provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of the module structure of an auditory attention decoding device based on a multi-sound source scenario provided by an embodiment of the present invention;

[0023] Figure 4 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0026] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0027] The embodiment of the present invention provides a method, device, equipment and medium for auditory attention decoding based on a multi-sound source scene. The auditory attention decoding method based on a multi-sound source scene can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant and a wearable device. The terminal device can be a server or a server cluster.

[0028] In a cocktail party, the human brain can easily focus on one speaker and suppress other sounds. However, for people wearing hearing aids or cochlear implants, although hearing aids can reduce background noise to a certain extent, they cannot effectively identify the auditory object that the wearer is paying attention to in a multi-person conversation, making it difficult for users to focus on the voice content, which can easily lead to social isolation and a decline in quality of life.

[0029] EEG signals have a speech envelope phenomenon for the auditory signals that are of interest. This feature helps to achieve Auditory Attention Decoding (AAD) and improve the information processing ability of people with hearing impairments in complex acoustic interaction scenarios. However, the existing technology has the following limitations: First, from the perspective of EEG data types, the existing AAD algorithm is mainly based on EEG signals collected by non-portable EEG devices and wet electrodes for algorithm verification, and lacks effective evaluation and corresponding optimization of portable dry electrode EEG data, which hinders the combination of AAD algorithms and hearing aids in natural scenarios. Most of the existing EEG acquisition devices are based on wet electrodes, and wet electrode devices usually require conductive paste, which is inconvenient to use and can easily cause discomfort when worn for a long time. The use of low-power dry electrode devices can meet the user's portable needs. More importantly, the method of using speech envelopes to identify auditory attention is greatly affected by the time window, and it takes a long time to accurately classify the voice stream of interest. There is a lack of research on the scenario of frequently switching the object of auditory attention, which leads to the overall inability of the product to match the actual needs of users. Existing hearing aids cannot identify the auditory object of user attention in multi-person conversations, affecting the quality of life. The auditory attention decoding (ADD) algorithm is mainly based on wet electrode EEG data, lacking optimization for portable dry electrodes, which limits its integration with hearing aids. Existing algorithms require a long time to recognize speech streams, and lack research on scenarios where auditory objects are frequently switched, which cannot meet user needs.

[0030] Existing auditory attention decoding techniques are often based on complex signal processing algorithms, which have the following shortcomings:

[0031] 1. Long processing time: Many existing algorithms require a long time window when decoding auditory attention, usually between several seconds and tens of seconds, resulting in poor real-time performance and inability to provide timely feedback on the user's attention status.

[0032] 2. Poor dynamic switching: Existing algorithms do not support scenarios where users frequently switch their attention objects well, especially in noisy environments or multi-person conversation scenarios, and it is difficult to quickly respond to users' attention shifts.

[0033] 3. Poor robustness: The lack of fusion of speech information leads to insufficient robustness of the algorithm under different audio stimuli, and the existing algorithm has poor robustness to noise. When the frequency of environmental noise is high, the existing frequency domain feature extraction method is difficult to effectively filter the noise. This greatly reduces the accuracy of auditory attention decoding in noisy scenes.

[0034] Many existing auditory attention decoding algorithms in the prior art are mainly developed and verified based on wet electrode EEG signals, lacking optimized processing of dry electrode acquisition signals, resulting in poor performance of dry electrode devices under these algorithms. For example, the model generalization ability is poor: the existing model has been optimized for wet electrode data processing, but it cannot maintain the same decoding accuracy when processing dry electrode data, especially due to the large noise of dry electrode signals, the lack of robustness of existing algorithms, and the lack of optimization of existing algorithms for portable devices, which makes these algorithms unable to be effectively combined with portable dry electrode devices, and the application scenarios are limited.

[0035] Therefore, an auditory attention decoding method based on multi-sound source scenarios is urgently needed to solve the problem that the existing decoding auditory attention algorithm fails to effectively integrate speech information, resulting in insufficient robustness of the algorithm when facing different audio stimuli, thereby reducing the accuracy of attention decoding in noisy environments.

[0036] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0037] Please refer to Figure 1 , Figure 1 A flowchart of an auditory attention decoding method based on a multi-sound source scenario is provided in an embodiment of the present invention.

[0038] like Figure 1 As shown, the auditory attention decoding method based on a multi-sound source scenario includes steps S101 to S105.

[0039] Step S101, obtaining target object data corresponding to multiple initial users in a multi-sound source scenario, wherein the target object data includes an initial EEG signal to which the initial user responds under a first sound stimulation, a multi-sound source speech signal corresponding to the first sound stimulation, and a sound source label corresponding to the multi-sound source speech signal, wherein the sound source label is used to characterize position information of a speaking object that the initial user needs to pay attention to.

[0040] Exemplarily, for example, if the multi-sound source scene is a cocktail party scene, target object data corresponding to multiple initial users in the multi-sound source scene are collected, and the target object data include an initial EEG signal to which the initial user responds to a first sound stimulus, a multi-sound source speech signal corresponding to the first sound stimulus, and a sound source label corresponding to the multi-sound source speech signal, and the sound source label represents the corresponding left-side speaking object or right-side speaking object, that is, the sound source label is used to represent the position information of the speaking object that the initial user needs to pay attention to.

[0041] Step S102: pre-process the initial EEG signal and the multi-source speech signal respectively to obtain the to-be-processed EEG signal corresponding to the initial EEG signal and the to-be-processed speech signal corresponding to the multi-source speech signal.

[0042] Exemplarily, a bandpass filter is applied to remove low-frequency drift and high-frequency noise in the initial EEG signal to obtain the EEG signal to be processed corresponding to the initial EEG signal.

[0043] Exemplarily, a noise suppression algorithm (such as spectral subtraction, Wiener filtering, etc.) is used to remove background noise and other non-target sounds to obtain a speech signal to be processed corresponding to the multi-source speech signal.

[0044] In some embodiments, the initial EEG signal and the multi-source speech signal are preprocessed respectively to obtain the EEG signal to be processed corresponding to the initial EEG signal and the speech signal to be processed corresponding to the multi-source speech signal, including: filtering the initial EEG signal with a filter to obtain a first EEG signal; performing notch filtering on the first EEG signal to obtain a second EEG signal; downsampling the second EEG signal to obtain a third EEG signal, and mean normalizing the third EEG signal to obtain the EEG signal to be processed; extracting Mel-spectrogram features from the multi-source speech signal to obtain the corresponding speech signal to be processed.

[0045] Exemplarily, the initial EEG signal is filtered using a filter to obtain a first EEG signal, the first EEG signal is then notch filtered to obtain a second EEG signal, the second EEG signal is then downsampled to obtain a third EEG signal, and the third EEG signal is mean normalized to obtain the EEG signal to be processed.

[0046] For example, the initial EEG signal is preprocessed, and an 8th-order Butterworth filter is used to perform 0.1-50 Hz bandpass filtering on the initial EEG signal to obtain a first EEG signal. Then, a 49-51 Hz notch filter is used to perform notch filtering on the first EEG signal to remove power frequency interference to obtain a second EEG signal. The second EEG signal is then downsampled to 128 Hz to obtain a third EEG signal. Finally, the third EEG signal is mean normalized to obtain the preprocessed EEG signal to be processed.

[0047] Exemplarily, the mel spectrum feature extraction operation is performed on the multi-source speech signal to obtain the corresponding speech signal to be processed. The multi-source speech signal is preprocessed to extract the mel spectrum feature of the multi-source speech signal, so as to more effectively represent the envelope structure of the multi-source speech signal, so as to provide support for the subsequent target auditory attention decoding model to better capture the sound object that the user pays attention to, and improve the robustness of the target auditory attention decoding model to noise.

[0048] Step S103: training the initial auditory attention decoding model according to the EEG signal to be processed, the speech signal to be processed and the sound source label to obtain a target auditory attention decoding model.

[0049] Exemplarily, the preprocessed EEG signals to be processed, the speech signals to be processed, and the corresponding sound source labels are integrated into a complete data set. Ensure that each data sample contains the EEG signals to be processed, the speech signals to be processed, and the corresponding sound source labels. Divide the data set into a training set, a validation set, and a test set. Usually, the training set is used for model training, the validation set is used for model selection, and the test set is used to evaluate model performance.

[0050] Exemplarily, features related to auditory attention are extracted from the EEG signal to be processed, such as time domain features, frequency domain features, or time-frequency features. Acoustic features, such as Mel-frequency cepstral coefficients (MFCC), speech prosody features, etc., are extracted from the speech signal to be processed. Thus, a model architecture is selected, such as a deep learning model (such as a convolutional neural network CNN, a recurrent neural network RNN, etc.) or a traditional machine learning model, and a loss function is defined, such as a cross entropy loss function, a mean square error loss function, etc. An optimizer is selected, such as gradient descent (SGD), Adam, etc., to update model parameters during training.

[0051] Exemplarily, the EEG signal features, speech signal features and corresponding sound source labels of the training set are loaded into the model. The initial auditory attention decoding model is trained using the training data in an iterative manner. In each iterative step, the prediction value is calculated by forward propagation, and then the loss function is calculated, and the model parameters are updated by back propagation. Monitor the loss function value and model performance during the training process to ensure that the model is learning and optimizing correctly, and then calculate the performance indicators of the model on the validation set, such as accuracy, recall rate, F1 score, etc., and adjust the hyperparameters or structure of the model according to the performance indicators to obtain better performance, and then use the test set data to test the final model, calculate the performance indicators of the model on the test set to evaluate the generalization ability of the model, and finally save the best performing model after training, verification and testing as the target auditory attention decoding model.

[0052] In some embodiments, the initial auditory attention decoding model is trained according to the EEG signal to be processed, the speech signal to be processed and the sound source label to obtain a target auditory attention decoding model, including: using the time domain feature extraction layer of the initial auditory attention decoding model to perform time domain feature extraction on the EEG signal to be processed to obtain corresponding target time domain feature information; using the frequency domain feature extraction layer of the initial auditory attention decoding model to perform frequency domain feature extraction on the EEG signal to be processed to obtain corresponding target frequency domain feature information; using the audio feature extraction layer of the initial auditory attention decoding model to perform audio feature extraction on the speech signal to be processed to obtain corresponding target audio feature information; using the feature fusion mapping layer of the initial auditory attention decoding model to perform model training according to the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label to obtain the target auditory attention decoding model.

[0053] Exemplarily, the initial auditory attention decoding model includes a time domain feature extraction layer, a frequency domain feature extraction layer, an audio feature extraction layer, and a feature fusion mapping layer.

[0054] Exemplarily, the time domain feature extraction layer in the initial auditory attention decoding model is used. The time domain feature extraction layer includes operations such as time domain filtering, windowing, and average calculation. The EEG signal to be processed is input into the time domain feature extraction layer. Through the time domain feature extraction layer, the features of the EEG signal in the time domain, such as mean, variance, waveform length, autocorrelation features, etc., are calculated, and then the extracted time domain features are determined as the target time domain feature information.

[0055] Exemplarily, the frequency domain feature extraction layer in the initial auditory attention decoding model is used. The frequency domain feature extraction layer includes operations such as fast Fourier transform and spectrum analysis. The EEG signal to be processed is input into the frequency domain feature extraction layer, and the frequency domain features of the EEG signal to be processed are calculated through the frequency domain feature extraction layer, such as power spectrum density, spectrum centroid, bandwidth, etc., and then the extracted frequency domain features are determined as the target frequency domain feature information.

[0056] Exemplarily, the audio feature extraction layer in the initial auditory attention decoding model is used. The audio feature extraction layer includes operations such as Mel Frequency Cepstral Coefficient (MFCC) calculation and speech prosody feature extraction. The speech signal to be processed is input into the audio feature extraction layer. Through the audio feature extraction layer, the acoustic features of the speech signal, such as MFCC, spectral envelope, fundamental frequency, etc., are calculated, and then the extracted audio features are determined as the target audio feature information.

[0057] Exemplarily, the feature fusion mapping layer in the initial auditory attention decoding model is used. The feature fusion mapping layer includes operations such as feature concatenation, dimensionality reduction, and mapping, and then fuses the target time domain feature information, the target frequency domain feature information, and the target audio feature information, so as to predict the sound source of the fused features to obtain the sound source prediction result, and then compare the sound source prediction result with the sound source label to obtain the corresponding loss value, so as to train the initial auditory attention decoding model according to the loss value until the target auditory attention decoding model is obtained.

[0058] In some embodiments, the time domain feature extraction layer includes multiple convolution branches and feature fusion networks, each of the convolution branches includes at least an input convolution block, a basic convolution block and an output convolution block, and the time domain feature extraction layer of the initial auditory attention decoding model is used to extract the time domain features of the EEG signal to be processed to obtain the corresponding target time domain feature information, including: using the first input convolution layer, the first batch normalization layer, the first activation layer and the first pooling layer of the input convolution block to perform data processing on the EEG signal to be processed in sequence to obtain the first feature information; using the second batch normalization layer, the second activation layer and the first convolution layer of the basic convolution block to perform data processing on the first feature information to obtain the second feature information; using the third batch normalization layer, the third activation layer, the second convolution layer and the second pooling layer of the output convolution block to perform data processing on the second feature information to obtain the third feature information; obtaining the third feature information corresponding to each convolution branch, and using the feature fusion network to splice all the third feature information and then perform a flattening operation to obtain the corresponding target time domain feature information.

[0059] Exemplarily, the time domain feature extraction layer includes multiple convolution branches and a feature fusion network, and each convolution branch includes at least an input convolution block, a basic convolution block and an output convolution block.

[0060] For example, multi-scale convolution extraction based on time domain features uses k convolution kernels of different sizes to enable the network to have different receptive fields. Convolution kernels of different scales are used to capture short-term and long-term feature changes in EEG signals, extract more complex time domain features, and improve the recognition ability of complex attention patterns. Each convolution branch contains an input convolution block, a basic convolution block, and an output convolution block.

[0061] Exemplarily, the first input convolution layer, the first batch of normalization layers, the first activation layer and the first pooling layer of the input convolution block are used to process the EEG signal to be processed in sequence to obtain the first feature information.

[0062] For example, the input convolution block includes the first input convolution layer, the first normalization layer, the first activation layer and the first pooling layer. The first input convolution layer has an input channel of 1, an output channel of 32, a convolution kernel size of 3*3, a step size of 1, and an external padding of 1; the first activation layer uses the Relu activation function; the first pooling layer uses the maximum pooling, the pooling kernel is 3*3, the step size is 2, and the external padding is 1. Thus, the EEG signal to be processed is processed in sequence according to the first input convolution layer, the first normalization layer, the first activation layer and the first pooling layer of the input convolution block to obtain the first feature information.

[0063] Exemplarily, the first feature information is processed using the second batch normalization layer, the second activation layer and the first convolution layer of the basic convolution block to obtain the second feature information. The basic convolution block can be repeated multiple times.

[0064] For example, the basic convolution block is repeated three times, and each basic convolution block contains the second batch of normalization layers, the second activation layer, and the first convolution layer, wherein the second activation layer uses the Relu activation function, the input channel of the first convolution layer is 32, the output channel is 32, the convolution kernel size is j*j, where j is the convolution kernel size of the branch, and the filling mode is the same, and then the first feature information is passed through the first basic convolution block to obtain the first result, and then the first result is passed through the second basic convolution block to obtain the second result, and then the second result is passed through the third basic convolution block to obtain the second feature information. For example, the structure of multiple convolution branches in the time domain feature extraction layer is shown in the following table.

[0065] Table 1 Schematic diagram of the structure of multiple convolution branches in the time domain feature extraction layer

[0066] name Number of channels Kernel size Input size Output size Input convolutional block 1 3 （128，1，64，128） （128，32，32，64） Basic convolutional block 32 j （128，32，32，64） （128，32，32，64） Output convolution block 32 8 （128，32，32，64） （128，128）

[0067] Exemplarily, the third batch normalization layer, the third activation layer, the second convolution layer, and the second pooling layer of the output convolution block are used to perform data processing on the second feature information to obtain the third feature information.

[0068] For example, the output convolution block includes the third batch normalization layer, the third activation layer, the second convolution layer and the second pooling layer, wherein the third activation layer uses the Relu activation function, the second convolution layer has 32 input channels, 8 output channels, and a convolution kernel size of j*j, where j is the convolution kernel size of the branch, the filling mode is the same, the second pooling layer is average pooling, and the pooling kernel size is 16*32. Thus, the third batch normalization layer, the third activation layer, the second convolution layer and the second pooling layer of the output convolution block are used to process the second feature information and obtain the third feature information.

[0069] Exemplarily, the third feature information corresponding to each convolution branch is obtained, and all the third feature information is concatenated and flattened using a feature fusion network to obtain the corresponding target time domain feature information.

[0070] For example, the time domain feature extraction layer includes four convolution branches, and the third feature information corresponding to the four convolution branches is obtained. Then, the four third feature information are concatenated and flattened to obtain the corresponding target time domain feature information.

[0071] In some embodiments, the frequency domain feature extraction layer includes a fast Fourier transform, a long short-term memory network, a fully connected layer, and an activation function layer. The frequency domain feature extraction layer of the initial auditory attention decoding model is used to perform frequency domain feature extraction on the EEG signal to be processed to obtain corresponding target frequency domain feature information, including: using the fast Fourier transform to convert the EEG signal to be processed from the time domain to the frequency domain to obtain first frequency domain feature information; using the long short-term memory network to perform feature embedding on the first frequency domain feature information to obtain second frequency domain feature information; using the fully connected layer to perform feature mapping on the second frequency domain feature information to obtain third frequency domain feature information; using the activation function layer to activate the third frequency domain feature information to obtain the target frequency domain feature information.

[0072] Exemplarily, the frequency domain feature extraction layer includes a fast Fourier transform, a long short-term memory network, a fully connected layer, and an activation function layer.

[0073] Exemplarily, based on the frequency domain feature extraction layer, the EEG signals to be processed in different frequency bands are refined, especially in combination with the characteristics of different frequencies in the EEG signals to be processed. First, the fast Fourier transform is used to convert the EEG signals to be processed from the time domain to the frequency domain to obtain the first frequency domain feature information. In order to avoid the overfitting problem that may occur when a complex network learns the first frequency domain feature information, a single-layer long short-term memory network is used to extract the first frequency domain feature information for feature embedding to obtain the second frequency domain feature information. Then, a fully connected layer with 128 neurons is used to perform feature mapping on the second frequency domain feature information to obtain the third frequency domain feature information. Finally, the activation function layer uses Relu to activate the third frequency domain feature information to obtain the target frequency domain feature information.

[0074] In some embodiments, the audio feature extraction layer includes at least a first audio convolution block, a second audio convolution block and a third audio convolution block, the first audio convolution block includes a second input convolution layer, a fourth batch normalization layer, and a fourth activation layer, the second audio convolution block includes a third input convolution layer, a fifth batch normalization layer, and a sixth activation layer, and the third audio convolution block includes a fourth input convolution layer, a sixth batch normalization layer, and a seventh activation layer; the audio feature extraction layer of the initial auditory attention decoding model is used to extract audio features from the speech signal to be processed to obtain corresponding target audio feature information, It includes: performing audio feature extraction on the speech signal to be processed according to the second input convolution layer, the fourth batch normalization layer, and the fourth activation layer of the first audio convolution block to obtain a first audio feature; performing audio feature extraction on the first audio feature according to the third input convolution layer, the fifth batch normalization layer, and the sixth activation layer of the second audio convolution block to obtain a second audio feature; performing audio feature extraction on the second audio feature according to the fourth input convolution layer, the sixth batch normalization layer, and the seventh activation layer of the third audio convolution block to obtain the target audio feature information.

[0075] Exemplarily, the audio feature extraction layer includes at least a first audio convolution block, a second audio convolution block, and a third audio convolution block, the first audio convolution block includes a second input convolution layer, a fourth batch normalization layer, and a fourth activation layer, the second audio convolution block includes a third input convolution layer, a fifth batch normalization layer, and a sixth activation layer, and the third audio convolution block includes a fourth input convolution layer, a sixth batch normalization layer, and a seventh activation layer.

[0076] Exemplarily, according to the second input convolution layer, the fourth batch normalization layer, and the fourth activation layer of the first audio convolution block, audio feature extraction is performed on the processed speech signal to obtain the first audio feature; according to the third input convolution layer, the fifth batch normalization layer, and the sixth activation layer of the second audio convolution block, audio feature extraction is performed on the first audio feature to obtain the second audio feature; according to the fourth input convolution layer, the sixth batch normalization layer, and the seventh activation layer of the third audio convolution block, audio feature extraction is performed on the second audio feature to obtain target audio feature information.

[0077] For example, speech feature extraction uses multi-layer audio convolution blocks to extract features from the processed speech signal, where each audio convolution block contains an input convolution layer, a batch normalization layer, and an activation layer. The input convolution layer has 1 input channel, 32 output channels, a convolution kernel size of 3*3, a step size of 1, and an external padding of 1; the activation layer uses the Relu activation function.

[0078] In some embodiments, the feature fusion mapping layer includes a feature level network and a classification network; the feature fusion mapping layer using the initial auditory attention decoding model performs model training according to the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label to obtain the target auditory attention decoding model, including: using the feature level network to perform adaptive weighted feature fusion on the target time domain feature information, the target frequency domain feature information and the target audio feature information to obtain splicing feature information; using the classification network to classify the sound source of the splicing feature information to obtain an initial classification result; performing error calculation based on the initial classification result and the sound source label to obtain a training error; and performing model parameter training on the initial auditory attention decoding model based on the training error to obtain the target auditory attention decoding model.

[0079] Exemplarily, the feature fusion mapping layer includes a feature level network and a classification network.

[0080] Exemplarily, the feature level network is used to perform adaptive weighted feature fusion on the target time domain feature information, the target frequency domain feature information and the target audio feature information to obtain the spliced ​​feature information

[0081] For example, the feature fusion mapping layer introduces a feature-level fusion with an adaptive weighting mechanism, and dynamically adjusts the weight distribution of time domain and frequency domain features according to different scenarios to obtain spliced ​​feature information to ensure decoding accuracy in different scenarios.

[0082] Exemplarily, a classification network is used to classify the sound source of the spliced ​​feature information to obtain an initial classification result; an error is calculated based on the initial classification result and the sound source label to obtain a training error; and model parameters of the initial auditory attention decoding model are trained based on the training error to obtain a target auditory attention decoding model.

[0083] For example, the data in the model training process is divided into training set, validation set and test set, and a 5-fold cross-validation method is used to verify the stability of the model. In each fold, the data volume of the training set, validation set and test set accounts for 60%, 20% and 20% of the total data volume respectively. The batch size in the training process is set to 128 to process data in parallel and speed up the training speed. The learning rate is set to 0.0001, the optimizer uses Adam, and the cross entropy loss function is used in the training process. Data verification is performed on public datasets to obtain the target auditory attention decoding model.

[0084] According to the above analysis, if Figure 2 As shown, the initial auditory attention decoding model includes a time domain feature extraction layer, a frequency domain feature extraction layer, an audio feature extraction layer, and a feature fusion mapping layer. Among them, the time domain feature extraction layer includes multiple convolution branches and feature fusion networks, each convolution branch includes at least an input convolution block, a basic convolution block, and an output convolution block, the frequency domain feature extraction layer includes a fast Fourier transform, a long short-term memory network, a fully connected layer, and an activation function layer, the audio feature extraction layer includes at least a first audio convolution block, a second audio convolution block, and a third audio convolution block, and the feature fusion mapping layer includes a feature level network and a classification network.

[0085] Step S104: obtaining user data corresponding to the target user in a multi-sound source scenario, wherein the user data includes a target EEG signal to which the target user responds under a second sound stimulation and a target speech signal corresponding to the second sound stimulation.

[0086] Exemplarily, user data corresponding to the target user in a multi-sound source scenario is collected, and the user data includes a target EEG signal to which the target user responds under a second sound stimulation and a target speech signal corresponding to the second sound stimulation.

[0087] For example, target EEG signals near the temporal lobe of the brain may be collected at 10 sites near the ears of the target user, and target speech signals corresponding to the second sound stimulation may be collected.

[0088] Step S105: Use the target auditory attention decoding model to perform auditory attention decoding on the target EEG signal and the target speech signal to obtain the attention target corresponding to the target user.

[0089] Exemplarily, the target auditory attention decoding model is used to perform auditory attention decoding on the target EEG signals and target speech signals in different time periods, and the speech source focused on in the current segment is determined to obtain the attention target corresponding to the target user. The attention target is the auditory attention projection target corresponding to the target user.

[0090] This application extracts EEG auditory attention features in multi-sound source scenarios from both the time domain and frequency domain. The target auditory attention decoding model fuses the time domain and frequency domain features of EEG signals. Through the combination of convolutional networks and long short-term memory networks, it can capture subtle changes in EEG signals, especially in complex auditory environments. The decoding accuracy is significantly improved. Most of the existing technologies focus on the extraction of a single modality and the adaptive ability of multiple time windows. The multi-scale convolution kernel network can effectively respond to the decoding needs under different time windows, which is not deeply explored by the existing technology. By constructing a multi-scale convolution kernel to extract time domain information combined with frequency domain information, the target auditory attention decoding model can dynamically adapt to the auditory attention differences of different subjects. Multi-scale convolution network combining time domain and frequency domain. The decoding speed and accuracy are greatly improved by combining the innovative multi-scale convolution network with time domain and frequency domain feature extraction technology. Compared with the existing technology, accurate attention decoding can be performed in a shorter time window to achieve better real-time performance.

[0091] Furthermore, this application combines EEG signals and audio signals at the same time. By fusing the Mel spectrum of the speech signal and using the Mel frequency scale that conforms to human auditory perception, it can more accurately capture the key features in the speech signal, especially in the processing of low-frequency and high-frequency signals. It has significant advantages. The target auditory attention decoding model can not only decode the user's attention target, but also perform speech recognition tasks at the same time, so that the model is more focused on the analysis of speech content during auditory decoding, and further improves the decoding accuracy. Compared with the prior art, this application not only has significant advantages in the accuracy of auditory attention decoding, but also hierarchically extracts the time-frequency features of the audio signal through a deep convolutional network, and deeply fuses it with the EEG signal, thereby enhancing the noise resistance and generalization ability of the target auditory attention decoding model.

[0092] See also Figure 3 , Figure 3An auditory attention decoding device 200 based on a multi-sound source scenario is provided in an embodiment of the present application. The auditory attention decoding device 200 based on a multi-sound source scenario includes a data acquisition module 201, a data processing module 202, a model training module 203, a data collection module 204, and a target recognition module 205, wherein the data acquisition module 201 is used to obtain target object data corresponding to multiple initial users in a multi-sound source scenario, and the target object data includes an initial EEG signal to which the initial user responds under a first sound stimulus, a multi-sound source speech signal corresponding to the first sound stimulus, and a sound source label corresponding to the multi-sound source speech signal, wherein the sound source label is used to characterize the position information of the speaking object that the initial user needs to pay attention to; the data processing module 202 is used to process the initial EEG signal and the multi-sound source speech signal; The speech signals are preprocessed respectively to obtain the EEG signal to be processed corresponding to the initial EEG signal and the speech signal to be processed corresponding to the multi-source speech signal; the model training module 203 is used to perform model training on the initial auditory attention decoding model according to the EEG signal to be processed, the speech signal to be processed and the sound source label to obtain the target auditory attention decoding model; the data acquisition module 204 is used to obtain the user data corresponding to the target user in the multi-source scene, and the user data includes the target EEG signal to which the target user responds under the second sound stimulation and the target speech signal corresponding to the second sound stimulation; the target recognition module 205 is used to perform auditory attention decoding on the target EEG signal and the target speech signal using the target auditory attention decoding model to obtain the attention target corresponding to the target user.

[0093] In some implementations, the data processing module 202 performs, during the process of preprocessing the initial EEG signal and the multi-source speech signal to obtain the EEG signal to be processed corresponding to the initial EEG signal and the speech signal to be processed corresponding to the multi-source speech signal:

[0094] Filtering the initial EEG signal using a filter to obtain a first EEG signal;

[0095] Performing notch filtering on the first EEG signal to obtain a second EEG signal;

[0096] Downsampling the second EEG signal to obtain a third EEG signal, and performing mean normalization processing on the third EEG signal to obtain the EEG signal to be processed;

[0097] The Mel-spectrogram feature extraction operation is performed on the multi-source speech signals to obtain the corresponding speech signals to be processed.

[0098] In some implementations, the model training module 203 performs, during the process of training the initial auditory attention decoding model according to the EEG signal to be processed, the speech signal to be processed, and the sound source label to obtain the target auditory attention decoding model:

[0099] Using the time domain feature extraction layer of the initial auditory attention decoding model to extract the time domain features of the EEG signal to be processed, to obtain the corresponding target time domain feature information;

[0100] Using the frequency domain feature extraction layer of the initial auditory attention decoding model to extract frequency domain features of the EEG signal to be processed to obtain corresponding target frequency domain feature information;

[0101] Using the audio feature extraction layer of the initial auditory attention decoding model to extract audio features from the speech signal to be processed to obtain corresponding target audio feature information;

[0102] The feature fusion mapping layer of the initial auditory attention decoding model is used to perform model training according to the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label to obtain the target auditory attention decoding model.

[0103] In some embodiments, the time domain feature extraction layer includes multiple convolution branches and feature fusion networks, each of the convolution branches includes at least an input convolution block, a basic convolution block and an output convolution block, and the model training module 203 performs the following steps in the process of extracting the time domain features of the EEG signal to be processed by the time domain feature extraction layer of the initial auditory attention decoding model to obtain the corresponding target time domain feature information:

[0104] Using the first input convolution layer, the first batch of normalization layers, the first activation layer and the first pooling layer of the input convolution block to sequentially process the EEG signal to be processed to obtain first feature information;

[0105] Using the second batch normalization layer, the second activation layer and the first convolution layer of the basic convolution block to perform data processing on the first feature information to obtain second feature information;

[0106] Using the third batch normalization layer, the third activation layer, the second convolution layer, and the second pooling layer of the output convolution block to perform data processing on the second feature information to obtain third feature information;

[0107] The third feature information corresponding to each of the convolution branches is obtained, and all the third feature information is concatenated and flattened using the feature fusion network to obtain the corresponding target time domain feature information.

[0108] In some embodiments, the frequency domain feature extraction layer includes a fast Fourier transform, a long short-term memory network, a fully connected layer, and an activation function layer. The model training module 203 performs the following steps in the process of extracting the frequency domain features of the EEG signal to be processed by the frequency domain feature extraction layer of the initial auditory attention decoding model to obtain the corresponding target frequency domain feature information:

[0109] Using the fast Fourier transform to convert the to-be-processed electroencephalogram signal from the time domain to the frequency domain to obtain first frequency domain feature information;

[0110] Using the long short-term memory network to perform feature embedding on the first frequency domain feature information to obtain second frequency domain feature information;

[0111] Using the fully connected layer to perform feature mapping on the second frequency domain feature information to obtain third frequency domain feature information;

[0112] The activation function layer is used to activate the third frequency domain feature information to obtain the target frequency domain feature information.

[0113] In some embodiments, the audio feature extraction layer includes at least a first audio convolution block, a second audio convolution block, and a third audio convolution block, the first audio convolution block includes a second input convolution layer, a fourth batch normalization layer, and a fourth activation layer, the second audio convolution block includes a third input convolution layer, a fifth batch normalization layer, and a sixth activation layer, and the third audio convolution block includes a fourth input convolution layer, a sixth batch normalization layer, and a seventh activation layer; the model training module 203 performs the following in the process of extracting audio features of the speech signal to be processed by the audio feature extraction layer of the initial auditory attention decoding model to obtain the corresponding target audio feature information:

[0114] Extracting audio features from the speech signal to be processed according to the second input convolution layer, the fourth batch normalization layer, and the fourth activation layer of the first audio convolution block to obtain a first audio feature;

[0115] Performing audio feature extraction on the first audio feature according to the third input convolution layer, the fifth batch normalization layer, and the sixth activation layer of the second audio convolution block to obtain a second audio feature;

[0116] The target audio feature information is obtained by performing audio feature extraction on the second audio feature according to the fourth input convolution layer, the sixth batch normalization layer, and the seventh activation layer of the third audio convolution block.

[0117] In some implementations, the feature fusion mapping layer includes a feature level network and a classification network; the model training module 203 performs the following steps in the process of performing model training according to the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label using the feature fusion mapping layer of the initial auditory attention decoding model to obtain the target auditory attention decoding model:

[0118] Using the feature level network, adaptively weighted feature fusion is performed on the target time domain feature information, the target frequency domain feature information and the target audio feature information to obtain splicing feature information;

[0119] Using the classification network to classify the sound source of the spliced ​​feature information to obtain an initial classification result;

[0120] Perform error calculation based on the initial classification result and the sound source label to obtain a training error;

[0121] Model parameters of the initial auditory attention decoding model are trained according to the training error to obtain the target auditory attention decoding model.

[0122] In some embodiments, the auditory attention decoding apparatus 200 based on a multi-sound source scenario can be applied to a terminal device.

[0123] It should be noted that, those skilled in the art can clearly understand that, for the sake of convenience and conciseness of description, the specific working process of the auditory attention decoding device 200 based on the multi-sound source scenario described above can refer to the corresponding process in the aforementioned embodiment of the auditory attention decoding method based on the multi-sound source scenario, and will not be repeated here.

[0124] See also Figure 4 , Figure 4 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0125] like Figure 4 As shown, the terminal device 300 includes a processor 301 and a memory 302 , and the processor 301 and the memory 302 are connected via a bus 303 , such as an I2C (Inter-integrated Circuit) bus.

[0126] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0127] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0128] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] The processor is used to run a computer program stored in the memory, and implement any one of the auditory attention decoding methods based on multiple sound source scenarios provided by the embodiments of the present invention when executing the computer program.

[0130] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0131] Obtaining target object data corresponding to multiple initial users in a multi-sound source scenario, the target object data including an initial EEG signal to which the initial user responds under a first sound stimulus, a multi-sound source speech signal corresponding to the first sound stimulus, and a sound source label corresponding to the multi-sound source speech signal, the sound source label being used to characterize position information of a speaking object that the initial user needs to pay attention to;

[0132] Preprocessing the initial EEG signal and the multi-source speech signal respectively to obtain an EEG signal to be processed corresponding to the initial EEG signal and a speech signal to be processed corresponding to the multi-source speech signal;

[0133] Performing model training on an initial auditory attention decoding model according to the to-be-processed electroencephalogram signal, the to-be-processed speech signal, and the sound source label to obtain a target auditory attention decoding model;

[0134] Obtaining user data corresponding to a target user in a multi-sound source scenario, the user data including a target electroencephalogram signal of the target user in response to a second sound stimulus and a target speech signal corresponding to the second sound stimulus;

[0135] The target auditory attention decoding model is used to perform auditory attention decoding on the target EEG signal and the target speech signal to obtain the attention target corresponding to the target user.

[0136] In some implementations, the processor 301 performs, during the process of preprocessing the initial EEG signal and the multi-source speech signal respectively to obtain the EEG signal to be processed corresponding to the initial EEG signal and the speech signal to be processed corresponding to the multi-source speech signal:

[0137] Filtering the initial EEG signal using a filter to obtain a first EEG signal;

[0138] Performing notch filtering on the first EEG signal to obtain a second EEG signal;

[0139] Downsampling the second EEG signal to obtain a third EEG signal, and performing mean normalization processing on the third EEG signal to obtain the EEG signal to be processed;

[0140] The Mel-spectrogram feature extraction operation is performed on the multi-source speech signals to obtain the corresponding speech signals to be processed.

[0141] In some implementations, the processor 301 performs, during the process of training the initial auditory attention decoding model according to the EEG signal to be processed, the speech signal to be processed, and the sound source label to obtain the target auditory attention decoding model:

[0142] Using the time domain feature extraction layer of the initial auditory attention decoding model to extract the time domain features of the EEG signal to be processed, to obtain the corresponding target time domain feature information;

[0143] Using the frequency domain feature extraction layer of the initial auditory attention decoding model to extract frequency domain features of the EEG signal to be processed, to obtain corresponding target frequency domain feature information;

[0144] Using the audio feature extraction layer of the initial auditory attention decoding model to extract audio features from the speech signal to be processed to obtain corresponding target audio feature information;

[0145] The feature fusion mapping layer of the initial auditory attention decoding model is used to perform model training according to the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label to obtain the target auditory attention decoding model.

[0146] In some embodiments, the time domain feature extraction layer includes multiple convolution branches and feature fusion networks, each of the convolution branches includes at least an input convolution block, a basic convolution block, and an output convolution block. The processor 301 performs the following steps in the process of extracting the time domain features of the EEG signal to be processed by the time domain feature extraction layer of the initial auditory attention decoding model to obtain the corresponding target time domain feature information:

[0147] Using the first input convolution layer, the first batch of normalization layers, the first activation layer and the first pooling layer of the input convolution block to sequentially process the EEG signal to be processed to obtain first feature information;

[0148] Using the second batch normalization layer, the second activation layer and the first convolution layer of the basic convolution block to perform data processing on the first feature information to obtain second feature information;

[0149] Using the third batch normalization layer, the third activation layer, the second convolution layer, and the second pooling layer of the output convolution block to perform data processing on the second feature information to obtain third feature information;

[0150] The third feature information corresponding to each of the convolution branches is obtained, and all the third feature information is concatenated and flattened using the feature fusion network to obtain the corresponding target time domain feature information.

[0151] In some embodiments, the frequency domain feature extraction layer includes a fast Fourier transform, a long short-term memory network, a fully connected layer, and an activation function layer. The processor 301 performs the following steps in the process of extracting the frequency domain features of the EEG signal to be processed by the frequency domain feature extraction layer of the initial auditory attention decoding model to obtain the corresponding target frequency domain feature information:

[0152] Using the fast Fourier transform to convert the to-be-processed electroencephalogram signal from the time domain to the frequency domain to obtain first frequency domain feature information;

[0153] Using the long short-term memory network to perform feature embedding on the first frequency domain feature information to obtain second frequency domain feature information;

[0154] Using the fully connected layer to perform feature mapping on the second frequency domain feature information to obtain third frequency domain feature information;

[0155] The activation function layer is used to activate the third frequency domain feature information to obtain the target frequency domain feature information.

[0156] In some embodiments, the audio feature extraction layer includes at least a first audio convolution block, a second audio convolution block, and a third audio convolution block, the first audio convolution block includes a second input convolution layer, a fourth batch normalization layer, and a fourth activation layer, the second audio convolution block includes a third input convolution layer, a fifth batch normalization layer, and a sixth activation layer, and the third audio convolution block includes a fourth input convolution layer, a sixth batch normalization layer, and a seventh activation layer; the processor 301 performs, in the process of extracting audio features of the speech signal to be processed by the audio feature extraction layer of the initial auditory attention decoding model to obtain the corresponding target audio feature information, the following:

[0157] Extracting audio features from the speech signal to be processed according to the second input convolution layer, the fourth batch normalization layer, and the fourth activation layer of the first audio convolution block to obtain a first audio feature;

[0158] Performing audio feature extraction on the first audio feature according to the third input convolution layer, the fifth batch normalization layer, and the sixth activation layer of the second audio convolution block to obtain a second audio feature;

[0159] The target audio feature information is obtained by performing audio feature extraction on the second audio feature according to the fourth input convolution layer, the sixth batch normalization layer, and the seventh activation layer of the third audio convolution block.

[0160] In some implementations, the feature fusion mapping layer includes a feature level network and a classification network; the processor 301 performs, in the process of performing model training using the feature fusion mapping layer of the initial auditory attention decoding model according to the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label to obtain the target auditory attention decoding model:

[0161] Using the feature level network, adaptively weighted feature fusion is performed on the target time domain feature information, the target frequency domain feature information and the target audio feature information to obtain splicing feature information;

[0162] Using the classification network to classify the sound source of the spliced ​​feature information to obtain an initial classification result;

[0163] Perform error calculation based on the initial classification result and the sound source label to obtain a training error;

[0164] Model parameters of the initial auditory attention decoding model are trained according to the training error to obtain the target auditory attention decoding model.

[0165] It should be noted that technical personnel in the relevant field can clearly understand that, for the convenience and conciseness of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned embodiment of the auditory attention decoding method based on multiple sound source scenarios, and will not be repeated here.

[0166] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement any step of the auditory attention decoding method based on a multi-sound source scenario provided in the description of the embodiment of the present invention.

[0167] The storage medium may be an internal storage unit of the terminal device described in the foregoing embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.

[0168] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0169] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0170] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An auditory attention decoding method based on a multi-sound source scene, characterized in that: The method comprises: Obtaining target object data corresponding to multiple initial users in a multi-sound source scenario, the target object data including an initial EEG signal to which the initial user responds under a first sound stimulus, a multi-sound source speech signal corresponding to the first sound stimulus, and a sound source label corresponding to the multi-sound source speech signal, the sound source label being used to characterize position information of a speaking object that the initial user needs to pay attention to; Preprocessing the initial EEG signal and the multi-source speech signal respectively to obtain an EEG signal to be processed corresponding to the initial EEG signal and a speech signal to be processed corresponding to the multi-source speech signal; Performing model training on an initial auditory attention decoding model according to the to-be-processed electroencephalogram signal, the to-be-processed speech signal, and the sound source label to obtain a target auditory attention decoding model; Obtaining user data corresponding to a target user in a multi-sound source scenario, the user data including a target electroencephalogram signal of the target user in response to a second sound stimulus and a target speech signal corresponding to the second sound stimulus; Using the target auditory attention decoding model to perform auditory attention decoding on the target EEG signal and the target speech signal to obtain an attention target corresponding to the target user; The method of training the initial auditory attention decoding model according to the electroencephalogram signal to be processed, the speech signal to be processed and the sound source label to obtain the target auditory attention decoding model includes: Using the time domain feature extraction layer of the initial auditory attention decoding model to extract the time domain features of the EEG signal to be processed, to obtain the corresponding target time domain feature information; Using the frequency domain feature extraction layer of the initial auditory attention decoding model to extract frequency domain features of the EEG signal to be processed, to obtain corresponding target frequency domain feature information; Using the audio feature extraction layer of the initial auditory attention decoding model to extract audio features from the speech signal to be processed to obtain corresponding target audio feature information; The feature fusion mapping layer of the initial auditory attention decoding model is used to perform model training according to the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label to obtain the target auditory attention decoding model.

2. The method according to claim 1, characterized in that The preprocessing of the initial EEG signal and the multi-source speech signal respectively to obtain the EEG signal to be processed corresponding to the initial EEG signal and the speech signal to be processed corresponding to the multi-source speech signal comprises: Filtering the initial EEG signal using a filter to obtain a first EEG signal; Performing notch filtering on the first EEG signal to obtain a second EEG signal; Downsampling the second EEG signal to obtain a third EEG signal, and performing mean normalization processing on the third EEG signal to obtain the EEG signal to be processed; The Mel-spectrogram feature extraction operation is performed on the multi-source speech signals to obtain the corresponding speech signals to be processed.

3. The method according to claim 1, characterized in that The time domain feature extraction layer includes a plurality of convolution branches and a feature fusion network, each of the convolution branches includes at least an input convolution block, a basic convolution block and an output convolution block, and the time domain feature extraction layer using the initial auditory attention decoding model extracts the time domain features of the EEG signal to be processed to obtain the corresponding target time domain feature information, including: Using the first input convolution layer, the first batch of normalization layers, the first activation layer and the first pooling layer of the input convolution block to sequentially process the EEG signal to be processed to obtain first feature information; Using the second batch normalization layer, the second activation layer and the first convolution layer of the basic convolution block to perform data processing on the first feature information to obtain second feature information; Using the third batch normalization layer, the third activation layer, the second convolution layer, and the second pooling layer of the output convolution block to perform data processing on the second feature information to obtain third feature information; The third feature information corresponding to each of the convolution branches is obtained, and all the third feature information is concatenated and flattened using the feature fusion network to obtain the corresponding target time domain feature information.

4. The method according to claim 1, characterized in that: The frequency domain feature extraction layer includes a fast Fourier transform, a long short-term memory network, a fully connected layer, and an activation function layer. The frequency domain feature extraction layer using the initial auditory attention decoding model performs frequency domain feature extraction on the EEG signal to be processed to obtain corresponding target frequency domain feature information, including: Using the fast Fourier transform to convert the to-be-processed electroencephalogram signal from the time domain to the frequency domain to obtain first frequency domain feature information; Using the long short-term memory network to perform feature embedding on the first frequency domain feature information to obtain second frequency domain feature information; Using the fully connected layer to perform feature mapping on the second frequency domain feature information to obtain third frequency domain feature information; The activation function layer is used to activate the third frequency domain feature information to obtain the target frequency domain feature information.

5. The method according to claim 1, characterized in that The audio feature extraction layer at least includes a first audio convolution block, a second audio convolution block and a third audio convolution block, the first audio convolution block includes a second input convolution layer, a fourth batch normalization layer, and a fourth activation layer, the second audio convolution block includes a third input convolution layer, a fifth batch normalization layer, and a sixth activation layer, and the third audio convolution block includes a fourth input convolution layer, a sixth batch normalization layer, and a seventh activation layer; the audio feature extraction layer of the initial auditory attention decoding model is used to extract audio features from the speech signal to be processed to obtain corresponding target audio feature information, including: Extracting audio features from the speech signal to be processed according to the second input convolution layer, the fourth batch normalization layer, and the fourth activation layer of the first audio convolution block to obtain a first audio feature; Performing audio feature extraction on the first audio feature according to the third input convolution layer, the fifth batch normalization layer, and the sixth activation layer of the second audio convolution block to obtain a second audio feature; The target audio feature information is obtained by performing audio feature extraction on the second audio feature according to the fourth input convolution layer, the sixth batch normalization layer, and the seventh activation layer of the third audio convolution block.

6. The method according to claim 1, characterized in that The feature fusion mapping layer includes a feature level network and a classification network; the feature fusion mapping layer using the initial auditory attention decoding model performs model training according to the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label to obtain a target auditory attention decoding model, including: Using the feature level network, adaptively weighted feature fusion is performed on the target time domain feature information, the target frequency domain feature information and the target audio feature information to obtain splicing feature information; Using the classification network to classify the sound source of the spliced ​​feature information to obtain an initial classification result; Perform error calculation based on the initial classification result and the sound source label to obtain a training error; Model parameters of the initial auditory attention decoding model are trained according to the training error to obtain the target auditory attention decoding model.

7. An auditory attention decoding device based on a multi-sound source scene, characterized in that: include: A data acquisition module, used to obtain target object data corresponding to multiple initial users in a multi-sound source scenario, wherein the target object data includes an initial EEG signal to which the initial user responds under a first sound stimulus, a multi-sound source speech signal corresponding to the first sound stimulus, and a sound source label corresponding to the multi-sound source speech signal, wherein the sound source label is used to characterize the position information of the speaking object that the initial user needs to pay attention to; A data processing module, used for preprocessing the initial EEG signal and the multi-source speech signal to obtain the EEG signal to be processed corresponding to the initial EEG signal and the speech signal to be processed corresponding to the multi-source speech signal; A model training module, used for performing model training on an initial auditory attention decoding model according to the EEG signal to be processed, the speech signal to be processed and the sound source label to obtain a target auditory attention decoding model, wherein the performing model training on the initial auditory attention decoding model according to the EEG signal to be processed, the speech signal to be processed and the sound source label to obtain a target auditory attention decoding model comprises: performing time domain feature extraction on the EEG signal to be processed by a time domain feature extraction layer of the initial auditory attention decoding model to obtain corresponding target time domain feature information; performing frequency domain feature extraction on the EEG signal to be processed by a frequency domain feature extraction layer of the initial auditory attention decoding model to obtain corresponding target frequency domain feature information; performing audio feature extraction on the speech signal to be processed by an audio feature extraction layer of the initial auditory attention decoding model to obtain corresponding target audio feature information; performing model training on the target time domain feature information, the target frequency domain feature information, the target audio feature information and the sound source label by a feature fusion mapping layer of the initial auditory attention decoding model to obtain the target auditory attention decoding model; A data acquisition module, used to obtain user data corresponding to a target user in a multi-sound source scenario, wherein the user data includes a target EEG signal of the target user in response to a second sound stimulus and a target speech signal corresponding to the second sound stimulus; The target recognition module is used to use the target auditory attention decoding model to perform auditory attention decoding on the target EEG signal and the target speech signal to obtain the attention target corresponding to the target user.

8. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program and implement the auditory attention decoding method based on a multi-sound source scenario as described in any one of claims 1 to 6 when executing the computer program.

9. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the auditory attention decoding method based on a multi-sound source scenario as described in any one of claims 1 to 6.

Images