Training methods, devices, equipment, and storage media for EEG-based auditory speech extraction models

Abstract

This application relates to a method, apparatus, computer device, computer-readable storage medium, and computer program product for training an EEG-based auditory-speech extraction model. The method includes: acquiring EEG data and mixed speech data; extracting features from the mixed speech data to obtain audio coding features, and extracting features from the EEG data to obtain EEG temporal features and EEG spatial features; wherein the EEG temporal features characterize the changing trend of EEG signal data over time, and the EEG spatial features characterize the changing characteristics of EEG signal data between different EEG channels, and then fusing them to obtain speech embedding features; separating the first predicted audio data of the target sound source from the mixed speech data; calculating the master model loss based on the first predicted audio data and the original audio data of the target sound source, thereby obtaining a trained EEG-based auditory-speech extraction model. This method can accurately identify the source of speech that the listener is interested in.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, storage medium and computer program product for training an EEG auditory speech extraction model. Background Technology

[0002] In real life, humans are usually able to naturally focus their attention on a specific speaker and ignore other distracting sounds in noisy, multi-speaker environments. This ability is known as "selective auditory attention." For example, in a cocktail party setting, even if there are multiple people speaking, most people can still clearly identify the speech of the speaker they are focusing on.

[0003] However, for people with hearing impairments, the reduced ability of their nervous system to process auditory information makes it difficult to distinguish the target speaker from multiple sources of speech interference, greatly limiting their communication efficiency and quality of life. Currently, while hearing aid technology can amplify sound, it cannot accurately identify the source of the speech that the listener is truly focused on. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium, and computer program product for an EEG auditory speech extraction model that can accurately identify the source of speech that the listener is interested in in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for training an EEG auditory speech extraction model, including:

[0006] Acquire electroencephalogram (EEG) data and mixed speech data; wherein the mixed speech data includes multiple raw audio data from different sound sources; the EEG data includes EEG signal data from multiple EEG channels;

[0007] Feature extraction is performed on the mixed speech data to obtain audio coding features, and feature extraction is performed on the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features; wherein, the EEG temporal features are used to characterize the changing trend of the EEG signal data over time, and the EEG spatial features are used to characterize the changing features of the EEG signal data between different EEG channels;

[0008] The audio coding features, the EEG temporal features, and the EEG spatial features are fused to obtain speech embedding features, and the first predicted audio data of the target sound source is separated from the mixed speech data based on the speech embedding features.

[0009] The master model loss is calculated based on the first predicted audio data and the original audio data of the target sound source, and the EEG auditory speech extraction model is trained based on the master model loss until a preset termination condition is met, thus obtaining the trained EEG auditory speech extraction model.

[0010] In one embodiment, the method further includes:

[0011] The speech embedding features are divided into multiple consecutive feature segments; each feature segment corresponds to an original audio segment in the original audio data of the target sound source;

[0012] For each feature segment, context fusion features are obtained based on the feature segment and the previous feature segment.

[0013] The second predicted audio data for separating the target sound source from the mixed speech data based on the context fusion features;

[0014] Calculate the causal layer loss based on the second predicted audio data and the original audio segment corresponding to the feature segment;

[0015] The step of training the EEG auditory-speech extraction model based on the master model loss until a preset termination condition is met, to obtain the trained EEG auditory-speech extraction model, includes:

[0016] The EEG auditory speech extraction model is trained based on the causal layer loss and the master model loss until a preset termination condition is met, resulting in a well-trained EEG auditory speech extraction model.

[0017] In one embodiment, the second predicted audio data for separating the target sound source from the mixed speech data based on the context fusion features includes:

[0018] Obtain the initial sampling probability and attenuation coefficient;

[0019] The sampling probability of the feature segment is calculated based on the initial sampling probability and the attenuation coefficient;

[0020] Based on the sampling probability, the context fusion feature or the feature fragment is selected as the target embedding feature;

[0021] The second predicted audio data, which separates the target sound source from the mixed speech data based on the target embedding features, is then used to separate the target sound source.

[0022] In one embodiment, the step of extracting features from the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features includes:

[0023] The electroencephalogram (EEG) data is time-location encoded to obtain a first EEG data sequence;

[0024] For the first EEG data sequence, convolution calculation is performed on the time axis to obtain EEG temporal features;

[0025] Channel location encoding is performed on the electroencephalogram (EEG) data to obtain a second EEG data sequence;

[0026] For the second EEG data sequence, convolution calculations are performed between different EEG channels to obtain EEG spatial features.

[0027] In one embodiment, fusing the audio coding features, the EEG temporal features, and the EEG spatial features to obtain speech embedding features includes:

[0028] Self-attention alignment is performed on the time axis of the EEG temporal features and the EEG spatial features to obtain EEG signal features;

[0029] The EEG signal features and the audio coding features are fused to obtain speech embedding features.

[0030] In one embodiment, fusing the EEG signal features and the audio coding features to obtain speech embedding features includes:

[0031] The number of speech frames is determined based on the audio coding features;

[0032] Linear interpolation calculations are performed on the EEG signal features based on the number of speech frames to update the EEG signal features;

[0033] The updated EEG signal features are aligned and fused with the audio coding features to obtain speech embedding features.

[0034] Secondly, this application also provides a training device for an EEG auditory speech extraction model, comprising:

[0035] The data acquisition module is used to acquire electroencephalogram (EEG) data and mixed speech data; wherein, the mixed speech data includes multiple raw audio data from different sound sources; and the EEG data includes EEG signal data from multiple EEG channels.

[0036] The feature extraction module is used to extract features from the mixed speech data to obtain audio coding features, and to extract features from the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features; wherein, the EEG temporal features are used to characterize the changing trend of the EEG signal data over time, and the EEG spatial features are used to characterize the changing features of the EEG signal data between different EEG channels;

[0037] The feature fusion module is used to fuse the audio coding features, the EEG temporal features and the EEG spatial features to obtain speech embedding features, and to separate the first predicted audio data of the target sound source from the mixed speech data based on the speech embedding features;

[0038] The model training module is used to calculate the master model loss based on the first predicted audio data and the original audio data of the target sound source, and to train the EEG auditory speech extraction model based on the master model loss until a preset termination condition is met, thereby obtaining the trained EEG auditory speech extraction model.

[0039] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0040] Acquire electroencephalogram (EEG) data and mixed speech data; wherein the mixed speech data includes multiple raw audio data from different sound sources; the EEG data includes EEG signal data from multiple EEG channels;

[0041] Feature extraction is performed on the mixed speech data to obtain audio coding features, and feature extraction is performed on the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features; wherein, the EEG temporal features are used to characterize the changing trend of the EEG signal data over time, and the EEG spatial features are used to characterize the changing features of the EEG signal data between different EEG channels;

[0042] The audio coding features, the EEG temporal features, and the EEG spatial features are fused to obtain speech embedding features, and the first predicted audio data of the target sound source is separated from the mixed speech data based on the speech embedding features.

[0043] The master model loss is calculated based on the first predicted audio data and the original audio data of the target sound source, and the EEG auditory speech extraction model is trained based on the master model loss until a preset termination condition is met, thus obtaining the trained EEG auditory speech extraction model.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0045] Acquire electroencephalogram (EEG) data and mixed speech data; wherein the mixed speech data includes multiple raw audio data from different sound sources; the EEG data includes EEG signal data from multiple EEG channels;

[0046] Feature extraction is performed on the mixed speech data to obtain audio coding features, and feature extraction is performed on the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features; wherein, the EEG temporal features are used to characterize the changing trend of the EEG signal data over time, and the EEG spatial features are used to characterize the changing features of the EEG signal data between different EEG channels;

[0047] The audio coding features, the EEG temporal features, and the EEG spatial features are fused to obtain speech embedding features, and the first predicted audio data of the target sound source is separated from the mixed speech data based on the speech embedding features.

[0048] The master model loss is calculated based on the first predicted audio data and the original audio data of the target sound source, and the EEG auditory speech extraction model is trained based on the master model loss until a preset termination condition is met, thus obtaining the trained EEG auditory speech extraction model.

[0049] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0050] Acquire electroencephalogram (EEG) data and mixed speech data; wherein the mixed speech data includes multiple raw audio data from different sound sources; the EEG data includes EEG signal data from multiple EEG channels;

[0051] Feature extraction is performed on the mixed speech data to obtain audio coding features, and feature extraction is performed on the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features; wherein, the EEG temporal features are used to characterize the changing trend of the EEG signal data over time, and the EEG spatial features are used to characterize the changing features of the EEG signal data between different EEG channels;

[0052] The audio coding features, the EEG temporal features, and the EEG spatial features are fused to obtain speech embedding features, and the first predicted audio data of the target sound source is separated from the mixed speech data based on the speech embedding features.

[0053] The master model loss is calculated based on the first predicted audio data and the original audio data of the target sound source, and the EEG auditory speech extraction model is trained based on the master model loss until a preset termination condition is met, thus obtaining the trained EEG auditory speech extraction model.

[0054] The aforementioned training method, device, computer equipment, storage medium, and computer program product for EEG-auditory-speech extraction models acquire EEG data and mixed speech data. The mixed speech data includes multiple raw audio data from different sound sources; the EEG data includes EEG signal data from multiple EEG channels. Feature extraction is performed on the mixed speech data to obtain audio coding features, and feature extraction is performed on the EEG data to obtain EEG temporal features and EEG spatial features. Feature extraction and semantic modeling are then performed on these two types of data respectively, enabling the system to effectively perceive and decode the listener's auditory attention direction in a multi-source speech environment. This yields audio coding features representing the audio content, time-varying EEG temporal features, and EEG spatial features reflecting the coordinated activity patterns of different brain regions, thereby achieving a fine-grained description of the listener's dynamic auditory attention state. Based on this, the audio coding features are jointly modeled with the EEG temporal and EEG spatial features to further generate speech embedding features. This embedded feature, serving as an intermediate representation of the "speech the listener is paying attention to," possesses semantic information that simultaneously includes speech content and attentional cues. Guided by this speech embedding feature, the mixed speech data is separated, accurately extracting the target sound source from the mixed sources to obtain the first predicted audio data. Next, by comparing the first predicted audio data with the original audio data corresponding to the target sound source and calculating the main model loss function, the EEG auditory speech extraction model can be effectively trained. Through continuous optimization of the model, and completing training when the loss function meets a preset termination condition, a well-trained EEG auditory speech extraction model capable of jointly determining the auditory attention object based on EEG and speech data is obtained. This model exhibits good generalization ability and stability, automatically inferring the source of the listener's attention based on their EEG responses. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a diagram illustrating the application environment of a brainwave-auditory speech extraction model training method in one embodiment.

[0057] Figure 2 This is a flowchart illustrating the training method for an EEG auditory speech extraction model in one embodiment;

[0058] Figure 3 Here is an architecture diagram of the MossFormer block in one embodiment;

[0059] Figure 4 This is a flowchart illustrating the training method for the EEG auditory speech extraction model in another embodiment;

[0060] Figure 5 This is an overall architecture diagram of a Mindflow model in one embodiment;

[0061] Figure 6 This is an architecture diagram of the context fusion module in one embodiment;

[0062] Figure 7 This is a structural block diagram of a training device for an EEG auditory speech extraction model in one embodiment;

[0063] Figure 8 This is an internal structural diagram of a computer device in one embodiment;

[0064] Figure 9 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] The EEG auditory speech extraction model training method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can acquire EEG data and mixed speech data, and send them to server 104 for processing. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated on server 104 or located on the cloud or other network servers. The data storage system can store mixed speech data, EEG data, and data used in steps such as feature extraction, feature fusion, and audio prediction. Terminal 102 can acquire EEG data, which can be an EEG data acquisition device, head-mounted device, etc. Terminal 102 can also be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices to acquire mixed speech signals and convert them into mixed speech data to send to server 104. IoT devices can be smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices; portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0067] In one exemplary embodiment, such as Figure 2 As shown, a method for training an EEG-based auditory speech extraction model is provided, which is then applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S202 to S208. Wherein:

[0068] Step S202: Obtain EEG data and mixed speech data.

[0069] Mixed speech data includes multiple raw audio data from different sound sources; electroencephalogram (EEG) data includes EEG signal data from multiple brainwave channels. Mixed speech data refers to the result of superimposing and synthesizing raw audio data from two or more different sound sources.

[0070] For example, in a real-world application scenario, such as simulating a "cocktail party" environment, suppose speaker A says "Hello," and speaker B simultaneously says "What time is it?" The mixed speech might be a linear overlap of these two audio segments on the time axis. Server 104 can acquire the mixed speech data collected by terminal 102 and treat the mixed speech as the original input signal x(t), serving as an approximate reproduction of the actual auditory scene and a direct stimulus source that triggers selective auditory attention in the brain. Let... For noisy multi-speaker mixed signals in the time domain:

[0071]

[0072] in This represents the voice signal of the target speaker selected by the user. This represents the speech of M interfering speakers. This indicates environmental noise.

[0073] Electroencephalogram (EEG) data refers to EEG signals recorded using high-density EEG acquisition equipment (such as a 64-channel Bio Semi ActiveTwo system). These signals reflect the neural responses of different brain regions to auditory stimuli during an auditory task. Specifically, EEG data is a time-varying multi-channel voltage signal matrix, where each channel represents an electrode acquisition point (e.g., Fz, Pz, Cz, etc.). These electrodes are distributed across different areas of the scalp surface to reflect the activity of the cerebral cortex at different frequency bands. For example, when the target speaker is located to the listener's right, the relevant channels in the subject's left frontal lobe may produce a more significant response waveform. Exemplarily, given a real-time EEG signal E(t) encoding the user's auditory attention, server 104 can extract:

[0074]

[0075] in Indicates having parameters The neural structure.

[0076] For example, server 104 can control the EEG acquisition device to start recording operations simultaneously with the subject completing the auditory attention task, ensuring complete synchronization between the EEG sampling time reference and speech playback. The raw EEG signal E_raw acquired in this process can be accompanied by a trigger signal (i.e., a trigger marker) to mark the time point at which the mixed speech begins to play. Furthermore, regarding the sampling frequency, the EEG acquisition can be set to 8196 Hz to capture high-precision neural potential changes, while the speech data sampling rate can be 44.1 kHz to preserve the sound quality and detail of the speech.

[0077] Subsequently, server 104 can save the raw EEG signal E_raw collected and sent by terminal 102 as a 64-channel voltage sequence, with each channel corresponding to the activity recording of a brain region. Simultaneously, server 104 can also record the complete mixed signal of the audio playback. The sound sample value at each moment can simultaneously contain the speech information of multiple speakers, as well as various environmental noises. In some embodiments, the server 104 can also access clean speech sources (i.e., the original speech tracks of speakers A and B respectively) as reference data for subsequent training and verification. Through this step, the server 104 can obtain raw input data representing auditory input (mixed speech) and auditory attention response (EEG) by simultaneously acquiring mixed speech data and EEG data.

[0078] Step S204: Extract features from the mixed speech data to obtain audio coding features, and extract features from the EEG data to obtain EEG temporal features and EEG spatial features.

[0079] Among them, EEG temporal features are used to characterize the changing trend of EEG signal data over time, while EEG spatial features are used to characterize the changing characteristics of EEG signal data between different EEG channels.

[0080] For example, for mixed speech data, server 104 can employ a one-dimensional convolutional encoder (Conv1DEncoder) and use the ReLU activation function to extract audio coding features. This encoder can perform convolution operations on the audio waveform using a sliding window and extract speech feature information from different frequencies and temporal patterns through multiple channels. For the input sequence... (Where B is the batch size and Ts is the input length), the encoder application size is... The core, with a step size of Generate encoded output Its definition is:

[0081]

[0082] Where N represents the number of filters. , indicating a reduced time dimension.

[0083] For example, server 104 can slice the audio into overlapping frame sequences, with each frame consisting of 20 sampling points and a stride of 10. It then performs feature transformation using convolutional kernels to compress the original signal while retaining the distinctive sound information. Each frame of audio features output by the convolutional encoder not only contains temporal information from the original waveform but also possesses a certain degree of perceptual semantic capability.

[0084] Furthermore, regarding the processing of EEG data, server 104 can employ a multi-stage structured encoding method to obtain EEG temporal and spatial features. For example, server 104 can perform temporal location encoding on the EEG data to obtain a first EEG data sequence; perform convolution calculations on the time axis for the first EEG data sequence to obtain EEG temporal features; perform channel location encoding on the EEG data to obtain a second EEG data sequence; and perform convolution calculations between different EEG channels for the second EEG data sequence to obtain EEG spatial features.

[0085] For example, to preserve the dynamic characteristics of EEG signals over time, server 104 can enhance the EEG input with time labels by introducing a temporal position encoding mechanism (i.e., position vector in the time dimension), thus clarifying the position of each frame of the EEG signal on the time axis. Subsequently, server 104 can input the encoded EEG signal into a self-attention module to extract long-range dependencies in the time dimension. Through this operation, server 104 can identify the inherent patterns existing at different time points from the original multi-channel EEG waveform, such as whether the neural response generated by a certain speech stimulus will affect the neural activity in several subsequent frames.

[0086] After initial temporal modeling, server 104 can further apply temporal convolution to the temporal signals of each EEG channel. The convolution kernel slides only along the temporal direction to enhance the identification of fluctuation trends within local time windows. For example, server 104 may slide the convolution kernel across 25 consecutive time points to identify whether there is a rapid rise or fall in voltage in that channel within a short period of time; these changes are often associated with speech stimuli. After pooling, normalization, and activation processing, the convolution output yields clear EEG temporal features, which are used to characterize the neural response trajectory of the brain during auditory processing over time.

[0087] For example, after the temporal feature extraction is completed, the server 104 can further apply spatial convolution to capture the synergistic relationships between different EEG channels. Cross-channel convolution processing can be performed between electrode channels; for instance, a convolution kernel covering all 64 channels can be used to identify whether synchronous activation or complementary inhibition exists between different brain regions at the same time. This spatial feature is the basis for the brain's neural processing of multi-source speech. For example, when a listener receives the primary sound stimulus in their right ear, the EEG signal in their left temporal lobe may be enhanced, while the right side may remain relatively stable. The server 104 defines this part of the output as EEG spatial features, used to depict the organized and synergistic responses between brain regions.

[0088] For example, for electroencephalogram (EEG) data (Where B is the batch size and C is the number of electrode channels,) (For the input length), server 104 can apply time-location encoding. The self-attention module (SA) encodes the time dependence in EEG signals. The formula is as follows:

[0089]

[0090] Furthermore, server 104 can extract temporal patterns using temporal and spatial convolutions. and spatial distribution characteristics As shown below:

[0091]

[0092]

[0093] TemporalConv2d() uses a size of (1, 25). Each filter performs a 2D convolution along the time axis. SpatialConv2d() uses a 2D convolution filter of the same size as the EEG channel to perform convolution along the electrode channel dimension. Norm refers to BatchNorm2D, and ELU refers to the activation function.

[0094] Through the above steps, server 104 obtains audio coding features, EEG temporal features (representing the dynamic changes of EEG over time), and EEG spatial features (representing spatial difference patterns between different channels). These features not only preserve the important semantic structure of the original signal, but also provide shape-uniform and semantically complementary input data for subsequent multimodal fusion and speech embedding modeling, thereby inferring whether the listener is currently focusing on a specific sound source in the speech.

[0095] Step S206: The audio coding features, EEG temporal features and EEG spatial features are fused to obtain speech embedding features, and the first predicted audio data of the target sound source is separated from the mixed speech data based on the speech embedding features.

[0096] For example, server 104 can perform shape alignment processing on audio-coded features, EEG temporal features, and EEG spatial features. In the aforementioned steps, the EEG temporal features and EEG spatial features, after being extracted through convolution and attention mechanisms, have been aligned to a unified number of frames and channels, consistent with the audio-coded features. Server 104 can concatenate the EEG temporal and spatial features along the channel dimension and map them to the same feature dimension as the audio-coded features through 1D convolution, ultimately forming a unified EEG fusion feature vector. To enable the network to distinguish which channel or brain region these EEG features originate from, server 104 can also introduce channel location encoding. This involves attaching a vector label to each EEG channel to indicate its physical spatial location (such as the left prefrontal lobe, right occipital lobe, etc.).

[0097] For example, server 104 can perform self-attention alignment on the time axis of EEG temporal features and EEG spatial features to obtain EEG signal features; and fuse the EEG signal features and audio coding features to obtain speech embedding features. Further, server 104 can determine the number of speech frames based on the audio coding features; perform linear interpolation calculation on the EEG signal features based on the number of speech frames to update the EEG signal features; and align and fuse the updated EEG signal features with the audio coding features to obtain speech embedding features.

[0098] For example, server 104 can perform channel concatenation on the audio encoded features and the fused EEG features, merging them along the feature dimension into a new composite input feature tensor. This tensor represents, at each frame time, both the information of the speech itself and the listener's response to the speech at that frame time. Server 104 can then perform channel compression on this concatenated tensor using 1D convolution, compressing it from the original 2N channels back to N channels to prevent excessive computation in subsequent networks, and retaining high-order feature patterns through non-linear activation operations. For example, server 104 can apply linear interpolation to strictly align the aforementioned EEG features with the audio features, thereby achieving time-aligned fusion and generating cue features. Its definition is:

[0099]

[0100] Furthermore, the fused multimodal features can be input into the speaker extraction module deployed on server 104. This module can employ a MossFormer structure that integrates local convolution and global attention mechanisms. MossFormer combines gated single-head attention with convolutional-enhanced joint self-attention, simultaneously modeling local and global dependencies in the speech signal. This significantly reduces computational complexity while balancing performance and computational cost. Therefore, the MossFormer network has the ability to process semantic features at different scales in parallel. Server 104 can use multiple stacked blocks in this module, each including substructures such as gated convolution, local attention, and global attention, used to model the local articulation features and overall vocalization patterns of the speaker's speech, respectively. For example, server 104 can obtain a mask representing the target speech for attention in this network, i.e., a weight matrix that floats in the temporal and channel dimensions. The higher the value of this mask, the more confident that the server 104 believes that the information in that channel of the frame comes from the speaker the listener is interested in. For example, server 104 may employ a speaker extraction module to estimate a mask M that only allows the speech of the speaker of interest to pass through. And masked speech embedding The calculation formula is:

[0101]

[0102] Next, server 104 can connect the mixed audio features along the channel dimension. and EEG characteristics These features are then combined using one-dimensional convolution:

[0103]

[0104] The fused features are then fed into the speech separation model to extract the mask M.

[0105] Subsequently, server 104 can multiply the mask element-wise with the initially extracted audio coding features to filter out unnoticed speech components and retain only the feature regions most likely to belong to the target sound source, forming the embedded feature vector of the target speech.

[0106] Furthermore, server 104 can feed the embedding vector into a decoder module, which is a deconvolutional network symmetrical to the encoder, to restore the embedded features into continuous speech waveform data. Finally, the first predicted audio data output by server 104 is the target speaker audio separated from the mixed speech based on the EEG attention state. For example, in a mixed speech containing male voice A and female voice B, if the listener's EEG response indicates that their attention is focused on female voice B, then the first predicted audio output by server 104 after the above processing is the restoration of the speech content of female voice B.

[0107] For example, the architecture diagram of the MossFormer block is as follows: Figure 3 As shown. The MossFormer block contains four convolutional modules (Conv Blocks), scaling and offset operations, a joint local and global single-head self-attention (SHSA) mechanism, and three gating operations.

[0108] The Convolutional Module (ConvM) is used to normalize the input sequence, then project it onto a linear layer, and finally use the SiLU activation function. Furthermore, a one-dimensional depthwise convolution is applied to each feature for processing. The addition of skip connections and dropout algorithms helps to train and normalize the network.

[0109] Among them, the attention gating mechanism integrates the attention mechanism into the triple gating process to enhance the model's capabilities. Given an input sequence The convolutional module processes the sequence to generate two outputs. and As shown below:

[0110]

[0111] Using attention matrix Calculate the output sequence of this block.

[0112] like:

[0113]

[0114]

[0115]

[0116] in, It is an element-wise activation function. , .

[0117] For long sequences (i.e., large S), directly computing the gated attention mechanism is computationally expensive. Server 104 can efficiently compute the gating mechanism by combining joint local and global attention mechanisms. and Server 104 allows input of sequences. Projected onto the shared representation via a convolutional neural network (ConvM):

[0118]

[0119] Next, server 104 can apply low-cost per-dimensional scalars, offsets, and RoPE (Rotary Position Embedding) to Z to generate a query. , s and keys , This is used for local and global attention computation. For the global attention mechanism, server 104 can employ a linearized, low-cost form to capture long-range interactions between V and U:

[0120]

[0121] in, , is the scaling factor. For the local secondary attention mechanism, V, U, Q, and K are divided into H non-overlapping segments of size P. The secondary attention mechanism is computed independently within each block:

[0122]

[0123] in, , which is the scaling factor. Then, the outputs of all blocks are concatenated along the time dimension to reconstruct the complete sequence. Finally, the outputs of the local and global attention mechanisms are summed to form the final joint attention mechanism:

[0124]

[0125] Furthermore, the speech decoder can be a one-dimensional transposed convolutional layer, using the same kernel size and stride as the encoder, to separate the feature sequences, which are ultimately decoded into waveforms by the decoder.

[0126]

[0127] The above module describes the single-step reasoning process based on EEG and audio target speaker extraction, and the model is represented as follows: :

[0128]

[0129] Through the above steps, server 104 can not only recognize the acoustic content of speech, but also use EEG responses to determine the user's auditory target and achieve accurate separation of the sound source, thus extracting the target of interest from both auditory and brain signals.

[0130] Step S208: Calculate the master model loss based on the first predicted audio data and the original audio data of the target sound source, and train the EEG auditory speech extraction model based on the master model loss until the preset termination condition is met, and obtain the trained EEG auditory speech extraction model.

[0131] For example, server 104 can use the first predicted audio data output from the previous stage as the current output of the model and compare it with the original audio data of the target sound source within the same time period. The original audio data of the target sound source is clean speech known in the training dataset, i.e., the real speech trajectory from a single speaker without any mixing processing. During training, server 104 has pre-labeled each data point with the identity of the speaker the subject is truly interested in, so it can directly retrieve the corresponding target audio from the annotation set as a reference. Server 104 can use the scale-invariant signal-to-noise ratio (SI-SDR) from the signal distortion rate metric as the basis for the loss function to measure the similarity between the model's output speech and the real speech. This metric does not concern itself with changes in overall loudness but focuses on the consistency of speech shape, making it particularly suitable for speech separation and enhancement tasks. Server 104 can input the first predicted audio data and the target audio data into the SI-SDR calculation module to obtain a numerical score; then, the inverse of this score is used as the main model loss (because the optimization objective is to maximize SI-SDR, i.e., minimize the negative value).

[0132] For example, the definition of SI-SDR can be:

[0133]

[0134] in, and These represent the extracted speech signal and the clean speech signal, respectively. Main model loss function. It can be:

[0135]

[0136] Furthermore, after obtaining the loss of the main model, server 104 can use this loss value as an optimization target to perform backpropagation in neural network training. This involves using an automatic differentiation mechanism to propagate the error from the output layer back to each module of the model, including the audio encoder, EEG encoder, multimodal fusion network, and speaker mask predictor. Each backpropagation process updates the trainable parameters in these modules, such as convolutional kernel weights, attention matrices, and gating coefficients, making the model more closely approximate the real target audio in the next prediction round. The training process is iterative; server 104 can treat each parameter update as a training epoch and record the current model's performance metrics on the validation set after each epoch, such as the average SI-SDR, speech intelligibility metric, and speech quality metric. Server 104 can determine whether the model has reached convergence based on preset termination conditions. These termination conditions can be reaching a specified number of training epochs (e.g., 100 epochs), a loss decrease of less than a threshold (e.g., 0.001) for several consecutive epochs, or a cessation of improvement in validation set metrics.

[0137] Once the training process meets the termination condition, server 104 can save the model in its current parameter state, thus obtaining the trained EEG auditory speech extraction model. This model has the ability to accurately extract the target sound source speech from the input mixed speech and EEG data, and can be used in subsequent inference deployment scenarios. For example, when a user wearing an EEG acquisition device enters a multi-person conversation environment, as long as the real-time mixed speech and its EEG signal are input into the model, server 104 can output the speech stream of the speaker that the user is currently paying attention to in real time, thereby empowering application devices such as smart hearing aids and brain-computer interface headphones, significantly improving their interaction capabilities in complex environments.

[0138] In the aforementioned EEG-auditory-speech extraction model training method, EEG data and mixed speech data are acquired. The mixed speech data includes multiple raw audio data from different sound sources; the EEG data includes EEG signal data from multiple EEG channels. Feature extraction is performed on the mixed speech data to obtain audio coding features, and feature extraction is performed on the EEG data to obtain EEG temporal features and EEG spatial features. Feature extraction and semantic modeling are performed on these two types of data respectively, enabling the system to effectively perceive and decode the listener's auditory attention direction in a multi-source speech environment. This yields audio coding features representing the audio content, EEG temporal features that change over time, and EEG spatial features reflecting the coordinated activity patterns of different brain regions, thereby achieving a fine-grained description of the listener's dynamic auditory attention state. Based on this, the audio coding features are jointly modeled with the EEG temporal and EEG spatial features to further generate speech embedding features. This embedded feature, serving as an intermediate representation of the "speech the listener is paying attention to," possesses semantic information that simultaneously includes speech content and attentional cues. Guided by this speech embedding feature, the mixed speech data is separated, accurately extracting the target sound source from the mixed sources to obtain the first predicted audio data. Next, by comparing the first predicted audio data with the original audio data corresponding to the target sound source and calculating the main model loss function, the EEG auditory speech extraction model can be effectively trained. Through continuous optimization of the model, and completing training when the loss function meets a preset termination condition, a well-trained EEG auditory speech extraction model capable of jointly determining the auditory attention object based on EEG and speech data is obtained. This model exhibits good generalization ability and stability, automatically inferring the source of the listener's attention based on their EEG responses.

[0139] In one exemplary embodiment, such as Figure 4 As shown, the above method may further include steps S302 to S310. Wherein:

[0140] Step S302: Divide the speech embedding features into multiple consecutive feature segments.

[0141] Each feature segment corresponds to an original audio segment in the original audio data of the target sound source.

[0142] For example, server 104 can introduce a causal inference chain (CIC) to simultaneously predict multiple time steps, while using hierarchical masks (parallel within each window, serial across windows) to strictly control the information flow. Server 104 can perform temporal partitioning on the aforementioned obtained speech embedding features, dividing the entire speech embedding sequence into several consecutive feature segments at equal intervals. Each segment corresponds to a continuous speech signal in the original audio of the target sound source. For example, if the original target audio is a 9-second mixed dialogue, server 104 can divide it into 9 feature segments, each corresponding to 1 second of content. These segments have a strict temporal sequence, and their arrangement is consistent with the actual playback order of the speech, ensuring that subsequent causal modeling conforms to the natural laws of semantic evolution. Each feature segment maintains a uniform number of time frames and feature channel dimensions, and preserves the embedded semantic expression after the initial speech and EEG fusion.

[0143] Step S304: For each feature segment, obtain the context fusion feature based on the feature segment and the previous feature segment.

[0144] For example, server 104 can process each feature segment sequentially to construct contextual fusion features. For the k-th feature segment, server 104 can not only read the speech embedding of the segment itself, but also obtain the output embedding or processing state of the previous segment (k-1-th segment). Server 104 can combine the embedding features of the current segment and the previous segment through channel concatenation to form a continuous contextual fusion feature. Subsequently, this fusion feature can be fed into a Transformer module (which can be understood as a semantic fusion structure with global modeling capabilities), and the semantic dependency between the two segments is modeled through a self-attention mechanism, thereby outputting a contextual fusion feature with temporal continuity. This fusion feature not only contains the semantic information of the current segment, but also incorporates the judgment criteria from the previous time segment, thus providing a reference memory for target sound source separation.

[0145] By way of example, combining the above embodiments, such as Figure 5 The overall architecture diagram of the proposed Mindflow model is shown, which consists of a main model. and multiple causal reasoning chains The system is composed of several modules. The speaker extraction module uses electroencephalography (EEG) as reference information to extract audio features of the target speaker from multi-speaker speech, which are then decoded into audio by the speech decoder. The context fusion module (CFM) refines the feature extraction at the current time step based on extracted past speech. The speaker extraction module and the speech decoder module share parameters at each layer. CIC can be composed of a single master model. Modules and multiple Composition. With In comparison, each Each module includes an additional context fusion module (CFM), while the other modules share parameters. Each layer of the causal inference chain is determined based on the corresponding input. and Generate masked speech vectors Main model You can directly from Decode the audio For the follow-up The context information is integrated through CFM and then decoded into audio. Context fusion module, such as Figure 6 As shown. The calculation is as follows:

[0146]

[0147]

[0148]

[0149] in The parameters represent the context fusion module, and TRM represents a single-layer Transformer module.

[0150] Step S306: Separate the second predicted audio data of the target sound source from the mixed speech data based on the context fusion features.

[0151] The aforementioned speech decoder can be a one-dimensional transposed convolutional layer, using the same kernel size and stride as the encoder to separate the feature sequences, which are then decoded into waveforms by the decoder.

[0152]

[0153] For example, server 104 may obtain initial sampling probability and attenuation coefficient; calculate sampling probability of feature segment based on initial sampling probability and attenuation coefficient; select context fusion feature or feature segment as target embedding feature based on sampling probability; and separate second predicted audio data of target sound source from mixed speech data based on target embedding feature.

[0154] Furthermore, server 104 can employ adaptive teacher-mandated training, dynamically transitioning from supervised instruction to free prediction. During training, the next level receives real speech embeddings from the previous level. And introduce a sampling probability that varies with time. To control the input source for the next time step. Probability It follows the law of exponential decay, as shown below:

[0155]

[0156]

[0157] in, It can be set to 0.9 per round. The exponential decay rate is used to gradually transition from guided supervision to autonomous prediction.

[0158] For example, server 104 can perform a second round of speech prediction extraction operation for the target speaker based on the aforementioned context fusion features, and obtain an initial sampling probability (e.g., set to) according to the training strategy. =0.9) and an attenuation coefficient (such as =0.98), and then the actual sampling probability of this segment was calculated using exponential decay. This refers to the probability that the segment uses the true embedding rather than the predicted embedding. Server 104 can use a random sampling mechanism to determine whether to use the current context-fused features or directly use the feature segment itself as the target embedding feature. This randomized guidance mechanism constitutes a teacher forcing strategy, which relies more on the true embedding in the early stages of training, and gradually decentralizes the model to predict itself in the later stages, enhancing its generalization and adaptive capabilities. After determining the target embedding feature, server 104 can send it to the speaker separation module, combine it with the original mixed speech, and use a masking mechanism to complete the second target sound source speech extraction, outputting the second predicted audio data corresponding to the segment.

[0159] Step S308: Calculate the causal layer loss based on the second predicted audio data and the original audio segments corresponding to the feature segments.

[0160] For example, server 104 can obtain the prediction error of the causal segment by calculating the difference between the second predicted audio data segment and the original target audio segment. Server 104 can use scale-invariant signal-to-noise ratio (SI-SDR) as an evaluation metric to calculate the semantic consistency between the model's prediction result and the true target, and then inversely transform it into a causal layer loss. It is worth noting that, to reflect the temporal importance of the causal inference chain, server 104 can assign different weights to different segments, such as using an exponential function weighting method, so that the later time segments account for a larger proportion of the total loss, thereby strengthening the model's ability to learn long-term attentional continuity.

[0161] Step S310: Train the EEG auditory speech extraction model based on the causal layer loss and the master model loss until the preset termination condition is met, and obtain the trained EEG auditory speech extraction model.

[0162] For example, server 104 can combine the causal layer loss and the main model loss to form an overall loss function, and perform joint training optimization on all model parameters. For example, the definition of SI-SDR can be:

[0163]

[0164] in, and These represent the extracted speech signal and the clean speech signal, respectively. Main model loss function. It can be:

[0165]

[0166] Furthermore, to mitigate error propagation in the early stages of the causal inference chain, server 104 can... Introducing exponential weights As shown below:

[0167]

[0168] Total loss The loss can be determined by the main model. And the hierarchical loss of the causal chain Composition. The main model provides global predictions, while the causal chain refines local details.

[0169]

[0170] Through backpropagation, server 104 can perform end-to-end updates on all modules in the causal inference chain, including the feature segmentation module, context fusion module, mask prediction module, and speech decoding module. Server 104 continuously repeats this process, traversing all training samples and all time segments, and evaluating the comprehensive performance index on the validation set after each training round to determine whether the preset termination condition is met. When the total loss value tends to stabilize, the validation index shows no significant improvement, or the maximum number of training rounds is reached, server 104 can save the parameters in the current state as the finally trained EEG auditory speech extraction model.

[0171] Through this causal reasoning chain mechanism, server 104 can not only extract the target sound source within a single time segment, but also maintain continuity and semantic coherence throughout the entire speech sequence, thereby significantly improving the stability and robustness of target speaker extraction. Especially when the subject's attention slightly drifts or the speech content changes, server 104 can also make more reasonable judgments based on the preceding and following context. This causal reasoning ability is a key advantage that traditional static models cannot possess.

[0172] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0173] Based on the same inventive concept, this application also provides an EEG auditory-speech extraction model training device for implementing the aforementioned EEG auditory-speech extraction model training method. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more EEG auditory-speech extraction model training device embodiments provided below can be found in the limitations of the EEG auditory-speech extraction model training method described above, and will not be repeated here.

[0174] In one exemplary embodiment, such as Figure 7 As shown, a training device for an EEG auditory speech extraction model is provided, comprising: a data acquisition module 702, a feature extraction module 704, a feature fusion module 706, and a model training module 708, wherein:

[0175] The data acquisition module 702 is used to acquire electroencephalogram (EEG) data and mixed speech data; wherein, the mixed speech data includes multiple raw audio data from different sound sources; and the EEG data includes EEG signal data from multiple EEG channels.

[0176] The feature extraction module 704 is used to extract features from mixed speech data to obtain audio coding features, and to extract features from electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features. Among them, the EEG temporal features are used to characterize the changing trend of EEG signal data over time, and the EEG spatial features are used to characterize the changing features of EEG signal data between different EEG channels.

[0177] The feature fusion module 706 is used to fuse audio coding features, EEG temporal features and EEG spatial features to obtain speech embedding features, and to separate the first predicted audio data of the target sound source from the mixed speech data based on the speech embedding features.

[0178] The model training module 708 is used to calculate the main model loss based on the first predicted audio data and the original audio data of the target sound source, and to train the EEG auditory speech extraction model based on the main model loss until the preset termination condition is met, so as to obtain the trained EEG auditory speech extraction model.

[0179] In one embodiment, the device further includes:

[0180] The speech processing module is used to divide the speech embedding features into multiple consecutive feature segments; each feature segment corresponds to a raw audio segment in the raw audio data of the target sound source.

[0181] The context fusion module is used to sequentially obtain context fusion features for each feature fragment, based on the feature fragment and the previous feature fragment;

[0182] The causal processing module is used to separate the second predicted audio data of the target sound source from the mixed speech data based on context fusion features; and to calculate the causal layer loss based on the second predicted audio data and the original audio segments corresponding to the feature segments.

[0183] The model training module 708 is specifically used to train the EEG auditory speech extraction model based on the causal layer loss and the master model loss until the preset termination condition is met, thus obtaining the trained EEG auditory speech extraction model.

[0184] In one embodiment, the causal processing module is specifically used to: obtain an initial sampling probability and an attenuation coefficient; calculate the sampling probability of a feature segment based on the initial sampling probability and the attenuation coefficient; select a context fusion feature or feature segment as a target embedding feature based on the sampling probability; and separate the second predicted audio data of the target sound source from the mixed speech data based on the target embedding feature.

[0185] In one embodiment, the feature extraction module 704 is specifically used to: perform time-position encoding on the EEG data to obtain a first EEG data sequence; perform convolution calculation on the time axis for the first EEG data sequence to obtain EEG time features; perform channel position encoding on the EEG data to obtain a second EEG data sequence; and perform convolution calculation between different EEG channels for the second EEG data sequence to obtain EEG spatial features.

[0186] In one embodiment, the feature fusion module 706 is specifically used to: perform self-attention alignment on the time axis of EEG temporal features and EEG spatial features to obtain EEG signal features; and fuse the EEG signal features and audio coding features to obtain speech embedding features.

[0187] In one embodiment, the feature fusion module 706 is further configured to: determine the number of speech frames based on the audio coding features; perform linear interpolation calculation on the EEG signal features based on the number of speech frames to update the EEG signal features; and align and fuse the updated EEG signal features with the audio coding features to obtain speech embedding features.

[0188] Each module in the aforementioned EEG auditory speech extraction model training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0189] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores mixed speech data, electroencephalogram (EEG) data, and data used in feature extraction, feature fusion, and audio prediction. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for training an EEG-auditory-speech extraction model.

[0190] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 9As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a training method for an EEG-based auditory-speech extraction model. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0191] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0192] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: acquiring electroencephalogram (EEG) data and mixed speech data; wherein the mixed speech data includes multiple raw audio data from different sound sources; the EEG data includes EEG signal data from multiple EEG channels; performing feature extraction on the mixed speech data to obtain audio coding features, and performing feature extraction on the EEG data to obtain EEG temporal features and EEG spatial features; wherein the EEG temporal features are used to characterize the changing trend of EEG signal data over time, and the EEG spatial features are used to characterize the changing features of EEG signal data between different EEG channels; fusing the audio coding features, EEG temporal features, and EEG spatial features to obtain speech embedding features, and separating first predicted audio data of the target sound source from the mixed speech data based on the speech embedding features; calculating the master model loss based on the first predicted audio data and the raw audio data of the target sound source, and training the EEG auditory speech extraction model based on the master model loss until a preset termination condition is met, thereby obtaining a trained EEG auditory speech extraction model.

[0193] In one embodiment, when the processor executes the computer program, it further performs the following steps: dividing the speech embedding features into a series of consecutive feature segments; each feature segment corresponds to an original audio segment in the original audio data of the target sound source; sequentially, for each feature segment, obtaining context fusion features based on the feature segment and the previous feature segment; separating the second predicted audio data of the target sound source from the mixed speech data based on the context fusion features; calculating the causal layer loss based on the second predicted audio data and the original audio segment corresponding to the feature segment; training the EEG auditory speech extraction model based on the causal layer loss and the master model loss until a preset termination condition is met, thereby obtaining the trained EEG auditory speech extraction model.

[0194] In one embodiment, when the processor executes the computer program, it further performs the following steps: obtaining an initial sampling probability and an attenuation coefficient; calculating the sampling probability of a feature segment based on the initial sampling probability and the attenuation coefficient; selecting a context fusion feature or feature segment as a target embedding feature based on the sampling probability; and separating the second predicted audio data of the target sound source from the mixed speech data based on the target embedding feature.

[0195] In one embodiment, when the processor executes the computer program, it further performs the following steps: temporal location encoding of the electroencephalogram (EEG) data to obtain a first EEG data sequence; performing convolution calculation on the time axis for the first EEG data sequence to obtain EEG temporal features; performing channel location encoding of the EEG data to obtain a second EEG data sequence; and performing convolution calculation between different EEG channels for the second EEG data sequence to obtain EEG spatial features.

[0196] In one embodiment, when the processor executes the computer program, it further performs the following steps: performing self-attention alignment on the time axis of EEG temporal features and EEG spatial features to obtain EEG signal features; and fusing the EEG signal features and audio coding features to obtain speech embedding features.

[0197] In one embodiment, when the processor executes the computer program, it further performs the following steps: determining the number of speech frames based on audio coding features; performing linear interpolation calculation on the EEG signal features based on the number of speech frames to update the EEG signal features; and aligning and fusing the updated EEG signal features with the audio coding features to obtain speech embedding features.

[0198] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0199] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0200] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0201] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0202] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0203] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training an EEG-based auditory speech extraction model, characterized in that, The method includes: Acquire electroencephalogram (EEG) data and mixed speech data; wherein the mixed speech data includes multiple raw audio data from different sound sources; the EEG data includes EEG signal data from multiple EEG channels; Feature extraction is performed on the mixed speech data to obtain audio coding features, and feature extraction is performed on the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features; wherein, the EEG temporal features are used to characterize the changing trend of the EEG signal data over time, and the EEG spatial features are used to characterize the changing features of the EEG signal data between different EEG channels; The audio coding features, the EEG temporal features, and the EEG spatial features are fused to obtain speech embedding features, and the first predicted audio data of the target sound source is separated from the mixed speech data based on the speech embedding features. The master model loss is calculated based on the first predicted audio data and the original audio data of the target sound source, and the EEG auditory speech extraction model is trained based on the master model loss until a preset termination condition is met, thus obtaining the trained EEG auditory speech extraction model.

2. The method according to claim 1, characterized in that, The method further includes: The speech embedding features are divided into multiple consecutive feature segments; each feature segment corresponds to an original audio segment in the original audio data of the target sound source; For each feature segment, context fusion features are obtained based on the feature segment and the previous feature segment. The second predicted audio data for separating the target sound source from the mixed speech data based on the context fusion features; Calculate the causal layer loss based on the second predicted audio data and the original audio segment corresponding to the feature segment; The step of training the EEG auditory-speech extraction model based on the master model loss until a preset termination condition is met, to obtain the trained EEG auditory-speech extraction model, includes: The EEG auditory speech extraction model is trained based on the causal layer loss and the master model loss until a preset termination condition is met, resulting in a well-trained EEG auditory speech extraction model.

3. The method according to claim 2, characterized in that, The second predicted audio data, which separates the target sound source from the mixed speech data based on the context fusion features, includes: Obtain the initial sampling probability and attenuation coefficient; The sampling probability of the feature segment is calculated based on the initial sampling probability and the attenuation coefficient; Based on the sampling probability, the context fusion feature or the feature fragment is selected as the target embedding feature; The second predicted audio data, which separates the target sound source from the mixed speech data based on the target embedding features, is then used to separate the target sound source.

4. The method according to any one of claims 1 to 3, characterized in that, The step of extracting features from the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features includes: The electroencephalogram (EEG) data is time-location encoded to obtain a first EEG data sequence; For the first EEG data sequence, convolution calculation is performed on the time axis to obtain EEG temporal features; Channel location encoding is performed on the electroencephalogram (EEG) data to obtain a second EEG data sequence; For the second EEG data sequence, convolution calculations are performed between different EEG channels to obtain EEG spatial features.

5. The method according to claim 4, characterized in that, The process of fusing the audio coding features, the EEG temporal features, and the EEG spatial features to obtain speech embedding features includes: Self-attention alignment is performed on the time axis of the EEG temporal features and the EEG spatial features to obtain EEG signal features; The EEG signal features and the audio coding features are fused to obtain speech embedding features.

6. The method according to claim 5, characterized in that, The process of fusing the EEG signal features and the audio coding features to obtain speech embedding features includes: The number of speech frames is determined based on the audio coding features; Linear interpolation is performed on the EEG signal features based on the number of speech frames to update the EEG signal features; The updated EEG signal features are aligned and fused with the audio coding features to obtain speech embedding features.

7. A training device for an EEG-auditory speech extraction model, characterized in that, The device includes: The data acquisition module is used to acquire electroencephalogram (EEG) data and mixed speech data; wherein, the mixed speech data includes multiple raw audio data from different sound sources; and the EEG data includes EEG signal data from multiple EEG channels. The feature extraction module is used to extract features from the mixed speech data to obtain audio coding features, and to extract features from the electroencephalogram (EEG) data to obtain EEG temporal features and EEG spatial features; wherein, the EEG temporal features are used to characterize the changing trend of the EEG signal data over time, and the EEG spatial features are used to characterize the changing features of the EEG signal data between different EEG channels; The feature fusion module is used to fuse the audio coding features, the EEG temporal features and the EEG spatial features to obtain speech embedding features, and to separate the first predicted audio data of the target sound source from the mixed speech data based on the speech embedding features; The model training module is used to calculate the master model loss based on the first predicted audio data and the original audio data of the target sound source, and to train the EEG auditory speech extraction model based on the master model loss until a preset termination condition is met, thereby obtaining the trained EEG auditory speech extraction model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images