Brain-controlled speaker extraction method and system based on multi-scale speech-EEG fusion
Through the multi-scale speech-Electrobrain fusion method, multi-scale features are extracted using speech encoder and EEE encoder, and combined with graph convolution network and DPRNN to generate masks, the shortcomings of speech separation in the multi-speaker environment in the prior art are solved. , realizes the ability to separate target voice based on personal consciousness, and improves extraction performance and perceived quality.
Patent Information
- Application Number
- CN202410155031.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-02-02
AI Technical Summary
The prior art lacks the effectiveness of human selective attention mechanism when extracting target speech in a multi-speaker environment, and the speech separation algorithm needs to know the number of speakers in advance, and cannot separate target speech based on the listener's subjective consciousness, and use EEG signals. The decoding method relies on auditory attention decoding accuracy and cannot effectively separate the target speaker.
The brain-controlled speaker extraction method of multi-scale speech-Electrobrain fusion is adopted. Multi-scale speech and Electrobrain fusion are extracted respectively through speech encoder and Electrobrain encoder, and corresponding masks are generated using graph convolution network and DPRNN. Dual-modal features filter interfering speech to achieve end-to-end network training and model optimization.
It improves the accuracy and perceived quality of speech extraction, can effectively separate the target speaker based on personal subjective consciousness, fully cover the time and frequency information of speech, and improves the brain-controlled speaker extraction performance.
Smart Images

Figure CN118197346B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target speaker extraction, and in particular to a method and system for extracting a brain-controlled speaker based on multi-scale speech-electroencephalogram fusion. Background Art
[0002] Sound is considered to be a carrier of information. The human brain has excellent selective auditory attention ability, which allows them to extract only the target auditory information in a multi-talker environment such as a cocktail party, while ignoring the interfering speakers. But for listeners with hearing loss, this is a major challenge.
[0003] The rapid development of speech enhancement and speaker extraction algorithms has promoted the progress of hearing aids and is used as a front-end speech processing technology to extract clear target speech for speech applications such as voice activity detection, speaker segmentation, and speech synthesis. However, these methods still lack the effectiveness of the neural mechanism of human selective attention and are subject to environmental limitations in practical applications.
[0004] In recent years, significant progress has been made in the field of speech separation, which aims to separate the voice of a single speaker from a scene with multiple speakers speaking simultaneously. Most speech separation algorithms require the number of speakers in the mixture to be known and fixed, which greatly limits the practicality of these algorithms.
[0005] Furthermore, the separated speech sources are independent of the listener's attention selection, which requires the subsequent speaker extraction network to use the given target speaker information, including neural signals or visual attention, to track the speech. However, in practical applications, the listener will only pay attention to one speaker, and the speech separation algorithm separates all the sound sources and cannot determine which speaker they belong to. Therefore, a speaker verification system needs to be introduced, which will further increase the computational complexity.
[0006] Speaker extraction takes a different strategy, using a speaker encoder to imitate top-down autonomous focusing given the reference clues of the target speaker and only extract the speech of the target speaker, thus avoiding the above problems.
[0007] Common auxiliary reference clues include pre-registered unseen target speech, observable lip movements, spatial position information, or understanding contextual relevance. However, it is very inconvenient to use clean speech signals or face video sequences to pre-register the target speaker's prior information, and it cannot effectively separate the target speaker that you want to focus on based on your subjective consciousness.
[0008] According to the latest research in neuroscience, it has been demonstrated that the auditory attention of a listener can be decoded from a recording of brain activity. EEG signals provide a non-invasive and effective way to study cortical neural activity, which makes it particularly suitable for auditory attention detection tasks. Previous studies have focused on improving the performance of the cascaded scheme of blind source separation and auditory attention decoding. First, the speech envelope of the target speaker is estimated from the listener's EEG signal, and then the estimated envelope is compared with each separated sound source, and the sound source with the highest similarity is selected as the target speech. The performance of this method will largely rely on the accuracy of auditory attention decoding, and it does not avoid the disadvantage of blind source separation that requires knowing the number of speakers in advance. Summary of the invention
[0009] In view of the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a method and system for brain-controlled speaker extraction based on multi-scale speech-EEG fusion, which adopts a time domain method for network design, follows the end-to-end architecture of encoder-decoder, and ensures that all modules are trained in a single neural architecture.
[0010] To this end, the present invention provides, on one hand, a method for extracting a brain-controlled speaker based on multi-scale speech-EEG fusion, comprising: S1, using a speech encoder to encode a mixed speech waveform into multi-scale speech features at multiple time resolutions, including short, medium and long scales; S2, using an EEG encoder to encode multi-channel EEG signals into low-dimensional feature expressions, and using a graph convolutional network to effectively extract non-Euclidean data of EEG from EEG experiments, and learn the interaction between various EEG channels to obtain feature representations of target speaker information; S3, using a speaker extraction network to fuse speech features and EEG features of different scales, and estimating a corresponding receptive mask for each speech feature at each scale for extracting the target speaker, and finally splicing the three receptive masks to obtain a receptive mask containing multi-scale information; S4, using a speech decoder to restore the masked speech features to the final time domain waveform.
[0011] According to another aspect of the present invention, a brain-controlled speaker extraction system based on multi-scale speech-EEG fusion is provided, comprising: a multi-scale speech encoder module, for converting an original mixed speech waveform into multi-scale speech features with different time resolutions, including three scales of short, medium and long; an EEG encoder module, for encoding multi-channel EEG signals into low-dimensional feature expressions, using a graph convolutional network to effectively extract EEG non-Euclidean data from EEG experiments, and learning the interactive relationship between various EEG channels to obtain feature representations of target speaker information; a speaker extraction network module, for respectively fusing the bimodal features of speech and EEG at various scales to utilize the common information for the target speaker contained in the EEG signal and speech, and then using a DPRNN neural network to generate a corresponding mask for speaker extraction, and using a masking-based method to filter out interfering speakers; and a speech decoder module, for restoring the masked speech features to the final time domain waveform.
[0012] The present invention also provides a brain-controlled speaker extraction device based on multi-scale speech-EEG fusion, on which an application is stored, which is used to execute the steps of the brain-controlled speaker extraction method based on multi-scale speech-EEG fusion described above when running.
[0013] Based on the existing brain-controlled speaker extraction method, the present invention realizes the fusion of speech features and EEG features with multiple time resolutions respectively, and further fully covers the time-frequency information of speech on the basis of effectively utilizing the common information contained in EEG and speech for the target speaker. The fusion method adopts three-layer convolution and multi-layer cross attention, which can realize the full learning of complementary information in multimodality. And through the multi-layer graph convolution network to learn the intrinsic relationship between each channel in the EEG signal, the model has a certain understanding ability of the spatial structure of the EEG signal, which is used to better extract the attention information in the listener's EEG. After the network model is trained and the weight parameters are iteratively updated, the brain-controlled speaker extraction performance of the present invention is significantly improved.
[0014] This method uses the EEG signals recorded by the listener to guide the extraction of the target speech, so that the speaker extraction method has the ability to effectively separate the target speaker according to the individual's subjective consciousness, and fuses the multi-scale speech features and EEG features respectively, which can more accurately capture the time domain characteristics of the speech and improve the perceived quality and intelligibility of the speech.
[0015] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0017] Figure 1 This is a flowchart of the steps of a method for extracting a brain-controlled speaker based on multi-scale speech-electroencephalogram fusion according to the first embodiment of the present invention;
[0018] Figure 2 This is a logic block diagram of a brain-controlled speaker extraction process in a brain-controlled speaker extraction method based on multi-scale speech-EEG fusion according to the first embodiment of the present invention;
[0019] Figure 3 This is the overall structural topology diagram of the brain-controlled speaker extraction system based on multi-scale speech-EEG fusion according to the second embodiment of the present invention. DETAILED DESCRIPTION
[0020] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.
[0021] Embodiment 1
[0022] The brain-controlled speaker extraction method based on multi-scale speech-electroencephalography fusion in this embodiment is as follows: Figure 1 As shown, the following steps S1-S4 are included.
[0023] S1. Use a speech encoder to encode a mixed speech waveform into multi-scale speech features at multiple time resolutions, including short, medium and long scales.
[0024] S2. Use the EEG encoder to encode multi-channel EEG signals into low-dimensional feature expressions, and use the graph convolutional network to effectively extract non-Euclidean EEG data from the EEG experiment, fully learn the interactive relationship between each EEG channel, and obtain the feature representation of the target speaker information.
[0025] S3. In the speaker extraction network module, speech features and EEG features of different scales are integrated respectively, and a corresponding receptive mask is estimated for the speech features at each scale to extract the target speaker. Finally, the three receptive masks are concatenated to obtain a receptive mask containing multi-scale information.
[0026] S4. Use the speech decoder to restore the masked speech features into the final time domain waveform and perform loss calculation.
[0027] Specifically, step S1 includes the following aspects:
[0028] Given a multi-speaker mixed speech signal x, which contains the target speech signal s and the interference speech signal b i :
[0029]
[0030] Where I represents the number of interfering speakers in the acoustic scene.
[0031] S101, a speech encoder composed of three parallel one-dimensional convolutions with different filter lengths and rectified linear units (ReLU activation function), converts the input mixed speech into speech features at three time scales. Each convolution is used for a different time resolution, and a multi-scale method is used to fully cover the time-frequency information of the speech:
[0032] X i =ReLU(conv1D(x,1,N,L i )) i=1,2,3
[0033] Among them, x is a given multi-speaker mixed speech signal, the input channel of the one-dimensional convolution is 1, the output channel is N, and the convolution kernel size is L i By varying the size of the convolution kernel, the speech segments are windowed with different window lengths to obtain speech features of three scales.
[0034] Extracting the characteristic representation of the EEG signal in step S2 includes the following steps S201-S203.
[0035] S201. Use an EEG signal e with N channels as an additional auxiliary reference clue, wherein the EEG signal and speech need to be recorded synchronously within the same duration to ensure that the two are aligned in time.
[0036] S202, modeling multi-channel EEG features through three-layer graph convolution, treating each lead in the EEG data as a node, dynamically learning the intrinsic relationship between different channels, and representing it with an N×N adjacency matrix, extracting more discriminative EEG signal features and improving the accuracy of target attention information;
[0037] S203. Use three residual network ResNet blocks to focus on the most important features of the target speaker in the EEG signal and downsample the channel dimension of the EEG features.
[0038] Step S3 is specifically as follows:
[0039] The speaker extraction network mainly consists of three independent branch paths. On each path:
[0040] S301. First, the speech and EEG features with different time scales are fused separately, and the fusion method adopts a convolutional multi-layer cross attention module.
[0041] S302, respectively send the fused features to the dual-path recurrent neural network DPRNN to estimate the target speaker mask M at different time scales i , which is used to filter out interfering speech and retain only the target speech features in the multi-scale speech features.
[0042] S303: The three masks M estimated by the speaker extraction network are i The target speech information at multiple time resolutions is further fused by concatenation to obtain the final mask M: M = concat(M1, M2, M3).
[0043] At the same time, we learn three speech features X from the speech encoder i Perform the same concatenation operation to obtain a speech feature X containing multi-scale information: X = concat(X1, X2, X3).
[0044] S304: Apply element-by-element multiplication between the speech feature X and the mask M to obtain the masked speech feature
[0045] Step S4 specifically includes:
[0046] The padding operation is used in the speech encoder to ensure that the three speech features obtained after the mixed speech has a consistent dimension after the parallel convolution operation, so there is no need to consider the problem of using filters of different lengths in the encoder in the speech decoder.
[0047] S401, the speech decoder uses a relative deconvolution operation, with N input channels and 1 output channel, the convolution kernel size is L1, and the masked speech features are Reconstructing time domain speech samples
[0048]
[0049] like Figure 2 As shown, the implementation strategy of the brain-controlled speaker extraction method of the present invention is as follows:
[0050] Firstly, a multi-scale speech encoder module based on convolutional networks with different filter lengths is constructed to extract multi-scale speech features and capture the time domain characteristics of speech more accurately.
[0051] Then, instead of using the speech envelope in the EEG encoder, feature representations for focused listening in the EEG signal are extracted. A three-layer graph convolutional network and three residual network ResNet blocks are used to deeply extract EEG features containing information about the target speaker based on the channel and time information in the EEG signal.
[0052] Next, the multi-scale speech features and EEG features are used as input. The speaker extraction network contains a multi-branch network structure. The speech features at different scales and the same EEG features are fused on each branch. In order to effectively model long sequence speech, the corresponding mask is generated by repeating the DPRNN dual-path recursive neural network four times. The masked speech features are obtained by multiplying the concatenated mask and the concatenated multi-scale speech features element by element, which only contains the information of the target speaker.
[0053] Finally, the speech decoder restores the masked speech features into a speech waveform to obtain the target speech result extracted by the model.
[0054] This embodiment requires the use of a multimodal dataset containing voice and EEG data synchronized in duration to train the network model. All modules are trained in a single neural architecture, and the parameters and weights of the overall network model are continuously updated after multiple iterations to obtain a brain-controlled speaker extraction model with the smallest validation set loss value during the training process and save it. Among them, the negative scale-invariant signal-distortion ratio (SI-SDR) is used as the objective function in the training stage, as shown below:
[0055]
[0056] in, and s represent the target speech extracted by the model and the real clean target speech, respectively.
[0057] Embodiment 2
[0058] This embodiment provides a brain-controlled speaker extraction system based on multi-scale speech-EEG fusion, which adopts the brain-controlled speaker extraction method based on multi-scale speech-EEG fusion as in Embodiment 1, analyzes the input mixed speech in a multi-speaker scenario and the EEG signal of a specified object, and outputs the extracted target speaker speech that the specified object is concerned about.
[0059] like Figure 3 As shown, the brain-controlled speaker extraction system based on multi-scale speech-EEG fusion of this embodiment includes: a speech encoder module, an EEG encoder module, a speaker extraction network module, and a speech decoder module. The system can also be called a brain-controlled speaker extraction model.
[0060] The multi-scale speech encoder module is used to convert the original mixed speech waveform into multi-scale speech features with different time resolutions, including short, medium and long scales.
[0061] The EEG encoder module is used to encode multi-channel EEG signals into low-dimensional feature expressions. It uses a graph convolutional network to effectively extract non-Euclidean EEG data from EEG experiments and fully learns the interactive relationship between various EEG channels to obtain the feature representation of the target speaker's information.
[0062] The speaker extraction network module integrates the bimodal features of speech and EEG at different scales, effectively utilizes the common information about the target speaker contained in EEG signals and speech, and then uses the DPRNN neural network to generate the corresponding mask for speaker extraction, and uses a masking-based method to filter out interfering speakers.
[0063] The speech decoder module restores the masked speech features into the final time domain waveform and performs loss calculation.
[0064] The following is a detailed introduction to the construction and training of the brain-controlled speaker extraction model.
[0065] To obtain sample EEG signals and sample speech signals in a multi-speaker scenario, it is necessary to ensure that the data is aligned in time, and to perform data preprocessing and data partitioning to obtain training sets, validation sets, and test sets in a certain proportion, without any overlap between the data sets.
[0066] Three convolutional layers with different filter lengths are used in the speech encoder to realize multi-scale feature extraction of speech, avoiding the disadvantage of traditional signal analysis methods that require balancing time-frequency resolution. The multi-resolution speech representation method can more accurately capture the time domain characteristics of speech.
[0067] In the EEG encoder, the envelope of the target speech does not need to be explicitly reconstructed. Instead, the feature representation used for focused listening in the EEG signal is extracted. A three-layer graph convolutional network and three residual network ResNet blocks are used to deeply extract the target speaker features according to the channel and time information in the EEG signal. The speaker extraction network contains a multi-branch network structure. On each branch, the speech features at different scales are fused with the same EEG features, and then the corresponding mask is generated by repeating the DPRNN dual-path recursive neural network four times. The masked speech features are obtained by multiplying the concatenated mask and the concatenated multi-scale speech features element by element, which only contains the information of the target speaker.
[0068] The masked speech features are restored to a speech waveform through a speech decoder to obtain the target speech result extracted by the model, and the loss value is calculated by comparing it with the real clean target speech to verify the model performance.
[0069] All modules are trained in a single neural architecture. After multiple iterations, the parameters and weights of the overall network model are continuously updated to obtain a brain-controlled speaker extraction model with the smallest validation set loss value during the training process and save it.
[0070] The best saved brain-controlled speaker extraction model was tested on the test set, and three objective evaluation indicators were used to measure the quality of speech extracted using the proposed method. The evaluation indicators included scale-invariant signal-distortion ratio (SI-SDR), short-time objective intelligibility (STOI) and perceived speech quality (PESQ).
[0071] The present invention also provides a brain-controlled speaker extraction device with multi-scale speech-electroencephalogram fusion. The extraction device can be an electronic device or a computer-readable storage medium on which an application is stored. When the program is run, it is used to execute the steps of the brain-controlled speaker extraction method according to the present invention.
[0072] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for extracting brain-controlled speakers based on multi-scale speech-EEG fusion, characterized in that: include: S1. Using a speech encoder, a mixed speech waveform is encoded into multi-scale speech features at multiple time resolutions, including short, medium and long scales; S2. Use the EEG encoder to encode the multi-channel EEG signals into low-dimensional feature expressions, and use the graph convolutional network to effectively extract the non-Euclidean data of the EEG from the EEG signals, and learn the interaction between the various EEG channels to obtain the feature representation of the target speaker information, including: S201, modeling multi-channel EEG features through three-layer graph convolution, treating each lead in the EEG data as a node, dynamically learning the intrinsic relationship between different channels, and representing it with an N×N adjacency matrix, extracting more discriminative EEG signal features and improving the accuracy of target attention information; S202, using three residual network ResNet blocks to focus on the most important features of the target speaker in the EEG signal and downsample the channel dimension of the EEG features, wherein the EEG signal and speech need to be recorded synchronously within the same duration to ensure that the two are aligned in time; S3. Use the speaker extraction network to fuse speech features and EEG features at different scales, and estimate a corresponding receptive mask for each speech feature at each scale to extract the target speaker. Finally, concatenate the three receptive masks to obtain a receptive mask containing multi-scale information, including: S301, fusing speech and EEG features with different time scales respectively, using a convolutional multi-layer cross attention module as the fusion method; S302, respectively send the fused features to the dual-path recurrent neural network DPRNN to estimate the target speaker mask M at different time scales i , used to filter out interfering speech and retain only the target speech features in the multi-scale speech features; S303: The three masks M estimated by the speaker extraction network are i We concatenate and further fuse the target speech information at multiple time resolutions to obtain the final mask M: M = concat(M1, M2, M3); at the same time, we learn the three speech features X i Perform the same concatenation operation to obtain a speech feature X containing multi-scale information: X = concat(X1, X2, X3); S304: Apply element-by-element multiplication between the speech feature X and the mask M to obtain the masked speech feature S4, using a speech decoder to restore the masked speech features into a final time domain waveform, including: S401, the speech decoder uses a relative deconvolution operation, with N input channels and 1 output channel, the convolution kernel size is L1, and the masked speech features are Reconstructing time domain speech samples 2. The method for extracting a brain-controlled speaker based on multi-scale speech-electroencephalogram fusion according to claim 1, characterized in that: Step S1 includes: S101, a speech encoder is composed of three parallel one-dimensional convolutions and rectified linear units with different filter lengths, which converts the input mixed speech into speech features of three time scales. Each convolution is used for a different time resolution, and a multi-scale method is used to fully cover the time-frequency information of the speech: X i =ReLU(covn1D(x,1,N,L i )) i=1,2,3 Among them, x is a given multi-speaker mixed speech signal, the input channel of the one-dimensional convolution is 1, the output channel is N, and the convolution kernel size is L i By using different convolution kernel sizes, we perform windowing operations on speech segments with different window lengths to obtain speech features of three scales. The multi-speaker mixed speech signal x contains the target speech signal s and the interference speech signal b i : Where I represents the number of interfering speakers in the acoustic scene.
3. A brain-controlled speaker extraction system based on multi-scale speech-EEG fusion, characterized in that: include: A multi-scale speech encoder module, which is used to convert the original mixed speech waveform into multi-scale speech features with different time resolutions, including short, medium and long scales; The EEG encoder module is used to encode multi-channel EEG signals into low-dimensional feature expressions. It uses a graph convolutional network to effectively extract non-Euclidean EEG data from EEG experiments and learn the interaction between EEG channels to obtain the feature representation of the target speaker information, including: S201, modeling multi-channel EEG features through three-layer graph convolution, treating each lead in the EEG data as a node, dynamically learning the intrinsic relationship between different channels, and representing it with an N×N adjacency matrix, extracting more discriminative EEG signal features and improving the accuracy of target attention information; S202, using three residual network ResNet blocks to focus on the most important features of the target speaker in the EEG signal and downsample the channel dimension of the EEG features, wherein the EEG signal and speech need to be recorded synchronously within the same duration to ensure that the two are aligned in time; The speaker extraction network module is used to fuse the bimodal features of speech and EEG at different scales to utilize the common information about the target speaker contained in EEG signals and speech. Then, the DPRNN neural network is used to generate the corresponding mask for speaker extraction, and the masking-based method is used to filter out interfering speakers, including: S301, fusing speech and EEG features with different time scales respectively, using a convolutional multi-layer cross attention module as the fusion method; S302, respectively send the fused features to the dual-path recurrent neural network DPRNN to estimate the target speaker mask M at different time scales i , used to filter out interfering speech and retain only the target speech features in the multi-scale speech features; S303: The three masks M estimated by the speaker extraction network are i We concatenate and further fuse the target speech information at multiple time resolutions to obtain the final mask M: M = concat(M1, M2, M3); at the same time, we learn the three speech features X i Perform the same concatenation operation to obtain a speech feature X containing multi-scale information: X = concat(X1, X2, X3); S304: Apply element-by-element multiplication between the speech feature X and the mask M to obtain the masked speech feature The speech decoder module is used to restore the masked speech features into the final time domain waveform, including: S401, the speech decoder uses a relative deconvolution operation, with N input channels and 1 output channel, the convolution kernel size is L1, and the masked speech features are Reconstructing time domain speech samples 4. The brain-controlled speaker extraction system based on multi-scale speech-electroencephalogram fusion according to claim 3 is characterized in that: The system training method is as follows: Obtain sample EEG signals and sample speech signals in a multi-talker scenario, align the data in time, and perform data preprocessing and data partitioning to obtain training sets, validation sets, and test sets in a set ratio, without any overlap between the data sets; All modules are trained in a single neural architecture. After multiple iterations, the parameters and weights of the overall network model are continuously updated to obtain and save the brain-controlled speaker extraction model with the smallest validation set loss value during the training process. The best saved brain-controlled speaker extraction model was tested on the test set, and three objective evaluation indicators were used to measure the quality of speech extracted using the proposed method. The evaluation indicators included: scale-invariant signal-distortion ratio SI-SDR, short-time objective intelligibility STOI, and perceptual speech quality evaluation PESQ.
5. A brain-controlled speaker extraction device based on multi-scale speech-electroencephalography fusion, on which an application is stored, characterized in that: When running, the program is used to execute the steps of the brain-controlled speaker extraction method based on multi-scale speech-electroencephalogram fusion according to claim 1 or 2.
