A multi-task target speech separation method and system integrating EEG signals

Abstract

The present application provides a multi-task target speech separation method and system that fuses EEG signals, the method comprising the steps of: acquiring EEG signals and mixed speech signals; extracting audio embedding features of the mixed speech signals; extracting EEG embedding features of the EEG signals; based on the attention mechanism, fusing the EEG embedding features and the audio embedding features to generate fused embedding features; extracting high-level feature data of the fused embedding features, and generating target speech signals through dimensionality reduction and decoding processing. The present application is based on convolutional neural networks and attention mechanisms, and establishes a multi-task target speech separation model by fusing EEG signals and speech signals, so as to directly and accurately extract the target speech without prior information, avoiding information errors caused by inaccurate prior information, and being able to resist interference from environmental noise and multiple people talking, enhancing the stability of the system, and realizing accurate speech separation in complex dialogue scenarios.

Description

Technical Field

[0001] The present application relates to the technical field of speech separation, and in particular to a multi-task target speech separation method and system integrating electroencephalogram signals. Background Art

[0002] Speech separation aims to extract the clear speech of the target speaker from a noisy environment. In the case of multi-person conversations, it becomes particularly difficult to extract the speech of the target speaker, because it is usually necessary to know the target speaker or the number of speakers in advance. At present, there are two main methods in the field of speech separation: time domain methods and time-frequency domain methods. Time domain methods directly predict the clean speech waveform, such as convolutional time-domain audio separation network (Conv-TasNet), but usually require prior information to optimize the prediction process. Time-frequency domain methods usually use short-time Fourier transform (STFT) for speech separation, but they can only enhance the spectral amplitude while keeping the phase unchanged, resulting in loss of phase information, which may lead to deterioration of speech quality. Therefore, existing speech separation methods have several disadvantages. First, they have a high dependence on prior information, which limits their application in practical scenarios such as multi-person conversations. Second, time-frequency domain methods have the problem of phase information loss, which may lead to deterioration of speech quality.

[0003] In group communication, the human auditory system naturally focuses on the speaker of interest while weakening the voices of other competing speakers. When focusing, corresponding EEG signals are generated. How to use this phenomenon, combine EEG signals with speech signals, and distinguish meaningful speech signals from background noise is a difficulty faced by the current field of speech separation technology.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present application provides a multi-task target speech separation method and system that fuses EEG signals, aiming to solve the problems in the prior art that speech separation technology requires prior information, cannot accurately extract speech information in complex environments, or information loss causes speech quality degradation. The present application provides a multi-task target speech separation method and system that fuses EEG signals, based on convolutional neural networks and attention mechanisms, by fusing EEG signals and speech signals, to establish a multi-task target speech separation model, to achieve direct and accurate extraction of the target speech, without the need for prior information, to avoid information errors caused by inaccurate prior information, and to resist interference from environmental noise and multiple people talking, to enhance the stability of the system, to achieve accurate speech separation in complex dialogue scenarios, and has wide applicability.

[0006] This application is implemented through the following technical solutions:

[0007] The first aspect of the present application is to provide a multi-task target speech separation method integrating EEG signals, comprising the following steps:

[0008] S1, obtaining EEG signals and mixed speech signals;

[0009] S2, extracting audio embedding features of the mixed speech signal;

[0010] S3, extracting EEG embedding features of the EEG signal;

[0011] S4. Based on the attention mechanism, the EEG embedding feature and the audio embedding feature are fused to generate a fused embedding feature;

[0012] S5. Extract high-level feature data of the fused embedded features, and generate a target speech signal through dimension reduction and decoding processing.

[0013] Specifically, the step S2 extracts the audio embedding features of the mixed speech signal, comprising the steps of:

[0014] S21, encoding the mixed speech signal to generate primary audio features;

[0015] S22. Perform convolution processing on the primary audio features to generate audio embedding features.

[0016] Specifically, the step S22, performing convolution processing on the primary audio features to generate audio embedding features, comprises the steps of:

[0017] S221, performing layer normalization on the primary audio features to accelerate convergence;

[0018] S222. Continue to perform multi-layer deep one-dimensional convolution processing on the primary audio features to extract features and generate the audio embedding features for feature fusion.

[0019] Specifically, the extracting of the EEG embedding feature of the EEG signal comprises the steps of:

[0020] S31, preprocessing the EEG signal to generate a standard EEG signal;

[0021] S32, performing convolution processing on the standard EEG signal to extract a first EEG embedding feature for the feature fusion.

[0022] Specifically, the step S32 of performing convolution processing on the standard EEG signal to extract the first EEG embedding feature for the feature fusion includes the following steps:

[0023] S321, performing a first convolution process on the standard EEG signal to extract primary EEG feature data;

[0024] S322, performing a second convolution process on the primary EEG feature data to extract the first EEG embedding feature for the feature fusion;

[0025] S323: Perform a third convolution process on the first EEG embedding feature to form a second EEG embedding feature for the feature fusion.

[0026] Specifically, the first EEG embedding feature is subjected to a third convolution process to form a second EEG embedding feature for the feature fusion, and then further comprises the steps of:

[0027] S324, performing pooling processing and full connection layer processing based on the attention mechanism on the second EEG embedding feature to generate an attention detection result;

[0028] The pooling process is used to reduce feature dimensions and enhance the globality of the second EEG embedding feature.

[0029] Specifically, S4, based on the attention mechanism, the EEG embedding feature and the audio embedding feature are feature-fused to generate a fused embedding feature, including the steps of:

[0030] S41, concatenating the EEG embedding feature and the audio embedding feature to form a feature vector;

[0031] S42, converting the feature vector into three matrices based on the attention mechanism after convolution processing;

[0032] S43. Calculate the attention weight based on the three matrices to obtain the fused embedding feature.

[0033] Specifically, the three matrices include a query matrix, a key matrix, and a value matrix; the calculation of the attention weights based on the three matrices to obtain the fused embedding features includes:

[0034] Calculating a dot product of the query matrix and the key matrix;

[0035] Using the Softmax function to obtain the attention weight;

[0036] The attention weight is weighted and summed with the value matrix to obtain the fused embedding feature.

[0037] The second aspect of the present application is to provide a multi-task target speech separation system integrating EEG signals, comprising:

[0038] A data acquisition module, used for acquiring EEG signals and mixed speech signals corresponding to the EEG signals;

[0039] The target speech separation module is used to extract audio embedding features, fuse the audio embedding features with the EEG embedding features, and then output the target speech signal after convolution processing, dimensionality reduction, and decoding, so as to realize multi-task target speech separation based on the attention mechanism;

[0040] The EEG signal processing module is used to extract the EEG embedding features and perform EEG attention detection, provide feature data support for the target speech separation module, and perform mutual verification with the target speech separation module.

[0041] The third aspect of the present application is to provide a storage medium, which is a non-transitory computer-readable storage medium, storing a program of a multi-task target speech separation method for fusing EEG signals, and when the program is executed by a processor, the following steps are implemented:

[0042] Acquire EEG signals and mixed speech signals;

[0043] Extracting audio embedding features of the mixed speech signal;

[0044] Extracting EEG embedding features of the EEG signal;

[0045] Based on the attention mechanism, the EEG embedding feature and the audio embedding feature are fused to generate a fused embedding feature;

[0046] The high-level feature data of the fused embedded features is extracted, and a target speech signal is generated through dimension reduction and decoding processing.

[0047] Compared with the prior art, the present application has the following beneficial effects: (1) Based on EEG signals, through feature extraction and fusion, and the reference of attention mechanism, the speech of the target speaker can be extracted in the complex environment of multi-person conversation, without prior information, and has wider applicability; (2) Directly using brain activity, the speech information of the target speaker is accurately extracted, avoiding the problems caused by inaccurate prior information in traditional methods; (3) Using EEG signals as auxiliary information, it is better to resist environmental noise and interference in multi-speaker situations, thereby improving the robustness of speech separation. In summary, the present invention has significant technical advantages and innovations in the field of speech enhancement, and provides a new solution to the problem of speech separation in multi-person conversation scenarios in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 This is a flowchart of a multi-task target speech separation method integrating EEG signals according to an embodiment of the present application.

[0050] Figure 2 This is a structural diagram of multi-task target speech separation for fusing EEG signals according to an embodiment of the present application.

[0051] Figure 3 It is a structural diagram of feature fusion of an embodiment of the present application.

[0052] Figure 4 This is a flowchart of EEG signal processing based on the attention mechanism according to an embodiment of the present application.

[0053] Figure 5 The present invention is a flowchart of feature fusion according to an embodiment of the present application.

[0054] Figure 6 This is a model framework diagram of multi-task target speech separation based on EEG signals in an embodiment of the present application.

[0055] Figure 7 This is a flowchart of preprocessing of EEG signals according to an embodiment of the present application.

[0056] Figure 8 A schematic diagram of a multi-task target speech separation system integrating EEG signals according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the implementation methods of this application to clearly and completely describe the technical solutions in the implementation methods of this application. Obviously, the described implementation methods are only part of the implementation methods of this application, not all of the implementation methods. Based on the implementation methods in this application, all other implementation methods obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] It should be understood that the terms used in this specification of the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the embodiments of the present application. As used in the specification of the embodiments of the present application and the appended claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms.

[0059] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0060] like Figure 1 As shown, the first aspect of the present application is to provide a multi-task target speech separation method integrating EEG signals, comprising the following steps:

[0061] S1, obtaining EEG signals and mixed speech signals;

[0062] S2, extracting audio embedding features of the mixed speech signal;

[0063] S3, extracting EEG embedding features of the EEG signal;

[0064] S4. Based on the attention mechanism, the EEG embedding feature and the audio embedding feature are fused to generate a fused embedding feature;

[0065] S5. Extract high-level feature data of the fused embedded features, and generate a target speech signal through dimension reduction and decoding processing.

[0066] Specifically, step S1, acquiring the EEG signal and the mixed voice signal is collected by an intelligent terminal, and the intelligent terminal may be an EEG cap, which simultaneously collects the EEG signal and the mixed audio signal corresponding to the EEG signal.

[0067] like Figure 6 As shown, the present application realizes data fusion processing and multi-task processing at the same time. Task one is to separate the target speech from the mixed speech signal, and task two is to perform EEG attention detection based on EEG signals and calculate the accuracy of attention detection. Data fusion is reflected in the interaction between task one and task two, specifically, the EEG high-level representation (EEG embedded features) extracted in task two interacts with the speech separation process of task one, and the final target speech is achieved through fusion processing. The specific fusion process is described in detail through the following process.

[0068] Specifically, step S2, extracting the audio embedding features of the mixed speech signal, comprises the following steps:

[0069] S21, encoding the mixed speech signal to generate primary audio features;

[0070] S22. Perform convolution processing on the primary audio features to generate audio embedding features.

[0071] like Figure 2As shown, the present application constructs a multi-task target speech separation model based on convolutional neural networks (CNN), fully connected networks, convolutional time-domain audio separation networks (Conv-TasNet), etc. First, the input mixed speech signal (SpeechMixture) is encoded by an audio encoder (Audio Encoder) by a one-dimensional convolutional layer (Conv1D) to generate primary audio features to extract time domain and frequency domain features; then, the primary audio features are convolved to generate audio embedding features.

[0072] Step S22 specifically includes:

[0073] S221, performing layer normalization on the primary audio features to accelerate convergence;

[0074] S222. Continue to perform multi-layer deep one-dimensional convolution processing on the primary audio features to extract features and generate the audio embedding features for feature fusion.

[0075] Specifically, the multi-layer deep one-dimensional convolution processing includes layer normalization processing and multi-layer deep one-dimensional convolution layer (Depth1-D Conv) processing.

[0076] Specifically, the encoder output is layer normalized (LayerNorm) to stabilize the separation process and accelerate convergence. Subsequently, the primary audio features are passed through N layers of deep one-dimensional convolutional layers (Depth 1-D Conv) for high-level feature extraction to generate audio embedding features for feature fusion. At the same time, the primary audio features and audio embedding features generated in the above steps are input into other units of the speech separation model after feature fusion to combine the high-level feature data after feature fusion, and are used to retain the original features through skip connections to prevent information loss.

[0077] Specifically, in a multi-task target speech separation method for fusing EEG signals of the present application, step S3, extracting EEG embedding features of the EEG signals, comprises the following steps:

[0078] S31, preprocessing the EEG signal to generate a standard EEG signal;

[0079] S32, performing convolution processing on the standard EEG signal to extract a first EEG embedding feature for the feature fusion.

[0080] Specifically, the EEG attention detection model can not only generate EEG embedding features for feature fusion, but also perform EEG attention detection to verify the accuracy of the results with the final target speech signal.

[0081] The raw EEG signal needs to be preprocessed and standardized. Figure 7 The preprocessing flowchart of the EEG signal shown in the figure performs re-referencing, band-pass filtering, down-sampling and normalization in sequence.

[0082] Specifically, re-referencing: re-referencing the EEG signal to the average response of all electrodes to eliminate differences between electrodes and improve signal consistency and comparability;

[0083] Bandpass filtering: The re-referenced EEG signal is bandpass filtered at 1-50 Hz to remove low-frequency drift and high-frequency noise and highlight the EEG activity of interest;

[0084] Downsampling: Downsample the filtered EEG signal and reduce the sampling rate to 128Hz to reduce the amount of data, improve the efficiency of subsequent processing, and still retain enough information to maintain the accuracy of the signal.

[0085] Normalization: Normalization of downsampled EEG signals ensures data consistency and comparability, making subsequent analysis and research more robust and reliable.

[0086] The preprocessed standard EEG signal can be used to extract EEG embedding features for data fusion.

[0087] Specifically, Figure 4 As shown, step S32, performing convolution processing on the standard EEG signal to extract the first EEG embedding feature for the feature fusion, includes the steps of:

[0088] S321, performing a first convolution process on the standard EEG signal to extract primary EEG feature data;

[0089] S322: Perform a second convolution process on the primary EEG feature data to extract the first EEG embedding feature for the feature fusion.

[0090] S323, performing a third convolution process on the first EEG embedding feature to form a second EEG embedding feature for the feature fusion;

[0091] S324: Perform pooling processing and fully connected layer processing on the second EEG embedding feature to generate an attention detection result.

[0092] Specifically, in step S321, the standard EEG signal can be used to extract low-level features after the first convolution process, which is a one-dimensional convolution layer (Conv1D) process. Then, in step S322, after the second convolution process, N layers of one-dimensional convolution operation (Depth 1-D Conv) are performed to further extract high-level features, thereby generating a first EEG embedding feature. Then, in step S323, the first EEG embedding feature is subjected to a third convolution process to form a second EEG embedding feature.

[0093] Specifically, in step S323, the third convolution processing of the first EEG embedding feature includes: the first PReLU activation function processing, the first layer normalization processing (LayerNorm), the one-dimensional convolution layer (Conv1D) processing, the second PReLU activation function processing and the second layer normalization processing (LayerNorm). The use of the PReLU activation function can increase nonlinearity, and the one-dimensional convolution layer (Conv1D) processing process can perform feature fusion.

[0094] like Figure 2 As shown, the first EEG embedding feature and the second EEG embedding feature are preliminarily fused in the EEG attention detection model, and then sent to the speech separation model to perform comprehensive feature fusion in combination with the audio embedding feature.

[0095] In addition, the generated second EEG embedding feature continues to be pooled (Pooling) and fully connected layer (FC Layer) processed in the EEG attention detection model. Pooling can reduce the feature dimension of the second EEG embedding feature and enhance the globality. After pooling (Pooling) and fully connected layer (FC Layer) processing, the attention detection result can be generated. The attention detection result is related to the selected experimental data. Preferably, in one embodiment, if the experimental data is gender classification related data, then the result of the attention detection is male or female. If the experimental data is boy 1 and boy 2, then the model output [0,1] represents boy 1, and [1,0] represents boy 2.

[0096] Specifically, Figure 5 As shown, S4 of a multi-task target speech separation method of the present application that fuses EEG signals, based on an attention mechanism, fuses the EEG embedding features and the audio embedding features to generate fused embedding features, including the steps of:

[0097] S41, concatenating the EEG embedding feature and the audio embedding feature to form a feature vector;

[0098] S42, converting the feature vector into three matrices based on the attention mechanism after convolution processing;

[0099] S43. Calculate the attention weight based on the three matrices to obtain the fused embedding feature.

[0100] Specifically, the EEG embedding feature (B-embedding) includes the first EEG embedding feature and the second EEG embedding feature.

[0101] Specifically, the three matrices include a query matrix Q, a key matrix K, and a value matrix V; the process of calculating the attention weights based on the three matrices to obtain the fused embedding features includes:

[0102] Calculating the dot product of the query matrix Q and the key matrix K;

[0103] Using the Softmax function to obtain the attention weight;

[0104] The attention weight is weighted and summed with the value matrix V to obtain the fused embedding feature.

[0105] Specifically, during the fusion process, Figure 3 As shown in the figure, the audio embedding feature (A-embedding) and the EEG embedding feature (B-embedding) are first concatenated to form a long feature vector, which is then processed through a one-dimensional convolutional layer (Conv1D). Subsequently, the feature vector is converted into three matrices: query (Q), key (K), and value (V), which are the core of the attention mechanism. When calculating the attention weight, the attention weight is obtained by calculating the dot product of Q and K, and then passing through the Softmax function (a softmax layer is passed after the dot product of Q and K, not shown in the figure). The attention weight is used to perform weighted summation on V to obtain the fused feature representation. Finally, the output of the attention mechanism is added and concatenated for the second time to enhance the richness of the feature representation, and the fused features are processed through a one-dimensional convolutional layer (Conv1D) to obtain the final fused embedded features for subsequent tasks. At this point, the feature fusion process is completed. In the subsequent processing, high-level feature data is extracted from the fused embedded features, and the target speech signal is generated through dimensionality reduction and decoding. Specifically, the present application utilizes the idea of ​​attention in Transformer and introduces an attention mechanism, so that the present application can dynamically adjust the degree of attention to audio and EEG signals.

[0106] This module accepts embeddings from two different sources as input, called A-embedding and B-embedding. First, A-embedding and B-embedding are subjected to one-dimensional convolution operations (Conv1D) to extract local features. Convolution operations help capture important patterns and relationships in the embeddings. Then, the convolved A-embedding and B-embedding are concatenated to form a joint embedding. This concatenation operation allows the module to combine information from both sources into a single representation, thereby enhancing the expressive power of the features. After the concatenation operation, A-embedding and B-embedding are further processed with the joint embedding through an attention mechanism. In this mechanism, the query (Q) is generated from either A-embedding or B-embedding, while the key (K) and value (V) matrices are generated from the joint embedding through a convolution operation (Conv1D). The query matrix Q is multiplied by the transpose of the key matrix K to obtain the attention scores. The attention weights are generated by normalizing the attention scores. These attention weights are then multiplied with the value matrix V to obtain the weighted embedding representation. The weighted embedding is further processed through residual connections and convolution operations to preserve the input features and enhance the representation power. The embeddings processed by the attention mechanism are concatenated again to form the final fused embedding. This fused embedding represents a comprehensive representation that combines information from different sources and weighted features. The final fused embedding is subjected to a one-dimensional convolution operation to extract the final feature representation. This operation helps smooth and integrate the effects from different features.

[0107] Through the above steps, the attention mechanism enables the system to:

[0108] Adaptive adjustment of attention: Dynamically adjust the attention to different features according to the actual situation of the input signal, making the system more flexible when dealing with complex environments.

[0109] Improve processing efficiency: Through weighted summation and feature enhancement, the model can more effectively utilize the information of input data and improve overall performance.

[0110] Coping with changing scenarios: In complex environments with multiple people talking and different sound sources, the attention mechanism can help the system quickly adapt to changes and improve its ability to capture important information.

[0111] Through this method, relevant acoustic features are extracted from EEG signals and combined with audio features to enhance the system's ability to separate the target speech. The preprocessed EEG signals and speech signals are input into the multi-task target speech separation model, which simultaneously trains the speech separation model and the EEG attention detection model to promote mutual learning and improvement between the two. The speech embedding is extracted through the encoder and the EEG embedding is extracted through the EEG attention detection module, and then the information fusion module is used for deep fusion, and finally the speech waveform of the target speaker is restored through the encoder.

[0112] Specifically, S5 of a multi-task target speech separation method for fusing EEG signals of the present application, extracting high-level feature data of the fused embedded features, and generating a target speech signal through dimensionality reduction and decoding processing, comprises the steps of:

[0113] S51, passing the fused embedded features through multiple layers of depth 1-D convolutional layers (Depth 1-D Conv) to extract high-level feature data;

[0114] S52, performing jump connection between the high-level feature data and the audio embedding data to form comprehensive feature data, so as to retain the original features and prevent information loss;

[0115] S53, use 1x1 one-dimensional convolution layer (Conv1D) to compress feature dimensions and reduce computational complexity;

[0116] S54. Use the Sigmoid activation function to generate masks for speech signal separation.

[0117] S55. Decode the target speech signal through a decoder and output the target speech signal.

[0118] Specifically, before step S53, it includes using a parameterized ReLU activation function (PReLU) to increase nonlinearity and improve the model's expressiveness.

[0119] Specifically, after step S5, it also includes S6, calculating the SI-SNR (Scale-invariant source-to-noise ratio) of the separation result and the accuracy of EEG attention detection as evaluation indicators to evaluate the accuracy of the target speech separation method of the fused EEG signal.

[0120] like Figure 8 As shown, the present application also provides a multi-task target speech separation system 600 integrating EEG signals, comprising:

[0121] A data acquisition module 10 is used to acquire EEG signals and mixed speech signals corresponding to the EEG signals;

[0122] The target speech separation module 20 is used to extract audio embedding features, fuse the audio embedding features with the EEG embedding features, and then output the target speech signal after convolution processing, dimensionality reduction, and decoding, so as to realize multi-task target speech separation based on the attention mechanism;

[0123] The EEG signal processing module 30 is used to extract the EEG embedding features and perform EEG attention detection, provide feature data support for the target speech separation module, and perform mutual verification with the target speech separation module.

[0124] Specifically, the encoder and decoder are both composed of convolutional neural networks with a convolution kernel size of 3 and a stride of 1. The speech separation model is composed of multiple deep convolutional neural networks with a convolution kernel size of 3 and a stride of 1, normalization layers, activation functions, and fusion modules. The EEG attention task model is composed of modules such as convolutional networks, fully connected networks, and activation functions. The EEG signal is input into this module to learn the EEG attention detection task, and the representation of the intermediate layer is used as a reference representation input into the fusion module of the speech separation module for representation fusion as guidance information for speech separation. The two tasks are jointly trained. The speech separation task uses SI-SNR Loss as its loss function, and the EEG signal attention detection task uses cross-entropy loss (Cross-Entropy Loss) as its loss function. The joint training loss function is:

[0125] ,

[0126] in, is the output of the speech separation model, is the clean target voice, is the output of the EEG attention detection model, is the EEG attention label. During the model training process, the Adam optimizer is used and the learning rate is set to 0.001. A total of 500 epochs are trained. In addition, in order to prevent overfitting, an early stopping mechanism is introduced.

[0127] The following is a description of the process of the multi-task target speech separation method that fuses EEG signals of the present application in combination with specific usage scenarios: the user wears an intelligent EEG set that matches the multi-task target speech separation method that fuses EEG signals of the present application, listens to mixed sound clips for 10 minutes, and the intelligent EEG set sends EEG signals and corresponding mixed sound clip signals to the multi-task target speech separation system that fuses EEG signals, extracts audio embedding features and EEG embedding features and performs feature fusion, and separates the target sound clip from other non-target sound clips through a speech separation model based on a convolutional neural network, outputs the target speech information, and verifies whether the separated target sound information is accurate by comparing it with the results output by the EEG attention detection model.

[0128] In one embodiment, the present application further provides a storage medium, which is a non-transitory computer-readable storage medium, storing a program of a multi-task target speech separation method for fusing EEG signals, and when the program is executed by a processor, the following steps are implemented:

[0129] Acquire EEG signals and mixed speech signals;

[0130] Extracting audio embedding features of the mixed speech signal;

[0131] Extracting EEG embedding features of the EEG signal;

[0132] Based on the attention mechanism, the EEG embedding feature and the audio embedding feature are fused to generate a fused embedding feature;

[0133] The high-level feature data of the fused embedded features is extracted, and a target speech signal is generated through dimension reduction and decoding processing.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-temporary computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0135] In summary, the present application provides a multi-task target speech separation method that fuses EEG signals, including the steps of: acquiring EEG signals and mixed speech signals; extracting audio embedding features of the mixed speech signals; extracting EEG embedding features of the EEG signals; based on the attention mechanism, fusing the EEG embedding features and the audio embedding features to generate fused embedding features; extracting high-level feature data of the fused embedding features, and generating target speech signals through dimensionality reduction and decoding processing. The present application is based on convolutional neural networks and attention mechanisms, and establishes a multi-task target speech separation model by fusing EEG signals and speech signals, so as to achieve direct and accurate extraction of the target speech without the need for prior information, thus avoiding information errors caused by inaccurate prior information, and being able to resist interference from environmental noise and multiple people talking, thereby enhancing the stability of the system, and achieving accurate speech separation in complex dialogue scenarios, and has wide applicability.

[0136] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A multi-task target speech separation method integrating EEG signals, characterized in that: The following steps are involved: Acquire EEG signals and mixed speech signals; Extracting audio embedding features of the mixed speech signal; Extracting EEG embedding features of the EEG signal; Based on the attention mechanism, the EEG embedding feature and the audio embedding feature are fused to generate a fused embedding feature; Extracting high-level feature data of the fused embedded features, and generating a target speech signal through dimension reduction and decoding processing; The step of extracting the EEG embedding features of the EEG signal comprises the following steps: Preprocessing the EEG signal to generate a standard EEG signal; Performing convolution processing on the standard EEG signal to extract a first EEG embedding feature for the feature fusion; The method of performing convolution processing on the standard EEG signal and extracting the first EEG embedding feature for the feature fusion comprises the following steps: Performing a first convolution process on the standard EEG signal to extract primary EEG feature data; Performing a second convolution process on the EEG primary feature data to extract the first EEG embedding feature for use in the feature fusion; Performing a third convolution process on the first EEG embedding feature to form a second EEG embedding feature for the feature fusion; The extracting of the EEG embedding feature of the EEG signal further includes: performing pooling processing and fully connected layer processing on the second EEG embedding feature based on the attention mechanism to generate an attention detection result, providing support and verification for the separation of the target speech to improve robustness; The pooling process is used to reduce feature dimensions and enhance the globality of the second EEG embedding feature.

2. The multi-task target speech separation method of fusion EEG signal according to claim 1 is characterized in that: The step of extracting the audio embedding features of the mixed speech signal comprises the following steps: Encoding the mixed speech signal to generate primary audio features; The primary audio features are convolved to generate audio embedding features.

3. The multi-task target speech separation method of fusion EEG signal according to claim 2 is characterized in that: The step of performing convolution processing on the primary audio features to generate audio embedding features comprises the following steps: Performing layer normalization on the primary audio features to accelerate convergence; Continue to perform multi-layer deep one-dimensional convolution processing on the primary audio features to extract features and generate the audio embedding features for feature fusion.

4. The multi-task target speech separation method of fusion EEG signal according to claim 1 is characterized in that: The method of fusing the EEG embedding feature and the audio embedding feature based on the attention mechanism to generate a fused embedding feature comprises the following steps: Concatenate the EEG embedding feature and the audio embedding feature to form a feature vector; The feature vector is converted into three matrices based on the attention mechanism after convolution processing; The attention weights are calculated based on the three matrices to obtain the fused embedding features.

5. The multi-task target speech separation method of integrating EEG signals according to claim 4 is characterized in that: The three matrices include a query matrix, a key matrix, and a value matrix; and the calculation of the attention weights based on the three matrices to obtain the fused embedding features includes: Calculating a dot product of the query matrix and the key matrix; Using the Softmax function to obtain the attention weight; The attention weight is weighted and summed with the value matrix to obtain the fused embedding feature.

6. A multi-task target speech separation system integrating EEG signals, characterized in that: include: A data acquisition module, used for acquiring EEG signals and mixed speech signals corresponding to the EEG signals; The target speech separation module is used to extract audio embedding features, fuse the audio embedding features with the EEG embedding features, and then output the target speech signal after convolution processing, dimensionality reduction, and decoding, so as to realize multi-task target speech separation based on the attention mechanism; The EEG signal processing module is used to extract the EEG embedding feature, perform pooling processing and fully connected layer processing on the second EEG embedding feature based on the attention mechanism, generate attention detection results, and provide support and verification for the separation of the target speech to improve robustness; the pooling processing is used to reduce the feature dimension and enhance the globality of the second EEG embedding feature; and perform EEG attention detection through pooling processing and fully connected layer processing to provide feature data support for the target speech separation module and perform mutual verification with the target speech separation module.

7. A storage medium, characterized in that: The storage medium is a non-transitory computer-readable storage medium, storing a program of a multi-task target speech separation method for fusing EEG signals, and when the program is executed by a processor, the following steps are implemented: Acquire EEG signals and mixed speech signals; Extracting audio embedding features of the mixed speech signal; Extracting EEG embedding features of the EEG signal; Based on the attention mechanism, the EEG embedding feature and the audio embedding feature are fused to generate a fused embedding feature; Extracting high-level feature data of the fused embedded features, and generating a target speech signal through dimension reduction and decoding processing; The step of extracting the EEG embedding features of the EEG signal comprises the following steps: Preprocessing the EEG signal to generate a standard EEG signal; Performing convolution processing on the standard EEG signal to extract a first EEG embedding feature for the feature fusion; The method of performing convolution processing on the standard EEG signal and extracting the first EEG embedding feature for the feature fusion comprises the following steps: Performing a first convolution process on the standard EEG signal to extract primary EEG feature data; Performing a second convolution process on the EEG primary feature data to extract the first EEG embedding feature for use in the feature fusion; Performing a third convolution process on the first EEG embedding feature to form a second EEG embedding feature for the feature fusion; The extracting of the EEG embedding feature of the EEG signal further includes: performing pooling processing and fully connected layer processing on the second EEG embedding feature based on the attention mechanism to generate an attention detection result, providing support and verification for the separation of the target speech to improve robustness; The pooling process is used to reduce feature dimensions and enhance the globality of the second EEG embedding feature.

Images