Method and device for extracting target person's speech by integrating sound and electroencephalogram signal

Abstract

The present invention belongs to the field of speech processing technology, and specifically relates to a method, system and device for extracting the speech of a target person by fusing sound and electroencephalogram signals. The method designs a speech separation model including an electroencephalogram encoder, a speech encoder, a separation network and a speech decoder, and uses the trained speech separation model to perform the speech extraction task of the target person. Among them, the electroencephalogram encoder includes a one-dimensional convolution layer and a KAN-based attention layer. The speech encoder includes a one-dimensional convolution layer and a bidirectional Mamba layer. The separation network uses a cross-attention module to perform feature fusion on the extracted speech features and electroencephalogram features, and parses the obtained fusion features to obtain a feature mask of the target person's speech. The speech decoder reconstructs the pure speech of the target person using the feature mask and the speech features. The present invention solves the problem that the existing solution is difficult to effectively capture deep electroencephalogram features and global speech features, and the accuracy of the reconstructed target person's speech is poor.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech processing, and in particular relates to a method for extracting speech of a target person by integrating sound and electroencephalogram signals, and a corresponding system and device. Background Art

[0002] In an environment with multiple speakers, the collected speech of the person is often affected by environmental noise and interfering speech signals from other speakers. This is called the cocktail party problem. To solve this problem, technicians have designed a variety of algorithms to achieve target person speech extraction.

[0003] In the existing target person speech extraction technology, the reference speech sample of the target speaker is usually used to imitate the top-down attention mechanism of the human brain. The reference speech guides the algorithm to focus on attention, so that the system can effectively extract the target speech. However, there is still an obvious problem with this type of scheme, that is, the algorithm model cannot actually understand the attention of the human brain, that is, the neuronal response at a cocktail party. In order to further simulate the auditory perception of the human brain, researchers decoded the target person's speech stream from the electroencephalogram (EEG), which is called auditory attention decoding. In recent years, the research on neural (EEG clues) target person speech extraction based on auditory attention decoding has gradually increased, and has achieved good results.

[0004] However, since speech is a signal that changes continuously over time, it is difficult to capture this long-distance dependency by only using a one-dimensional convolutional network to extract speech features. Moreover, EEG contains a large amount of interference components. Only convolutional networks or deep convolutional networks cannot extract deeper information about the target person in the EEG signal. The receptive field is limited, so the computational complexity will be very high. Therefore, those skilled in the art are in urgent need of a target person speech extraction solution that can enhance the feature extraction method to solve the above problems. Summary of the invention

[0005] In order to solve the problem that the existing target person speech extraction methods are difficult to effectively obtain deep information in EEG signals and speech signals, which leads to poor accuracy of reconstructed pure speech, the present invention provides a target person speech extraction method, system and device that integrates sound and EEG signals.

[0006] The technical solution provided by the present invention is:

[0007] A method for extracting a target person's voice by integrating sound and EEG signals is used to extract the target person's pure voice from a mixed audio signal containing the target person's voice by combining the target person's EEG signals. The method includes the following steps:

[0008] S1: Build a speech separation model consisting of an EEG encoder, a speech encoder, a separation network, and a speech decoder.

[0009] The EEG encoder includes a one-dimensional convolutional layer and a KAN-based attention layer, and is used to extract EEG features contained in the input EEG signal. The speech encoder includes a one-dimensional convolutional layer and a bidirectional Mamba layer, and is used to extract speech features contained in the input mixed audio signal.

[0010] The separation network uses a cross-attention module to fuse the extracted speech features and EEG features, and analyzes the fused features to obtain the feature mask of the target person's speech. The speech decoder uses the feature mask to mask the speech features to obtain masked speech, and then reconstructs the target person's pure speech from the masked speech through a deconvolution module.

[0011] S2: Obtain a large number of speech signals, EEG signals and mixed audio signals of the target person that are synchronized in the time domain; use the associated mixed audio signals and EEG signals as sample information and the speech signals as label information to obtain the required sample data set.

[0012] S3: Use the scale-invariant signal-distortion ratio SI-SDR as the loss function L SI-SDR , the speech separation model is trained using the sample data set; the model parameters of the network model that maximizes SI-SDR are retained.

[0013] S4: The EEG signal and mixed audio signal of the target person collected synchronously are input into the trained speech separation model, and the speech separation model outputs the reconstructed pure speech of the target person.

[0014] As a further improvement of the present invention, in step S1, the attention layer used by the EEG encoder includes 1 multi-head attention module, 1 front Dropout layer, 1 front normalization layer, 3 KAN layers, 1 rear Dropout layer and 1 rear normalization layer in sequence. Among them, the output of the front Dropout layer and the input of the multi-head attention module are concatenated with features and then used as the input of the front normalization layer; the output of the rear Dropout layer and the output of the front normalization layer are concatenated with features and then used as the input of the rear normalization layer.

[0015] As a further improvement of the present invention, in step S1, in the bidirectional Mamba layer adopted by the speech encoder, the mixed audio signal after convolution is divided into two paths, one of which is directly processed by multiple Mamba modules; the other is also processed by multiple Mamba modules after signal inversion processing; finally, the feature information output from the two branches is spliced ​​to obtain the extracted speech features.

[0016] As a further improvement of the present invention, the data processing process of the Mamba module is as follows:

[0017] After the input feature information enters the Mamba module, it is divided into two paths. One path is processed by the linear layer and the SiLU activation layer in sequence; the other path is processed by the linear layer, the one-dimensional convolutional layer, the SiLU activation layer, the state space model SSM and the linear layer in sequence; finally, the feature vectors output in the two paths are multiplied to obtain the output features.

[0018] As a further improvement of the present invention, the separation network includes 1 layer normalization layer, 8 front depth convolution layers, 1 cross attention module, 16 rear depth convolution layers, 1 PReLU activation layer and 1 one-dimensional convolution layer. After the speech features are processed by the layer normalization layer and the front depth convolution layer, they are input into the cross attention module together with the EEG features to obtain fused features. The fused features are processed by the rear depth convolution layer, the PReLU activation layer and the one-dimensional convolution layer in turn to obtain the required feature mask.

[0019] As a further improvement of the present invention, the cross attention module includes two branches of EEG feature processing and speech feature processing, and is composed of multiple layers of cross attention units and normalization layers. Each layer of cross attention units performs jump connection and group normalization between two branches. The speech features and EEG features of the left and right branches are superimposed layer by layer, and finally spliced ​​with the original speech features and EEG features and processed by one-dimensional convolution to obtain the desired fusion features.

[0020] As a further improvement of the present invention, the convolution kernel size of the one-dimensional convolution layer and the deconvolution layer in the speech separation model is 32, and the step size is 8.

[0021] As a further improvement of the present invention, in step S2, the sampling rate of the EEG signal is uniformly set to 128 Hz.

[0022] In step S3, the expression of SI-SDR is:

[0023]

[0024] In the above formula, s Represents the original speech signal of the target person in the sample data set; Represents the speech signal of the target person reconstructed by the network model.

[0025] The present invention also includes a target person voice extraction system, which is used to extract the target person's pure voice contained in the mixed audio signal according to the target person's EEG signal and the mixed audio signal. The target person voice extraction system includes: a signal receiving module, a preprocessing module, and a voice reconstruction module.

[0026] Among them, the signal receiving module is used to obtain the mixed audio signal and EEG signal of the target person containing timestamp information, and align the time axis of the two.

[0027] The preprocessing module is used to reduce noise and resample the received EEG signals, filter out interference information, and set the sampling rate of the EEG signals to 128 Hz.

[0028] The speech reconstruction module uses the speech separation model trained in the target person speech extraction method of integrating sound and EEG signals as described above. The speech separation model is used to reconstruct and output the target person's pure speech based on the input target person's EEG signal and mixed audio signal.

[0029] The present invention also includes a target person voice extraction device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, a target person voice extraction system as described above is created, and is used to output the target person's pure voice contained in the mixed audio signal according to the input target person's EEG signal and mixed audio signal.

[0030] The technical solution provided by the present invention has the following beneficial effects:

[0031] The present invention designs a new type of speech separation network for extracting the speech of a target person from mixed audio in combination with EEG signals. In the speech separation network, on the one hand, the present invention introduces SpeechBiMamba that can model long sequences of speech signals to construct a new dual-branch Mamba, and uses it as part of the speech encoder. On the other hand, the KAN network is combined with the attention mechanism to construct an EEGKAN for EEG encoding, and uses it as part of the EEG encoder. The speech encoder and EEG encoder of the improved feature extraction part of the present invention can obtain global speech features and deep EEG features, and then use the two types of feature information to recognize and extract the speech of the target person contained in the mixed audio.

[0032] The solution provided by the present invention shows significant advantages in the target task of speech extraction. The solution does not rely on any prior knowledge about the target speaker, thereby showing stronger robustness and applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of the steps of the target person's voice extraction method that integrates sound and EEG signals provided in Example 1 of the present invention.

[0034] Figure 2 This is the network architecture of the speech separation model designed in Example 1 of the present invention.

[0035] Figure 3 This is a schematic diagram of the KAN-based attention layer used in the EEG encoder of Example 1 of the present invention.

[0036] Figure 4 This is a schematic diagram of the Mamba module used in the speech encoder of Example 1 of the present invention.

[0037] Figure 5 This is a schematic diagram of the cross-attention module used in the separation network of Example 1 of the present invention.

[0038] Figure 6 This is a functional schematic diagram of the target person voice extraction system provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0040] Example 1

[0041] The present embodiment provides a method for extracting the speech of a target person by fusing sound and EEG signals, which is used to extract the pure speech of the target person from a mixed audio signal containing the speech of the target person in combination with the EEG signals of the target person. In view of the shortcomings of the existing scheme, the present embodiment mainly makes improvements in the feature extraction network, so as to better extract speech information and EEG information. Specifically, on the one hand, the present embodiment proposes a new bidirectional Mamba structure in the feature extraction of speech signals, so as to better extract global speech information. On the other hand, a KAN-based attention mechanism is introduced in the feature extraction of EEG signals, so as to better extract deep EEG information. The improvements in the present embodiment can significantly improve the extraction effect of the scheme on the speech of the target person.

[0042] Specifically, Figure 1 As shown, the target person voice extraction method provided by this embodiment that integrates sound and EEG signals includes the following process:

[0043] S1: Construct a Figure 2 The speech separation model shown includes an EEG encoder, a speech encoder, a separation network, and a speech decoder.

[0044] The EEG encoder includes a one-dimensional convolution layer and a KAN-based attention layer, and is used to extract EEG features contained in the input EEG signal. The speech encoder includes a one-dimensional convolution layer and a bidirectional Mamba layer, and is used to extract speech features contained in the input mixed audio signal. The separation network uses a cross-attention module to fuse the extracted speech features and EEG features, and parses the obtained fused features to obtain the feature mask of the target person's speech. The speech decoder uses the feature mask to mask the speech features to obtain the masked speech, and then reconstructs the target person's pure speech from the masked speech through the deconvolution module.

[0045] In order to make the solution clearer, the following is a detailed description of the different modules in the designed speech separation model:

[0046] 1. EEG encoder

[0047] In this embodiment, in order to facilitate feature extraction of EEG signals, so that they can be better integrated with speech signals and input into the EEG encoder, the sampling rate of the EEG signals needs to be uniformly set to 128 Hz. Therefore, for EEG signals with a sampling rate generally higher than 128 Hz, it is usually necessary to downsample the EEG signals input into the EEG encoder.

[0048] In the EEG encoder, the input EEG signal is first processed by the convolution layer, which includes a one-dimensional convolution module. The one-dimensional convolution module uses a convolution kernel of size 32 and a step size of 8. The convolution layer can capture the dynamic changes in the spatial dimension of the EEG signal. Next, the EEG signal that has passed the convolution layer is input into the attention layer designed based on the KAN network to complete the extraction of deep EEG information.

[0049] Multilayer Perceptron (MLP) is a feedforward artificial neural network model that can extract multidimensional feature information. KAN has the same fully connected structure as MLP. However, unlike the traditional MLP, which sets a fixed activation function on the node, KAN uses a learnable activation function. At the same time, KAN nodes do not need to perform any nonlinear transformation on the input signal, only a simple summation is required, so KAN is superior to MLP in terms of accuracy and interoperability.

[0050] In order to take advantage of the KAN network and better extract the features of EEG signals with the help of the attention mechanism, this embodiment proposes a new attention layer based on the multi-head attention mechanism and uses KAN instead of MLP, and applies it to the EEG encoder. Specifically, Figure 3As shown, the KAN-based attention layer used in the EEG encoder of this embodiment includes 1 multi-head attention module (Multi-head Attention), 1 front Dropout layer, 1 front normalization layer (Layer Norm), 3 KAN layers, 1 rear Dropout layer and 1 rear normalization layer (Layer Norm). Among them, the output of the front Dropout layer and the input of the multi-head attention module are concatenated with features and used as the input of the front normalization layer; the output of the rear Dropout layer and the output of the front normalization layer are concatenated with features and used as the input of the rear normalization layer.

[0051] 2. Speech Encoder

[0052] In this embodiment, the mixed audio signal containing the target person's voice is output to the speech encoder to extract the speech features contained therein. The mixed audio signal is first input into the convolution layer. The convolution layer used by the speech encoder includes a one-dimensional convolution module. The one-dimensional convolution module uses a convolution kernel of size 32 and a step size of 8. The convolution layer can preliminarily capture the local features of the speech signal. Next, the feature information extracted by the convolution layer is input into the bidirectional Manba layer to realize the capture of global features in the speech signal.

[0053] The Manba module is a feature extraction module designed based on a state space model, which can handle modeling tasks of long sequence data (such as speech signals) well and show advanced performance in feature extraction of speech signals. In this embodiment, in order to more fully capture the global features of speech information, a bidirectional Mamba layer is designed.

[0054] In the bidirectional Mamba layer adopted by the speech encoder of the present embodiment, the feature information of the mixed audio signal after the convolution layer is divided into two paths, which become a forward branch and a reverse branch respectively. In the forward branch, the feature information is processed by multiple Mamba modules. In the reverse branch, the feature information is copied, and then a flip operation is performed, and then the reverse feature information is sent to multiple Mamba modules for processing. Finally, the feature information output in the two branches is spliced ​​to obtain the extracted speech features containing local features and global features.

[0055] In the bidirectional Mamba layer of this embodiment, the number of Mamba modules used in the forward branch and the reverse branch is the same, and the number of Mamba modules actually used can be flexibly adjusted according to the model performance. In practical applications, the optimal number of Mamba modules in the two branches of the bidirectional Mamba layer is 4.

[0056] In this embodiment, the module architecture of the Mamba module is as follows: Figure 4As shown. The Mamba module includes three linear layers (Linear), a one-dimensional convolution module (Conv1D), two activation layers, and an SSM network. Among them, the model size of the Mamba module is 32, the state expansion coefficient of the SSM (state space model) network is 16, the local convolution width is 4, and the module expansion coefficient is 2. The data processing process of the Mamba module is as follows:

[0057] After the input feature information enters the Mamba module, it is divided into two paths. One path is processed by the linear layer and the SiLU activation layer in sequence; the other path is processed by the linear layer, the one-dimensional convolutional layer, the SiLU activation layer, the state space model SSM and the linear layer in sequence; finally, the feature vectors output in the two paths are multiplied to obtain the output features.

[0058] In this embodiment, the SSM network adopted by the bidirectional Mamba layer can be used to perform sequence-to-sequence mapping, and its expression is as follows:

[0059]

[0060] In the above formula, A is the state transfer matrix, B is the input projection matrix, C is the output projection matrix, A∈R N×N , B, C∈R N ; h ( t ) represents the hidden state of the previous layer output, Indicates the hidden state of the current layer; h ( t )、 ∈R N .

[0061] 3. Separate Network

[0062] Combination Figure 1 It can be found that the separation network used in this embodiment includes 1 layer normalization layer (LayerNorm), 8 front depth convolution layers (Depth Conv1D, ×8), 1 cross attention module (CMCA), 16 rear depth convolution layers (Depth Conv1D, ×16), 1 PReLU activation layer and 1 one-dimensional convolution layer (Conv1D, 1×1). After the speech features are processed by the layer normalization layer and the front depth convolution layer, they are input into the cross attention module together with the EEG features to obtain fused features. The fused features are processed by the rear depth convolution layer, PReLU activation layer and one-dimensional convolution layer in turn to obtain the required feature mask.

[0063] In this embodiment, the architecture of the cross-attention module used by the separation network is as follows Figure 5As shown in the figure, it includes two branches of EEG feature processing and speech feature processing, and is composed of multiple layers of cross attention units (CrossAttention) and normalization layers (Group Layer Norm). Each layer of cross attention units performs jump connection and group normalization between the two branches. The speech features and EEG features of the left and right branches are superimposed layer by layer, and finally concatenated with the original speech features and EEG features and processed by one-dimensional convolution to obtain the required fusion features. Among them, the number of cross attention units and normalization layers used in the cross attention module can also be flexibly set according to needs.

[0064] 4. Voice Decoder

[0065] In this embodiment, the speech features extracted by the speech encoder are divided into two paths. One path is used to be input into the separation network together with the EEG features extracted by the EEG encoder for feature fusion and analysis to obtain mask features related to the target person's speech. The other path is input into the speech masker together with the mask features to reconstruct the target person's pure speech.

[0066] In the speech encoder, the feature mask is firstly multiplied element by element with the speech feature, and then the speech feature is masked by the feature mask to obtain the masked speech, which is the speech feature of the target person. Then, a deconvolution module is used to reconstruct the pure speech of the target person from the masked speech. The deconvolution module uses a convolution kernel size of 32 and a step size of 8.

[0067] S2: Obtain a large number of speech signals, EEG signals and mixed audio signals of the target person that are synchronized in the time domain; use the associated mixed audio signals and EEG signals as sample information and the speech signals as label information to obtain the required sample data set.

[0068] In practical applications, existing public datasets can be directly used as the required sample datasets, such as the KUL dataset and the AVED dataset. Considering the data requirements of the speech separation model of this embodiment, the sample data in the two datasets also need to be preprocessed.

[0069] For the KUL dataset, each sample data needs to be filtered through a high-pass filter with a cutoff of 0.5 Hz, and the EEG signal is downsampled from 8192 Hz to 128 Hz, and the MWF filtering method is used to remove artifacts in the signal. For the AVED dataset, the EEG data needs to be downsampled from 1kHz to 128Hz. The original EEG signal also needs to be bandpass filtered in the range of 0.1Hz to 9Hz, retaining only the relevant frequency band; and then re-referenced to electrode CZ.

[0070] S3: Use the scale-invariant signal-distortion ratio SI-SDR as the loss function L SI-SDR , the speech separation model is trained using the sample data set; the model parameters of the network model that maximizes SI-SDR are retained.

[0071] Among them, the expression of the loss function SI-SDR is:

[0072]

[0073] In the above formula, s Represents the original speech signal of the target person in the sample data set; Represents the speech signal of the target person reconstructed by the network model.

[0074] In practical applications, when the speech separation model is trained using gradient descent, the loss function L can be SI-SDR Set to the inverse of the scale-invariant signal distortion ratio SI-SDR of the sample data prediction result. That is:

[0075]

[0076] S4: The EEG signal and mixed audio signal of the target person collected synchronously are input into the trained speech separation model, and the speech separation model outputs the reconstructed pure speech of the target person.

[0077] Example 2

[0078] Based on the solution of Example 1, this embodiment further provides a target person voice extraction system, which is used to extract the target person's pure voice contained in the mixed audio signal according to the target person's EEG signal and the mixed audio signal. Figure 6 As shown, the target person voice extraction system includes: a signal receiving module, a preprocessing module, and a voice reconstruction module.

[0079] Among them, the signal receiving module is used to obtain the mixed audio signal and EEG signal of the target person containing timestamp information, and align the time axis of the two.

[0080] The preprocessing module is used to reduce noise and resample the received EEG signals, filter out interference information, and set the sampling rate of the EEG signals to 128 Hz.

[0081] The speech reconstruction module uses the speech separation model trained in the target person speech extraction method of fusing sound and EEG signals in Example 1. The speech separation model is used to reconstruct and output the pure speech of the target person based on the input EEG signal and mixed audio signal of the target person.

[0082] Example 3

[0083] On the basis of the schemes in Embodiments 1 and 2, this embodiment further provides a target person voice extraction device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, a target person voice extraction system as in Embodiment 2 is created, and is used to output the target person's pure voice contained in the mixed audio signal according to the input target person's EEG signal and mixed audio signal.

[0084] The target person voice extraction device provided in this embodiment is essentially a computer device used to implement the solution in Example 1. The computer device can be a smart terminal, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server or cabinet server (including an independent server, or a server cluster composed of multiple servers) that can execute programs.

[0085] The computer device indicated in this embodiment includes at least but is not limited to: a memory and a processor that can be connected to each other through a system bus. Among them, the memory (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0086] The processor may be a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor is generally used to control the overall operation of a computer device. In this embodiment, the processor is used to run program codes stored in a memory or process data.

[0087] Simulation test

[0088] In order to verify the performance of the target person speech extraction method that integrates sound and EEG signals provided by the present invention, the technicians formulated an experimental plan, trained the corresponding speech separation model (denoted as IFENet) and tested its performance. In order to verify the advantages of the solution of the present invention, the technicians also compared the performance of the solution with the existing solutions in the prior art. In addition, in order to analyze the contribution of the various improvements in the solution of the present invention to the performance improvement of the solution, the technicians also conducted an ablation experiment on the solution. The contents of this experiment are as follows:

[0089] 1. Dataset

[0090] In this experiment, the public datasets KUL and AVED are used to train the speech separation model. The characteristics of the two datasets are as follows:

[0091] KUL dataset: This dataset consists of 16 normal hearing subjects, each with 20 trials. We use the first 8 trials of each subject, where different speech sounds are presented to the left and right ears of the subject. The participants are asked to pay attention to the sound in one ear and ignore the sound in the other ear. The BioSemi ActiveTwo system is used to record 64-channel EEG signals at a sampling rate of 8196 Hz.

[0092] AVED dataset: This dataset includes 20 subjects with normal hearing, with an average age of 20 years old, who all signed informed consent. Each subject performed 16 consecutive trials. Similar to the KUL dataset, each subject wore in-ear headphones and was asked to pay attention to the speech on one side. All speeches came from Mandarin stories told by a man and a woman. 32-channel EEG data were recorded at a rate of 1kHz.

[0093] In this experiment, the sample data in the two datasets were downsampled to 128Hz, and the speech sampling rate was set to 44.1 kHz. Due to the uniqueness of EEG, the EEG of each subject is very different. In order to make full use of EEG information, this experiment trained and tested the model on the data of each subject, and the final result was the average of all subjects. During the experiment, each trial of the subject was divided into a training set, a validation set, and a test set, with a ratio of 80%, 10%, and 10%, respectively, and there was no overlap in the speech signals between the groups.

[0094] 2. Experimental environment and training parameters

[0095] This experiment was conducted on PyTorch, and the model training was completed on an NVIDIA GeForce RTX 4090 GPU. On the KUL dataset, all models were trained using the Adam optimizer with 300 epochs and 8 batches. The maximum learning rate was 0.0002, and the learning rate adjustment strategy was linear warmup under cosine annealing with a warmup ratio of 5%. On the AVED dataset, all models were trained with 400 epochs, and the rest of the settings were the same as those on the KUL dataset.

[0096] 3. Evaluation indicators

[0097] This experiment mainly uses three indicators to evaluate the effectiveness of the scheme, including SI-SDR in dB, PESQ and short-term objective intelligibility (STOI). Among them, SI-SDR is often used in speech separation tasks and has good robustness. PESQ and STOI are used to evaluate the perceptual quality; as well as the intelligibility of the extracted speech relative to the unprocessed multi-talker speech signal. In addition, this experiment will also expand the short-term objective intelligibility (ESTOI) as a supplement.

[0098] 4. Control group

[0099] In the experiment, three existing schemes, BASEN, BASEN* and MSFNet, were selected as the control group for performance comparison with this scheme. Among them, the BASEN scheme is derived from "J. Zhang, Q. Xu, Q.-S. Zhu, and Z.-H. Ling," BASEN: Time-domain brain-assisted speech enhancement network with convolutional cross attention in multi-talker conditions," in Proc. Interspeech, 2023, pp. 3117_3121.". The MSFNet scheme is derived from "Z. Pan, G. Wichern, FG Germain, S. Khurana, and J. LeRoux,'Neuroheed+: Improving neuro-steered speaker extraction with joint auditory attention detection,' in Proc. ICASSP. 2024, pp. 11 456-11460.". The BASEN* scheme is a BASEN model that uses a multi-head attention mechanism in the EEG encoder.

[0100] 5. Performance comparison experimental data and analysis

[0101] The performance of the scheme of the present invention and the three control group schemes on the two data sets are shown in Table 1 below:

[0102] Table 1: Performance of different solutions in control experiments

[0103]

[0104] From the data in the analysis table, we can find that: on the KUL dataset, the scheme of the present invention is 36% higher than MSFNet in SI-SDR index. It is also better than BASEN, BASEN* and MSFNet in PESQ, STOI and ESTOI indicators. In addition, compared with the other two control group schemes, BASEN* performs better in various indicators. For example, the BASEN* scheme is only 0.55 dB lower than the scheme of the present invention in SI-SDR index, and higher than MSFNet. This actually reflects the role of attention mechanism in EEG feature extraction from the side.

[0105] On the AVED dataset, the proposed solution also performs best, with a performance of 1.98 dB higher than MSFNet in SI-SDR index, and a performance improvement of 29%. In addition, in PESQ index, the proposed solution is 0.2 higher than MSFNet, in STOI index, the proposed solution is 0.3 higher than MSFNet, and in ESTOI index, the proposed solution is 0.6 higher than MSFNet. Similarly, on this data, BASEN* also performs well, close to MSFNet in all indicators, and only lower than the proposed solution.

[0106] In summary, it can be concluded that under different data sets and experimental settings, the speech separation model provided by the present invention shows more competitive performance compared with multiple other neural network-based target person speech extraction schemes.

[0107] 5. Ablation experiment

[0108] In order to verify the contribution of the key improvements made by the present invention in the feature extraction network of the speech separation model to the performance of the solution, the technicians also designed an ablation experiment to test the performance comparison of the complete solution of the present invention with the solution without the KAN-based attention layer (EEGKAN) in the EEG encoder and the solution without the bidirectional Mamba layer (SpeechBiManba) in the speech encoder on two data sets. The experimental data obtained are shown in the following table:

[0109] Table 2: Performance of different schemes in ablation experiments

[0110]

[0111] By analyzing the data in the above table, we can find that: From the data, both EEGKAN and SpeechbMamba have contributed to the performance improvement of the solution of the present invention. On the AVED dataset and the KUL dataset, IFENet without SpeechBiMamba is still stronger than IFENet without EEGKAN, which shows that between the two, EEGKAN contributes more to the performance improvement than SpeechbMamba.

[0112] The above experimental data also clearly reveals that the joint use of SpeechBiMamba and EEGKAN can produce the most obvious enhancement in the performance of the scheme. On the KUL dataset, IFENet achieved remarkable results with an SI-SDR of 6.85 dB, an STOI of 0.87, and a PESQ of 2.44. Although the PESQ lags slightly behind the 2.45 achieved when using the SpeechMamba block alone, it still shows good performance. Similarly, in the AVED dataset, IFENet stands out as an excellent model in various indicators, especially achieving an SI-SDR of 8.76 dB, an STOI of 0.88, and a PESQ of 1.91.

[0113] The experimental data of the ablation experiment also proves the importance of comprehensively utilizing the global information in the extracted speech features and the deep information in the EEG features in the target person's speech extraction. The EEGKAN in the EEG encoder and the SpeechBiMamba in the speech encoder provided by the present invention can realize the extraction of relevant feature information.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for extracting target person's speech by integrating sound and EEG signals, characterized in that: It is used to extract the target person's pure voice from the mixed audio signal containing the target person's voice by combining the target person's EEG signal; it includes the following process: S1: Build a speech separation model including EEG encoder, speech encoder, separation network and speech decoder; The EEG encoder includes a one-dimensional convolutional layer and a KAN-based attention layer, and is used to extract EEG features contained in the input EEG signal; The speech encoder includes a one-dimensional convolutional layer and a bidirectional Mamba layer, and is used to extract speech features contained in an input mixed audio signal; The separation network uses a cross attention module to perform feature fusion on the extracted speech features and EEG features, and analyzes the obtained fusion features to obtain a feature mask of the target person's speech; The speech decoder uses the feature mask to mask the speech feature to obtain masked speech, and then reconstructs the pure speech of the target person from the masked speech through a deconvolution module; S2: Obtain a large number of speech signals, EEG signals and mixed audio signals of the target person that are synchronized in the time domain; use the associated mixed audio signals and EEG signals as sample information and the speech signals as label information to obtain the required sample data set; S3: Use the scale-invariant signal-distortion ratio SI-SDR as the loss function L SI-SDR , using the sample data set to train the speech separation model; retaining the model parameters of the network model that maximizes SI-SDR; S4: The EEG signal and mixed audio signal of the target person collected synchronously are input into the trained speech separation model, and the speech separation model outputs the reconstructed pure speech of the target person.

2. The method for extracting target person's speech by fusing sound and EEG signals as claimed in claim 1, characterized in that: In step S1, the attention layer includes one multi-head attention module, one front Dropout layer, one front normalization layer, three KAN layers, one rear Dropout layer and one rear normalization layer in sequence; Among them, the output of the front Dropout layer and the input of the multi-head attention module are used as the input of the front normalization layer after feature splicing; the output of the rear Dropout layer and the output of the front normalization layer are used as the input of the rear normalization layer after feature splicing.

3. The target person voice extraction method of fusing sound and EEG signals as claimed in claim 1, characterized in that: In step S1, in the bidirectional Mamba layer, the mixed audio signal after convolution is divided into two paths, one of which is directly processed by multiple Mamba modules; the other is processed by multiple Mamba modules after signal inversion; finally, the feature information output from the two branches is spliced ​​to obtain the extracted speech features.

4. The target person voice extraction method of fusing sound and EEG signals as claimed in claim 3, characterized in that: The data processing process of the Mamba module is as follows: After the input feature information enters the Mamba module, it is divided into two paths. One path is processed by the linear layer and the SiLU activation layer in sequence; the other path is processed by the linear layer, the one-dimensional convolutional layer, the SiLU activation layer, the state space model SSM and the linear layer in sequence; finally, the feature vectors output in the two paths are multiplied to obtain the output features.

5. The method for extracting target person's speech by fusing sound and EEG signals as claimed in claim 1, characterized in that: The separation network includes 1 layer normalization layer, 8 front deep convolution layers, 1 cross attention module, 16 rear deep convolution layers, 1 PReLU activation layer and 1 one-dimensional convolution layer; after the speech features are processed by the layer normalization layer and the front deep convolution layer, they are input into the cross attention module together with the EEG features to obtain fused features, and the fused features are processed by the rear deep convolution layer, the PReLU activation layer and the one-dimensional convolution layer in turn to obtain the required feature mask.

6. The method for extracting target person's speech by fusing sound and EEG signals as claimed in claim 5, characterized in that: The cross-attention module includes two branches of EEG feature processing and speech feature processing, and is composed of multiple layers of cross-attention units and normalization layers; each layer of cross-attention units performs jump connection and group normalization between the two branches; the speech features and EEG features of the left and right branches are superimposed layer by layer, and finally concatenated with the original speech features and EEG features and processed by one-dimensional convolution to obtain the required fusion features.

7. The method for extracting target person's speech by fusing sound and EEG signals as claimed in claim 5, characterized in that: The convolution kernel size of the one-dimensional convolution layer and the deconvolution layer in the speech separation model is 32, and the step size is 8.

8. The method for extracting target person's speech by fusing sound and EEG signals as claimed in claim 1, characterized in that: In step S2, the sampling rate of the EEG signal is uniformly set to 128 Hz; In step S3, the expression of SI-SDR is: ， In the above formula, s Represents the original speech signal of the target person in the sample data set; Represents the speech signal of the target person reconstructed by the network model.

9. A target person voice extraction system, characterized in that: It is used to extract the pure voice of the target person contained in the mixed audio signal according to the EEG signal of the target person and the mixed audio signal; the target person voice extraction system includes: A signal receiving module, which is used to obtain a mixed audio signal and an EEG signal of a target person containing timestamp information, and align the two signals in time axis; A preprocessing module, which is used to reduce noise and resample the received EEG signal, filter out interference information, and set the sampling rate of the EEG signal to 128 Hz; and A speech reconstruction module adopts a speech separation model trained in the target person speech extraction method integrating sound and EEG signals as described in any one of claims 1 to 8; the speech separation model is used to reconstruct and output the target person's pure speech based on the input target person's EEG signal and mixed audio signal.

10. A target person voice extraction device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, a target person speech extraction system as described in claim 9 is created, and is used to output the target person's pure speech contained in the mixed audio signal based on the input target person's EEG signal and mixed audio signal.

Images