Electroencephalogram auxiliary speech enhancement method and system based on sparse channel

Abstract

The invention belongs to the technical field of speech enhancement, and discloses an electroencephalogram auxiliary speech enhancement method and system based on sparse channels. According to the method, a channel selection technology is fused into brain-assisted speech enhancement, and a time-domain speech signal and an electroencephalogram signal are fused based on a cross attention mechanism, so that end-to-end speech enhancement extraction of a target speaker is realized. According to the brain-assisted speech enhancement technology based on electroencephalogram sparse channel selection, time-domain speech signals and electroencephalogram signals are fused based on a cross attention mechanism, enhancement extraction is carried out on the speech of a target speaker, the number of electroencephalogram signal channels is reduced, and the smaller the number of the electroencephalogram signal channels is, the smaller the number of the electroencephalogram signal channels is, the larger the number of the electroencephalogram signal channels is. The cost of equipment for extracting the electroencephalogram signals is lower. According to the method, the electroencephalogram signals of the listener are used as additional modal information, and the voice signals of the target speaker can be extracted without knowing the information of the target speaker in advance.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech enhancement, and in particular relates to an electroencephalogram-assisted speech enhancement method and system based on sparse channels. Background Art

[0002] For speech enhancement, existing technologies generally include speech enhancement technology, brain-assisted speech enhancement technology, EEG-based target speaker extraction technology, and end-to-end network technology.

[0003] Existing speech enhancement based on brain-assisted technology can extract the target speaker without relying on prior knowledge of the target speaker's information, but brain-assisted speech enhancement still faces challenges in performance and equipment cost.

[0004] Through the above analysis, the problems and defects of the existing technology are: for the hearing-impaired group with reduced auditory attention, the existing technology is poor in accurately focusing on the target speaker's voice in a noisy environment, and the equipment cost for extracting EEG signals is high. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide an EEG-assisted speech enhancement method and system based on sparse channels, and more particularly, relate to an end-to-end EEG-assisted speech enhancement technology based on sparse channel selection.

[0006] The technical solution is as follows: an EEG-assisted speech enhancement method based on sparse channels integrates channel selection technology into brain-assisted speech enhancement, fuses speech signals and EEG signals in the time domain based on a cross-attention mechanism, and realizes end-to-end enhanced extraction of the target speaker's speech; specifically, the following steps are included:

[0007] S1, EEG brain wave signal is filtered by channel selection method to obtain a small number of EEG channel signals, and then EEG feature extraction is performed; the speech signal is passed through the speech signal coding network to obtain the time series speech features, and the speech signal is passed through the speech signal coding network, which is composed of one-dimensional convolution, and the speech signal is extracted through one-dimensional convolution feature extraction to obtain the time series speech features;

[0008] S2, input the speech features and EEG features obtained from the two channels into the multi-layer cross attention mechanism module for feature fusion;

[0009] S3, based on the speech decoding network, decodes the fused features into the enhanced target speaker's speech.

[0010] In step S1, the EEG signal is filtered through a channel selection method to select a small number of EEG channel signals, including:

[0011] In the training process, for N EEG channels, K neurons are selected for screening. The input and output size of each neuron are the number of EEG channels. Through the Gumbel Softmax function, each neuron selects a channel from N EEG signals, and finally K neurons select K EEG channels.

[0012] In the inference process, the EEG signal passes through the EEG channel selection module and outputs a small number of filtered EEG channel signals.

[0013] Furthermore, the training process includes: the EEG channel selection module adopts a residual structure, which is expressed as:

[0014] z final = ( 1-r)z ori +rPadding(z choice )

[0015] In the formula, z final is the final EEG channel signal, r is the coefficient, z ori is the original EEG channel signal, Padding() is the filling operation, z choice is the EEG channel after selection;

[0016] After the original EEG channel signal passes through the EEG channel selection module, the selected EEG channel z is obtained. choice , then multiplied by the coefficient r after the Padding operation, and added to the original EEG channel signal z ori Multiply by the (1-r) coefficient and add the two together to get the final EEG channel signal z final ,Finally, the speech signals of multiple speakers are mixed and input into the main network to obtain the enhanced speech.

[0017] Furthermore, the inference process includes: removing the residual process, the EEG signal passes through the EEG channel selection network, is filled, and is input into the main network together with the mixed speech signals of multiple speakers, and finally the enhanced speech is obtained by the main network.

[0018] In step S2, feature fusion is performed, including: fusing the extracted speech features and EEG features with each other, fusing them through a multi-layer cross-attention mechanism, and then splicing the two features to obtain a fused feature.

[0019] Furthermore, after the fusion of multiple layers of cross-attention mechanisms, fusion is performed through a cross-attention unit module, and the cross-attention unit module performs different processing according to the different input speech features and EEG features.

[0020] Furthermore, for speech features, the Q matrix is ​​obtained after deep separable convolution feature extraction, and the K and V matrices are obtained for EEG features. The attention coefficient matrix is ​​obtained by multiplying the Q matrix by the transpose of the K matrix, and the attention coefficient matrix is ​​then matrix multiplied with the V matrix to obtain the fusion features of the speech features after cross-attention of the EEG features.

[0021] For EEG features, the Q matrix is ​​obtained through the EEG brain electrical features, and the K and V matrices are obtained through the speech features. Finally, the fusion features are obtained after cross-attention of the EEG brain electrical features and the speech features.

[0022] Another object of the present invention is to provide an EEG-assisted speech enhancement system based on sparse channels, which implements the EEG-assisted speech enhancement method based on sparse channels, and the system comprises:

[0023] The EEG channel selection module is used to filter a small number of EEG channel signals through the channel selection method, and then extract EEG features; and to obtain the time-series speech features of speech signals through the speech signal coding network;

[0024] The training and reasoning process module is used to perform training and reasoning on EEG signals and speech signals;

[0025] A multi-layer cross attention mechanism module is used to simultaneously input the acquired speech features and the extracted EEG features into the multi-layer cross attention mechanism module for feature fusion;

[0026] The cross attention unit module is used to perform the fusion of speech features and EEG features, and obtain the fusion features after the speech features are cross-attended with the EEG features;

[0027] The speech signal coding network module is used to decode the fused features into the enhanced target speaker's speech.

[0028] Furthermore, the system is mounted on a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can realize the functions of the above-mentioned sparse channel-based EEG-assisted speech enhancement system.

[0029] Furthermore, the system is mounted on a server, and the server is used to provide a user input interface to implement the functions of the sparse channel-based EEG-assisted speech enhancement system as described above when executed on an electronic device, and the electronic device is mounted on a headset.

[0030] In combination with all the above-mentioned technical solutions, the beneficial effects of the present invention are as follows: the present invention fuses the speech signal and EEG signal in the time domain based on the cross-attention mechanism, enhances the extraction of the target speaker's speech, and reduces the use of the number of EEG signal channels. The fewer the EEG signal channels, the lower the cost of the equipment for extracting EEG signals. For people with hearing impairments, there is a problem of decreased auditory attention and inability to accurately focus on the target speaker's speech in a noisy environment. With the development of brain-computer interface technology, the present invention uses the listener's EEG signal as additional modal information, and can extract the target speaker's speech signal without relying on prior knowledge of the target speaker's information. The present invention is used to extract the target speaker's speech in a multi-speaker scenario while relying only on a small number of EEG channel signals. Through this solution, the number of EEG channels is reduced, which can reduce the cost of EEG signal collection equipment.

[0031] Traditional speech enhancement methods usually rely on a large number of EEG channels, which not only increases the complexity and cost of the equipment, but also limits the popularity of EEG equipment. The present invention uses innovative channel selection technology to significantly reduce the number of EEG channels required while ensuring the speech enhancement effect, thereby reducing the cost and complexity of the equipment. Although increasing the number of channels can improve the speech enhancement effect, this will lead to a significant increase in the cost and complexity of the equipment, limiting the popularity of EEG equipment. The present invention successfully solves this problem through innovative channel selection technology, achieving high-quality speech enhancement while reducing the number of channels. The traditional concept is that in order to achieve high-quality speech enhancement effects, it is necessary to rely on a large number of EEG channels, which leads to an increase in equipment cost and complexity. This prejudice limits the popularity and application of EEG equipment. The present invention uses innovative channel selection technology to prove that effective speech enhancement effects can be achieved even when the number of channels is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0033] Figure 1 is a flow chart of an EEG-assisted speech enhancement method based on sparse channels provided by an embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of the main network process module provided by an embodiment of the present invention;

[0035] Figure 3 is a block diagram of an EEG-assisted speech enhancement system based on sparse channels provided by an embodiment of the present invention;

[0036] Figure 4It is a schematic diagram of a multi-layer cross attention mechanism module provided by an embodiment of the present invention;

[0037] Figure 5 is a schematic diagram of a cross attention unit module provided by an embodiment of the present invention;

[0038] Figure 6 is a schematic diagram of an EEG channel selection module provided by an embodiment of the present invention;

[0039] Figure 7 is a principle diagram of a training process provided by an embodiment of the present invention;

[0040] Figure 8 is a reasoning flow chart provided by an embodiment of the present invention;

[0041] In the figure: 1. EEG channel selection module; 2. Training reasoning process module; 3. Multi-layer cross attention mechanism module; 4. Cross attention unit module; 5. Speech signal encoding network module. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0043] The innovation of the present invention lies in that: by proposing a channel selection technology, the present invention can effectively reduce the dependence of speech enhancement using EEG signals on the number of EEG channels. Generally, the more channels there are, the better the speech enhancement effect is, but the equipment cost will increase. The present invention can effectively reduce the number of EEG channel extractions while ensuring the speech enhancement effect, reduce the equipment cost, and further promote the popularization of EEG equipment.

[0044] The speech signal and EEG signal in the time domain are fused based on the cross-attention mechanism, and the target speaker's speech is enhanced and extracted end-to-end, so that only a small number of EEG channels are needed to complete the enhanced extraction of the target speaker's speech in a noisy environment. Since the more channels a EEG signal collection device has, the higher the cost, the present invention reduces the cost of using EEG information collection equipment, and only a small number of channels of EEG signal collection equipment are needed.

[0045] In Example 1, the sparse channel-based EEG-assisted speech enhancement method provided in the embodiment of the present invention integrates channel selection technology into brain-assisted speech enhancement, fuses the time domain speech signal and EEG signal based on the cross-attention mechanism, and realizes end-to-end enhanced extraction of the target speaker's speech. Specifically, it includes:

[0046] S1, EEG brain wave signal is filtered by channel selection method to obtain a small number of EEG channel signals, and then EEG feature extraction is performed; the speech signal is passed through the speech signal coding network to obtain the time series speech features, and the speech signal is passed through the speech signal coding network, which is composed of one-dimensional convolution, and the speech signal is extracted through one-dimensional convolution feature extraction to obtain the time series speech features;

[0047] S2, input the speech features and EEG features obtained from the two channels into the multi-layer cross attention mechanism module for feature fusion;

[0048] S3, based on the speech decoding network, decodes the fused features into the enhanced target speaker's speech.

[0049] Example 2, illustratively, the overall architecture of the sparse channel-based EEG-assisted speech enhancement system provided by the embodiment of the present invention adopts a main network process module;

[0050] The main network process module is as follows Figure 2 As shown in , the present invention is verified on the EEG and audio data sets of Trinity University Berlin. The EEG signal enters the EEG channel selection module, and the EEG small channel signal after being screened by the EEG channel selection module is then extracted by the EEG signal encoding module. The speech signal passes through the speech signal encoding network to obtain the time-series speech features;

[0051] The speech features and the extracted EEG features are simultaneously input into the multi-layer cross attention mechanism module for feature fusion. Finally, based on the speech decoding network, the fused features are decoded into the enhanced target speaker's speech.

[0052] Embodiment 3, another exemplary, such as Figure 3 As shown, the sparse channel-based EEG-assisted speech enhancement system provided by the embodiment of the present invention includes: an EEG channel selection module 1, a training reasoning process module 2, a multi-layer cross attention mechanism module 3, a cross attention unit module 4, and a speech signal encoding network module 5;

[0053] EEG channel selection module 1, used for filtering a small number of EEG channel signals through a channel selection method, and then extracting EEG features; and for obtaining time-series speech features of speech signals through a speech signal coding network;

[0054] Training and reasoning process module 2, used for training and reasoning EEG signals and speech signals;

[0055] A multi-layer cross attention mechanism module 3 is used to simultaneously input the acquired speech features and the extracted EEG features into the multi-layer cross attention mechanism module for feature fusion;

[0056] The cross attention unit module 4 is used to perform the fusion of speech features and EEG features, and obtain the fusion features after the speech features are cross-attended with the EEG features;

[0057] The speech signal encoding network module 5 is used to decode the fused features into the enhanced target speaker's speech.

[0058] EEG channel selection module 1 Figure 6 As shown in . At present, the mainstream EEG electroencephalogram signal usually needs to sample 128 channels of EEG signals, but EEG acquisition equipment is relatively expensive, and the more channels, the more time-consuming the acquisition is, and the cost is high. The present invention selects a small number of EEG channels from 128 channels through the EEG channel selection module 1, and ensures that the reduction in the number of channels does not affect the effect of model speech enhancement.

[0059] Exemplarily, in the training inference process module 2, during the training process, the specific screening and calculation process is as follows: for N EEG channels, K neurons are selected for screening, and the input and output sizes of each neuron are the number of EEG channels. Through the Gumbel Softmax function, each neuron selects a channel from N EEG signals, and finally K neurons select K EEG channels. Gumbel Softmax is a technique for sampling in discrete distributions, especially when dealing with discrete variables, it can effectively perform gradient propagation. The core idea of ​​Gumbel Softmax is to approximate discrete sampling by introducing Gumbel noise while maintaining differentiability.

[0060] During the inference process, the EEG signal passes through the EEG channel selection module and outputs a small number of filtered EEG channel signals.

[0061] Exemplarily, the training process of the present invention is as follows Figure 7 As shown in . The mixed speech signals of multiple speakers and the collected EEG signals are input into the audio signal extraction module and the EEG feature channel selection module respectively. During the training process, the EEG channel selection module adopts a residual structure, as shown in formula (1).

[0062] z final = ( 1-r)z ori +rPadding(z choice ) (1)

[0063] In the formula, z final is the final EEG channel signal, r is the coefficient, z ori is the original EEG channel signal, Padding() is the filling operation, z choice is the EEG channel after selection;

[0064] After the original EEG channel signal passes through the EEG channel selection module, the selected EEG channel z is obtained. choice , then multiplied by the coefficient r after the Padding operation, and added to the original EEG channel signal z ori Multiply by the (1-r) coefficient and add the two together to get the final EEG channel signal z final ,Finally, the speech signals of multiple speakers are mixed and input into the main network to obtain the enhanced speech.

[0065] The reasoning process of the present invention is as follows Figure 8 As shown in the figure, the residual process is removed during the inference process, the EEG signal passes through the EEG channel selection network, the residual result is used during the training process, and the residual result is removed during the inference. After filling, the filtered EEG features are expanded in channels, and the retained channels are copied in the channel dimension, so that the size of the EEG features after the channel selection network remains consistent with the original features. The mixed speech signals of multiple speakers are input into the main network together, and the enhanced speech is finally obtained by the main network.

[0066] For example, the multi-layer cross attention mechanism module 3 is as follows Figure 4 This module fuses the extracted speech features and EEG features, and then splices the two features to obtain fused features after fusion through multiple layers of cross-attention mechanism. The speech features and EEG features are fused through multiple layers of cross-attention layers. In each layer, the speech features and EEG features are input into the Figure 5 The cross attention unit in the fusion is used for feature fusion, and two fusions are performed in each layer. The first fusion is performed to obtain the Q matrix from the speech features and the KV matrix from the EEG features. The second fusion is performed to obtain the KV matrix from the speech features and the EEG features as the Q matrix.

[0067] Another exemplary embodiment of the cross attention unit module 4 is as follows: Figure 5 As shown in . The input of the cross-attention unit module is speech features and EEG features. The cross-attention unit module has different processing flows depending on the input signal. For speech features, the Q matrix is ​​obtained after deep separable convolution feature extraction, and the K and V matrices are obtained for EEG features. The attention coefficient matrix is ​​obtained by multiplying the Q matrix by the transpose of the K matrix, and the attention coefficient matrix is ​​then matrix multiplied with the V matrix to obtain the fusion features of the speech features after cross-attention of the EEG features.

[0068] For EEG features, the Q matrix is ​​obtained through the EEG brain electrical features, and the K and V matrices are obtained through the speech features, and finally the fusion features after the EEG brain electrical features and the speech features are cross-attentioned are obtained. It can be seen from the above embodiments that the present invention can be used to implement speech enhancement technology that only relies on EEG signals of a small number of EEG channels.

[0069] The present invention has been experimentally verified on the EEG and audio data sets of Trinity University Berlin. Based on the design of the present invention, after reducing the number of channels to 18 channels, the effect is less reduced than that of 128 channels, which proves that the scheme can effectively reduce the dependence on the number of channels, as shown in Table 1.

[0070] Table 1 Channel quantity monitoring table

[0071]

[0072] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. An EEG-assisted speech enhancement method based on sparse channels, characterized in that: This method integrates channel selection technology into brain-assisted speech enhancement, fuses time-domain speech signals and EEG signals based on a cross-attention mechanism, and realizes end-to-end enhanced extraction of the target speaker's speech. Specifically, it includes the following steps: S1, EEG brain wave signal is filtered by channel selection method to obtain a small number of EEG channel signals, and then EEG feature extraction is performed; the speech signal is passed through the speech signal coding network to obtain the time series speech features, and the speech signal is passed through the speech signal coding network, which is composed of one-dimensional convolution, and the speech signal is extracted through one-dimensional convolution feature extraction to obtain the time series speech features; S2, input the speech features and EEG features obtained from the two channels into the multi-layer cross attention mechanism module for feature fusion; S3, based on the speech decoding network, decodes the fused features into the enhanced target speaker's speech.

2. The sparse channel based EEG-assisted speech enhancement method according to claim 1, characterized in that: In step S1, the EEG signal is filtered through a channel selection method to select a small number of EEG channel signals, including: In the training process, for N EEG channels, K neurons are selected for screening. The input and output size of each neuron are the number of EEG channels. Through the Gumbel Softmax function, each neuron selects a channel from N EEG signals, and finally K neurons select K EEG channels. In the inference process, the EEG signal passes through the EEG channel selection module and outputs a small number of filtered EEG channel signals.

3. The sparse channel based EEG-assisted speech enhancement method according to claim 2, characterized in that: The training process includes: the EEG channel selection module adopts a residual structure, and the expression is: With final =(1-r)z ori +rPadding(z choice ) In the formula, z final is the final EEG channel signal, r is the coefficient, z ori is the original EEG channel signal, Padding() is the padding operation, z choice is the EEG channel after selection; After the original EEG channel signal passes through the EEG channel selection module, the selected EEG channel z is obtained. choice , then multiplied by the coefficient r after the Padding operation, and added to the original EEG channel signal z ori Multiply by the (1-r) coefficient and add the two together to get the final EEG channel signal z final ,Finally, the speech signals of multiple speakers are mixed and input into the main network to obtain the enhanced speech.

4. The sparse channel based EEG-assisted speech enhancement method according to claim 2, characterized in that: The reasoning process includes: removing the residual process, the EEG signal passes through the EEG channel selection network, is filled, and is input into the main network together with the mixed speech signals of multiple speakers, and finally the enhanced speech is obtained by the main network.

5. The sparse channel based EEG-assisted speech enhancement method according to claim 1, characterized in that: In step S2, feature fusion is performed, including: fusing the extracted speech features and EEG features with each other, fusing them through a multi-layer cross-attention mechanism, and then splicing the two features to obtain a fused feature.

6. The sparse channel based EEG-assisted speech enhancement method according to claim 5, characterized in that: After multiple layers of cross-attention mechanism fusion, the fusion is performed through a cross-attention unit module, which performs different processing according to the different input speech features and EEG features.

7. The sparse channel based EEG-assisted speech enhancement method according to claim 6, characterized in that: For speech features, the Q matrix is ​​obtained after deep separable convolution feature extraction, and the K and V matrices are obtained for EEG features. The attention coefficient matrix is ​​obtained by multiplying the Q matrix by the transpose of the K matrix, and the attention coefficient matrix is ​​then matrix multiplied with the V matrix to obtain the fusion features of the speech features after cross-attention of the EEG features. For EEG features, the Q matrix is ​​obtained through EEG brain wave features, and the K and V matrices are obtained through speech features. Finally, the fusion features are obtained after cross-attention of EEG brain wave features and speech features.

8. An EEG-assisted speech enhancement system based on sparse channels, characterized in that: The system implements the sparse channel-based EEG-assisted speech enhancement method according to any one of claims 1 to 7, and the system comprises: An EEG channel selection module (1) is used to filter a small number of EEG channel signals through a channel selection method and then extract EEG features; and is used to obtain time-series speech features of speech signals through a speech signal coding network; A training and reasoning process module (2) is used to perform training and reasoning on EEG brain wave signals and speech signals; A multi-layer cross attention mechanism module (3) is used to simultaneously input the acquired speech features and the extracted EEG features into the multi-layer cross attention mechanism module for feature fusion; The cross attention unit module (4) is used to perform the fusion of speech features and EEG features, and obtain the fusion features after the speech features are cross-attended with the EEG features; The speech signal encoding network module (5) is used to decode the fused features into the enhanced speech of the target speaker.

9. The sparse channel based EEG-assisted speech enhancement system according to claim 8, characterized in that: The system is mounted on a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the functions of the above-mentioned EEG-assisted speech enhancement system based on sparse channels can be realized.

10. The sparse channel based EEG-assisted speech enhancement system according to claim 8, characterized in that: The system is mounted on a server, and the server is used to provide a user input interface to implement the functions of the sparse channel-based EEG-assisted speech enhancement system as described above when executed on an electronic device, and the electronic device is mounted on headphones.

Images