Speech enhancement method based on bidirectional gating neural network and reference noise

Abstract

The invention provides a speech enhancement method based on BGRU and reference noise, which comprises the following network structures: during reasoning, receiving two paths of input: reference noise and noisy speech, inputting the two paths of noise into an enhancement network module through a short-time Fourier transform encoder respectively to obtain noise estimation and speech estimation, performing inverse short-time Fourier transform on the voice estimation to obtain voice after noise reduction; during training, four paths of input including reference noise, noisy voice, noise truth value and voice truth value are received, the four paths of noise respectively pass through the short-time Fourier transform encoder, and the encoded reference noise and noisy voice are calculated through the enhancement network module to obtain noise estimation and voice estimation; and the encoded voice truth value, noise truth value, noise estimation and voice estimation are all sent to a 3CL loss module, loss is calculated and fed back to the enhanced network module, parameters are adjusted, and the performance of the enhanced network module is improved. According to the invention, the voice quality and intelligibility can be improved under the condition of extremely low signal-to-noise ratio.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech enhancement, and particularly relates to a speech enhancement method based on a bidirectional gated neural network and reference noise. Background Art

[0002] 1.1 Speech Enhancement

[0003] Speech Enhancement refers to the operation of processing low-quality speech signals to improve the signal perception quality and intelligibility, and is an important part of speech processing. In a broad sense, speech enhancement integrates technologies such as speech noise reduction, speech separation, and speech dereverberation. Through technical means, the damaged speech signal is repaired and denoised to improve the intelligibility and quality of the speech.

[0004] Currently, speech enhancement methods are mainly divided into two categories: methods based on digital signal processing and methods based on machine learning. Methods based on digital signal processing are divided into single-channel speech enhancement and multi-channel speech enhancement. Among them, single-channel speech enhancement mainly includes: spectral subtraction method, Wiener filtering method, method based on statistical model, adaptive filtering method, method based on subspace, method based on wavelet transform. Multi-channel speech enhancement mainly includes fixed beamforming and adaptive beamforming. Methods based on machine learning mainly include: speech enhancement based on hidden Markov model, speech enhancement based on non-negative matrix factorization, speech enhancement based on shallow neural network, speech enhancement based on deep neural network. The most widely used are the single-channel speech enhancement method based on digital signal processing and the deep neural network speech enhancement method based on machine learning.

[0005] 1.2 Speech Enhancement with Extremely Low Signal-to-Noise Ratio

[0006] The signal-to-noise ratio (SNR) is a parameter for measuring the signal quality, which represents the ratio of the signal power to the noise power in the noisy signal. The lower the SNR, the greater the interference to the signal and the greater the difficulty of signal enhancement; conversely, the higher the SNR, the smaller the interference to the signal and the smaller the difficulty of signal enhancement.

[0007] Generally, speech enhancement with an input signal SNR > 0 dB is called high signal-to-noise ratio speech enhancement; speech enhancement with an input signal -5 dB < SNR < 0 dB is called medium signal-to-noise ratio speech enhancement; speech enhancement with an input signal -15 dB < SNR < -5 dB is called low signal-to-noise ratio speech enhancement; speech enhancement with an input signal SNR < -15 dB is called speech enhancement with extremely low signal-to-noise ratio.

[0008] Although the speech enhancement method based on digital signal processing has a good enhancement effect under high and medium signal-to-noise ratio conditions, the enhancement effect drops rapidly for low signal-to-noise ratio input signals, and even almost cannot be enhanced. The speech enhancement method based on deep learning neural network can handle speech enhancement under low signal-to-noise ratio conditions, but it can only be used for specific noise types and is not as widely applicable as the speech enhancement method based on digital signal processing. However, so far, there is almost no mature method that can enhance noisy speech under extremely low signal-to-noise ratio conditions.

[0009] 1.3BGRU Network

[0010] Bilateral Gated Recurrent Unit (BGRU) is a recurrent neural network composed of two GRU units. It processes time series data from both the forward and reverse directions, and can capture the forward and backward information of time series data at the same time, which can better understand and predict time series. Audio information is a kind of time series, so it can be well processed by BGRU.

[0011] GRU is the basic unit of BGRU, which consists of update gate, reset gate, hidden state and candidate state.

[0012] in:

[0013] Update Gate:

[0014] z t =σ(W z ·[h t-1 , x t ]+b z )

[0015] Reset the gate:

[0016] r t =σ(W r ·[h t-1 , x t ]+b r )

[0017] Update candidate status:

[0018]

[0019] Update hidden state:

[0020]

[0021] 1.4 3CL loss

[0022] Component Loss (CL) is derived from the white-box testing method, which allows the performance of speech enhancement to be measured from three main aspects: damage to the pure speech component, removal of the noise component, and the naturalness of the residual noise (hence the name 3CL). Assume that the noise component of the noisy speech is D(k), the speech component is S(k), and the mask (Mask) of speech enhancement is M(k), and the noise component after filtering is obtained. Filtered speech component Final enhanced speech We can get:

[0023]

[0024] Damage to pure speech components:

[0025]

[0026] Removal of noise components:

[0027]

[0028] Naturalness of the participating noise:

[0029]

[0030] Considering the different importance of each part, add weight

[0031] α∈[0,1],β∈[0,1],α+β∈[0,1], the 3CL loss is:

[0032]

[0033] Among the existing speech enhancement technologies, the speech enhancement methods based on digital signal processing can only process noisy speech with a medium or higher input signal signal-to-noise ratio; the speech enhancement methods based on machine learning can only process noisy speech with a low or higher signal-to-noise ratio. There is no mature method to process noisy speech signals under extremely low signal-to-noise ratio conditions. In application scenarios such as drone sound collection and long-distance sound pickup, extremely low signal-to-noise ratio scenarios are very common. Therefore, the present invention proposes a speech enhancement method under extremely low signal-to-noise ratio conditions with reference noise based on a bidirectional gated neural network, which is used to improve speech quality and intelligibility under extremely low signal-to-noise ratio conditions. Summary of the invention

[0034] In view of this, the present invention aims to overcome the deficiencies of the above-mentioned problems in the prior art and proposes a speech enhancement method based on a bidirectional gated neural network and reference noise.

[0035] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0036] A first aspect of the present invention provides a speech enhancement method based on a bidirectional gated neural network and reference noise, comprising:

[0037] Construct a speech enhancement model, including a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module;

[0038] The speech enhancement model is trained. The input module receives four signals: the true value of noise, the true value of speech, the reference noise, and the noisy speech. The four noises are respectively passed through the short-time Fourier transform encoder. The encoded reference noise and the noisy speech are calculated through the enhancement network module to obtain the noise estimation and speech estimation. The encoded speech true value and noise true value as well as the noise estimation and speech estimation are sent to the 3CL loss module. After the loss is calculated, it is fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module.

[0039] The reference noise and noisy speech are used as input signals, and short-time Fourier transform operations are performed on them respectively and then input into the enhancement network module to obtain the speech estimation signal, which is then processed by inverse short-time Fourier transform to obtain the enhanced speech.

[0040] Furthermore, the enhanced network module divides the input signal into three paths for processing:

[0041] Path 1: The input encoded reference noise signal is input to the reference information extraction module after two-dimensional convolution and normalization operations. The reference information extraction module is repeated N times, and the extracted reference information is normalized after linear mapping to obtain the noise mask 1;

[0042] The second path: the encoded noisy speech signal is input into the noise extraction module after two-dimensional convolution and normalization operations. The extracted signal is normalized after linear mapping to obtain the noise mask 2;

[0043] The third path: The encoded noisy speech signal is input into the speech extraction module after two-dimensional convolution and normalization operations. The extracted signal is normalized after linear mapping to obtain the speech mask;

[0044] The memory information output by the reference information extraction module repeated N times is concatenated and convolved as the initial memory of the noise, and then input into the noise extraction module and the speech extraction module;

[0045] A noise mask is obtained by averaging noise mask 1 and noise mask 2, and the noise mask is multiplied by the encoded noisy speech to obtain a noise estimate;

[0046] The speech mask is multiplied by the noisy speech to obtain speech estimation 1, the encoded noisy speech is subtracted from the noise estimation to obtain speech estimation 2, and the speech estimation 1 and the speech estimation 2 are weightedly added to obtain the speech estimation;

[0047] The noise estimate and speech estimate are taken as the output of the enhancement network module.

[0048] Furthermore, the reference information extraction module includes four parts, wherein:

[0049] Part 1: It includes a bidirectional gated neural network layer and a linear layer. The input signal enters the bidirectional gated neural network layer, enters the linear layer after calculation, and is added to the input signal after calculation to obtain the intermediate signal x 1 ;

[0050] Part 2: Intermediate signal x 1 After passing through the bidirectional gated neural network layer, it enters the linear layer and obtains the intermediate signal x after calculation. 2 ;

[0051] Part 3: Intermediate signal x 2 After the self-attention mechanism and convolution layer, the intermediate signal x 2 Add together to get the intermediate signal x 3 ;

[0052] Part 4: Intermediate signal x 3 After the activation layer, the output signal is obtained after normalization;

[0053] In this process, the memory signals generated by the bidirectional gated neural network are spliced ​​to obtain the memory output.

[0054] Furthermore, the speech extraction module includes M+1 parts, wherein:

[0055] Part 1: The input signal enters the encoder module, and the result is divided into two paths, one for the input of the next part, and the other for adding the output of the next part and inputting it into the decoder. The result obtained after calculation is the output signal;

[0056] Part 2: The input signal is obtained from the previous part, and the result obtained by the encoder calculation is divided into two paths, one for the next part input, and the other is added with the output of the next part and input to the decoder, and after calculation, it is provided as output to the previous part;

[0057] The same structure is repeated until Part M;

[0058] Part M+1: The input signal from the Mth layer passes through the encoder, self-attention mechanism, and decoder in sequence to obtain the output of this layer and return to the Mth layer;

[0059] All encoders and decoders initialize their memories using externally provided memory signals.

[0060] Furthermore, the noise extraction module includes M+1 parts, wherein:

[0061] Part 1: The input signal enters the encoder module, and the result is divided into two paths, one for the input of the next part, and the other for adding the output of the next part and inputting it into the decoder. The result obtained after calculation is the output signal;

[0062] Part 2: The input signal is obtained from the previous part, and the result obtained by the encoder calculation is divided into two paths, one for the next part input, and the other is added with the output of the next part and input to the decoder, and after calculation, it is provided as output to the previous part;

[0063] The same structure is repeated until Part M;

[0064] Part M+1: The input signal from the Mth layer passes through the encoder and decoder in sequence to obtain the output of this layer, and then returns to the Mth layer;

[0065] All encoders and decoders initialize their memories using externally provided memory signals.

[0066] Furthermore, the structure of the encoder specifically includes: the input signal is sequentially passed through a bidirectional gated neural network module, a convolutional layer, an activation layer, and normalized to obtain an output, and the bidirectional gated neural network is initialized using an externally provided memory signal.

[0067] Furthermore, the structure of the decoder specifically includes: the input signal is sequentially passed through a bidirectional gated neural network module, a deconvolution layer, an activation layer, and normalized to obtain an output, and the bidirectional gated neural network is initialized using an externally provided memory signal.

[0068] A second aspect of the present invention provides a speech enhancement device based on a bidirectional gated neural network and reference noise, comprising:

[0069] A construction unit, for constructing a speech enhancement model, including a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, a decoder, and an output module;

[0070] The training unit is used to train the speech enhancement model. The input module receives four signals, namely, the true value of noise, the true value of speech, the reference noise and the noisy speech. The four noises are respectively passed through the short-time Fourier transform encoder. The encoded reference noise and the noisy speech are calculated through the enhancement network module to obtain the noise estimation and the speech estimation. The encoded speech true value and noise true value as well as the noise estimation and the speech estimation are all sent to the 3CL loss module. After the loss is calculated, it is fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module.

[0071] The enhancement unit is used to take the reference noise and the noisy speech as input signals, perform short-time Fourier transform operations on them respectively, and then input them into the enhancement network module to obtain a speech estimation signal, and obtain the enhanced speech after inverse short-time Fourier transform processing.

[0072] A third aspect of the present invention provides an electronic device, comprising a processor and a memory connected to the processor for storing executable instructions of the processor, wherein the processor is used to execute the above-mentioned speech enhancement method based on a bidirectional gated neural network and reference noise.

[0073] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned speech enhancement method based on a bidirectional gated neural network and reference noise.

[0074] Compared with the prior art, the speech enhancement method based on bidirectional gated neural network and reference noise described in the present invention has the following advantages:

[0075] The present invention introduces reference noise into the model of speech enhancement to solve the problem that the speech components and noise components in noisy speech cannot be effectively distinguished due to the extremely low signal-to-noise ratio of noisy speech. The method uses a reference information extraction module to extract the main information of the reference noise from the reference noise that is strongly correlated with the actual noise, and models the noise. By sharing memory, the memory of the noise in the reference information extraction module is passed to the speech extraction module and the noise extraction module; the speech extraction module extracts the speech components in the noisy speech more accurately based on the input memory of the noise; the noise extraction module extracts the noise part in the noisy speech more accurately based on the input memory of the noise. Finally, by fusing the reference noise information extracted by the reference information module, the noise components extracted by the noise extraction module, and the speech components extracted by the speech module, the enhancement of noisy speech in low signal-to-noise ratio scenarios is achieved, and the quality and intelligibility of the speech are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0077] Figure 1 It is a schematic diagram of the structure of the speech enhancement model of the present invention;

[0078] Figure 2 This is a schematic diagram of the structure of the enhanced network module of the present invention;

[0079] Figure 3 It is a schematic diagram of the structure of the reference information extraction module of the present invention;

[0080] Figure 4 It is a structural schematic diagram of the speech extraction module of the present invention;

[0081] Figure 5 It is a schematic diagram of the structure of the noise extraction module of the present invention;

[0082] Figure 6 A schematic diagram of an encoder of the present invention;

[0083] Figure 7 Schematic diagram of a decoder of the present invention. DETAILED DESCRIPTION

[0084] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0085] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0086] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0087] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0088] Embodiment 1:

[0089] The present invention proposes a speech enhancement method with reference noise based on BGRU. By collecting reference noise with strong correlation with actual noise, the method realizes extracting noise information from reference noise and noisy speech respectively and fusing them, enhancing noisy speech by fusing noise, and transferring the memory of noise extracted from reference noise to speech enhancement part, so as to realize the improvement of speech enhancement intelligibility and speech quality under low signal-to-noise ratio conditions.

[0090] The speech enhancement method with reference noise based on BGRU of the present invention specifically includes:

[0091] Construct a speech enhancement model, including a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module; Figure 1 As shown;

[0092] The model accepts four inputs:

[0093] Noise true value: the noise component contained in noisy speech used for speech enhancement;

[0094] Speech truth value: the speech component contained in noisy speech used for speech enhancement;

[0095] Reference noise: Acquisition noise that is strongly correlated with the true value of noise;

[0096] Noisy speech: Used to enhance the source signal for processing.

[0097] Due to the characteristics of deep learning algorithms, the working process of speech enhancement methods is divided into training and inference:

[0098] During training: Mainly used to train the enhancement network. The model accepts four signals: true noise value, true speech value, reference noise, and noisy speech. The true noise value and true speech value are the target signals for adjusting the enhancement network, and the reference noise and noisy speech are the input signals for the enhancement network to process. The reference noise and noisy speech are respectively processed by short-time Fourier transform and input into the enhancement network, and the noise estimate and speech estimate are calculated. The true noise value and the true speech value are respectively processed by short-time Fourier transform, and then input into the 3CL loss module with the noise estimate and the speech estimate, and the comprehensive errors such as the noise estimate and the true noise value, the speech estimate and the true speech value, and the residual noise are calculated, and the error is used to optimize the enhancement network. By using a large number of low-SNR noisy speech and its reference noise, true noise value, and true speech value to continuously train the enhancement network, it has a strong speech enhancement ability and can stably enhance extremely low SNR speech.

[0099] Inference: Mainly used for speech enhanced by using an enhancement network. The model accepts reference noise and noisy speech as input signals. The reference noise and noisy speech are respectively subjected to short-time Fourier transform operations and then input into the enhancement network to obtain a speech estimation signal, which is then subjected to inverse short-time Fourier transform processing to obtain the enhanced speech.

[0100] The above process is the overall training and reasoning process of the speech enhancement model. The following mainly describes the core of the algorithm: Enhanced network module, such as Figure 2 As shown:

[0101] The network is divided into three parts: reference part, enhancement part, and memory transfer.

[0102] Reference part: Extract noise information from reference noise. First, the input reference noise passes through the convolution layer and the normalization layer, and the result is input into the reference information extraction module. The reference information extraction module can be repeated N times according to the quality of the reference noise and the computing power of the hardware device. Finally, the noise mask 1 is obtained through the linear layer and the normalization layer.

[0103] The reference noise information extraction module structure is as follows: Figure 3 As shown, the module can be divided into three parts:

[0104] Part 1: It consists of a BGRU layer and a linear layer. The input signal enters the BGRU layer, enters the linear layer after calculation, and is added to the input signal after calculation to obtain the intermediate signal x 1 ;

[0105] Part 2: Intermediate signal x 1 After passing through the BGRU layer, it enters the linear layer and obtains the intermediate signal x after calculation. 2 ;

[0106] Part 3: Intermediate signal x2 After the self-attention mechanism and convolution layer, the intermediate signal x 2 Add together to get the intermediate signal x 3 ;

[0107] Intermediate signal x 3 The output is obtained after activation layer and normalization.

[0108] Enhancement part: Separate speech and noise from noisy speech.

[0109] For the noise separation part: the input noisy speech signal passes through the convolution layer and the normalization layer, and then is input into the noise extraction module. The result is passed through the linear layer and the normalization layer to obtain the noise mask 2.

[0110] The structure of the noise extraction module is as follows: Figure 5 As shown: The noise extraction module consists of M+1 parts, among which:

[0111] Part 1: The input signal enters the encoder module, and the result is divided into two paths. One path is left for the input of the next part, and the other path is added to the output of the next part and input to the decoder. The result obtained after calculation is the output signal.

[0112] Part 2: Similar to Part 1, the input signal is obtained from the previous part, and the result obtained by the encoder calculation is divided into two paths, one is left for the input of the next part, and the other is added to the output of the next part and input to the decoder. After calculation, it is provided to the previous part as output.

[0113] The same structure is repeated until Part M.

[0114] Part M+1: The input signal from the Mth layer passes through the encoder and decoder in sequence to obtain the output of this layer and return to the Mth layer.

[0115] For the speech separation part: the input noisy speech signal passes through the convolution layer and the normalization layer, and then is input into the speech extraction module. The result is passed through the linear layer and the normalization layer to obtain the speech mask 1.

[0116] The structure of the speech extraction module is as follows: Figure 4 As shown: The speech extraction module consists of M+1 parts, including:

[0117] Part 1: The input signal enters the encoder module, and the result is divided into two paths. One path is left for the input of the next part, and the other path is added with the output of the next part and input to the decoder. The result obtained after calculation is the output signal.

[0118] Part 2: Similar to Part 1, the input signal is obtained from the previous part, and the result obtained by the encoder calculation is divided into two paths, one is left for the input of the next part, and the other is added to the output of the next part and input to the decoder. After calculation, it is provided to the previous part as output.

[0119] The same structure is repeated until Part M.

[0120] Part M+1: The input signal from the Mth layer passes through the encoder, self-attention mechanism, and decoder in sequence to obtain the output of this layer and return to the Mth layer.

[0121] All the above encoders have the same structure, as Figure 6 As shown in the figure: the input signal passes through the BGRU module, convolution layer, activation layer, and normalization to obtain the output.

[0122] All the above decoders have the same structure, as Figure 7 As shown in the figure: the input signal passes through the BGRU module, deconvolution layer, activation layer, and normalization to obtain the output.

[0123] Fusion of the reference part and the enhancement part: The noise mask 1 and the noise mask 2 are averaged and multiplied with the noisy speech input to obtain the noise estimate; the speech mask 1 is multiplied with the noisy speech input to obtain the speech estimate 1; the noisy speech input is subtracted from the noise estimate to obtain the speech estimate 2; the speech estimate 2 is weightedly summed with the speech estimate 1 to obtain the speech estimate.

[0124] Memory transfer: In the reference information extraction module, the BGRU layer can generate memory of the reference noise. The memory information generated by all N reference noise extraction modules is concatenated and processed through the convolution layer to obtain the processed memory information of the noise, which is distributed to all encoders and decoders for their memory initialization.

[0125] The various modules work together to finally obtain a complete BGRU-based speech enhancement network with reference noise, which can achieve efficient enhancement of extremely low signal-to-noise ratio speech input in the presence of reference noise.

[0126] Embodiment 2:

[0127] A speech enhancement device based on a bidirectional gated neural network and reference noise, comprising:

[0128] A construction unit, for constructing a speech enhancement model, comprising a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module;

[0129] The training unit is used to train the speech enhancement model. The input module receives four signals, namely, the true value of noise, the true value of speech, the reference noise and the noisy speech. The four noises are respectively passed through the short-time Fourier transform encoder. The encoded reference noise and the noisy speech are calculated through the enhancement network module to obtain the noise estimation and the speech estimation. The encoded speech true value and noise true value as well as the noise estimation and the speech estimation are all sent to the 3CL loss module. After the loss is calculated, it is fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module.

[0130] The enhancement unit is used to take the reference noise and the noisy speech as input signals, perform short-time Fourier transform operations on them respectively, and then input them into the enhancement network module to obtain a speech estimation signal, and obtain the enhanced speech after inverse short-time Fourier transform processing.

[0131] Embodiment three:

[0132] An electronic device includes a processor and a memory connected to the processor for storing instructions executable by the processor, wherein the processor is used to execute the above-mentioned speech enhancement method based on a bidirectional gated neural network and reference noise.

[0133] Embodiment 4:

[0134] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for speech enhancement based on a bidirectional gated neural network and reference noise is implemented.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A speech enhancement method based on a bidirectional gated neural network and reference noise, characterized in that: include: Construct a speech enhancement model, including a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module; The speech enhancement model is trained. The input module receives four signals: the true value of noise, the true value of speech, the reference noise, and the noisy speech. The four noises are respectively passed through the short-time Fourier transform encoder. The encoded reference noise and the noisy speech are calculated through the enhancement network module to obtain the noise estimation and speech estimation. The encoded speech true value and noise true value as well as the noise estimation and speech estimation are sent to the 3CL loss module. After the loss is calculated, it is fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module. The reference noise and noisy speech are used as input signals, and short-time Fourier transform operations are performed on them respectively and then input into the enhancement network module to obtain the speech estimation signal, which is then processed by inverse short-time Fourier transform to obtain the enhanced speech.

2. The method for speech enhancement based on a bidirectional gated neural network and reference noise according to claim 1, characterized in that: The enhanced network module divides the input signal into three paths for processing: Path 1: The input encoded reference noise signal is input to the reference information extraction module after two-dimensional convolution and normalization operations. The reference information extraction module is repeated N times, and the extracted reference information is normalized after linear mapping to obtain the noise mask 1; The second path: the encoded noisy speech signal is input into the noise extraction module after two-dimensional convolution and normalization operations. The extracted signal is normalized after linear mapping to obtain the noise mask 2; The third path: The encoded noisy speech signal is input into the speech extraction module after two-dimensional convolution and normalization operations. The extracted signal is normalized after linear mapping to obtain the speech mask; The memory information output by the reference information extraction module repeated N times is concatenated and convolved as the initial memory of the noise, and then input into the noise extraction module and the speech extraction module; A noise mask is obtained by averaging noise mask 1 and noise mask 2, and the noise mask is multiplied by the encoded noisy speech to obtain a noise estimate; The speech mask is multiplied by the noisy speech to obtain speech estimation 1, the encoded noisy speech is subtracted from the noise estimation to obtain speech estimation 2, and the speech estimation 1 and the speech estimation 2 are weightedly added to obtain the speech estimation; The noise estimate and speech estimate are taken as the output of the enhancement network module.

3. The method for speech enhancement based on a bidirectional gated neural network and reference noise according to claim 2, characterized in that: The reference information extraction module includes four parts, among which: Part 1: includes a bidirectional gated neural network layer and a linear layer. The input signal enters the bidirectional gated neural network layer, enters the linear layer after calculation, and is added to the input signal after calculation to obtain the intermediate signal x1; Part 2: The intermediate signal x1 passes through the bidirectional gated neural network layer and then enters the linear layer, and after calculation, the intermediate signal x2 is obtained; Part 3: The intermediate signal x2 is added to the intermediate signal x2 after the self-attention mechanism and convolution layer to obtain the intermediate signal x3; Part 4: After the intermediate signal x3 passes through the activation layer, it is normalized to obtain the output signal; In this process, the memory signals generated by the bidirectional gated neural network are spliced ​​to obtain the memory output.

4. The method for speech enhancement based on a bidirectional gated neural network and reference noise according to claim 2, characterized in that: The speech extraction module includes M+1 parts, wherein: Part 1: The input signal enters the encoder module, and the result is divided into two paths, one for the input of the next part, and the other for adding the output of the next part and inputting it into the decoder. The result obtained after calculation is the output signal; Part 2: The input signal is obtained from the previous part, and the result obtained by the encoder calculation is divided into two paths, one for the next part input, and the other is added with the output of the next part and input to the decoder, and after calculation, it is provided as output to the previous part; The same structure is repeated until Part M; Part M+1: The input signal from the Mth layer passes through the encoder, self-attention mechanism, and decoder in sequence to obtain the output of this layer and return to the Mth layer; All encoders and decoders initialize their memories using externally provided memory signals.

5. The method for speech enhancement based on bidirectional gated neural network and reference noise according to claim 2, characterized in that: The noise extraction module includes M+1 parts, wherein: Part 1: The input signal enters the encoder module, and the result is divided into two paths, one for the input of the next part, and the other for adding the output of the next part and inputting it into the decoder. The result obtained after calculation is the output signal; Part 2: The input signal is obtained from the previous part, and the result obtained by the encoder calculation is divided into two paths, one for the next part input, and the other is added with the output of the next part and input to the decoder, and after calculation, it is provided as output to the previous part; The same structure is repeated until Part M; Part M+1: The input signal from the Mth layer passes through the encoder and decoder in sequence to obtain the output of this layer, and then returns to the Mth layer; All encoders and decoders initialize their memories using externally provided memory signals.

6. A method for speech enhancement based on a bidirectional gated neural network and reference noise according to any one of claims 4 or 5, characterized in that: The structure of the encoder specifically includes: the input signal is sequentially passed through a bidirectional gated neural network module, a convolutional layer, an activation layer, and normalized to obtain an output, and the bidirectional gated neural network is initialized using an externally provided memory signal.

7. A method for speech enhancement based on a bidirectional gated neural network and reference noise according to any one of claims 4 or 5, characterized in that: The structure of the decoder specifically includes: the input signal is sequentially passed through a bidirectional gated neural network module, a deconvolution layer, an activation layer, and normalized to obtain an output, and the bidirectional gated neural network is initialized using an externally provided memory signal.

8. A speech enhancement device based on a bidirectional gated neural network and reference noise, characterized in that: include: A construction unit, for constructing a speech enhancement model, comprising a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module; The training unit is used to train the speech enhancement model. The input module receives four signals, namely, the true value of noise, the true value of speech, the reference noise and the noisy speech. The four noises are respectively passed through the short-time Fourier transform encoder. The encoded reference noise and the noisy speech are calculated through the enhancement network module to obtain the noise estimation and the speech estimation. The encoded speech true value and noise true value as well as the noise estimation and the speech estimation are all sent to the 3CL loss module. After the loss is calculated, it is fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module. The enhancement unit is used to take the reference noise and the noisy speech as input signals, perform short-time Fourier transform operations on them respectively, and then input them into the enhancement network module to obtain a speech estimation signal, and obtain the enhanced speech after inverse short-time Fourier transform processing.

9. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute a speech enhancement method based on a bidirectional gated neural network and reference noise as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech enhancement method based on a bidirectional gated neural network and reference noise described in any one of claims 1 to 7 is implemented.

Images