A speech enhancement method based on bidirectional gated neural network and reference noise

Abstract

The application provides a speech enhancement method based on BGRU and reference noise, and contains the following network structure: during reasoning, two-way input is accepted: reference noise and noisy speech, the two-way noise is respectively input to an enhancement network module through a short-time Fourier transform encoder to obtain noise estimation and speech estimation, and the speech estimation is input to an inverse short-time Fourier transform to obtain the speech after noise reduction; during training, four-way input is accepted: reference noise, noisy speech, noise true value and speech true value, the four-way noise is respectively input to a short-time Fourier transform encoder, the encoded reference noise and noisy speech are calculated through the enhancement network module to obtain noise estimation and speech estimation; the encoded speech true value and noise true value, and the noise estimation and speech estimation are all input to a 3CL loss module, the loss is calculated and fed back to the enhancement network module to adjust the parameters and improve the performance of the enhancement network module. The application can improve the speech quality and intelligibility under the condition of extremely low signal-to-noise ratio.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech enhancement, and in particular, relates to a speech enhancement method based on a bidirectional gated neural network and reference noise. Background Art

[0002] 1.1 Speech Enhancement

[0003] Speech Enhancement refers to the operation of processing low-quality speech signals to improve the signal perception quality and intelligibility, and is an important part of speech processing. Broadly speaking, speech enhancement integrates technologies such as speech noise reduction, speech separation, and speech dereverberation. Through technical means, the damaged speech signal is repaired and denoised to improve the intelligibility and quality of speech.

[0004] Currently, speech enhancement methods are mainly divided into two categories: methods based on digital signal processing and methods based on machine learning. Methods based on digital signal processing are divided into single-channel speech enhancement and multi-channel speech enhancement. Among them, single-channel speech enhancement mainly includes: spectral subtraction method, Wiener filtering method, method based on statistical model, adaptive filtering method, subspace-based method, wavelet transform-based method. Multi-channel speech enhancement mainly includes fixed beamforming and adaptive beamforming. Methods based on machine learning mainly include: speech enhancement based on hidden Markov model, speech enhancement based on non-negative matrix factorization, speech enhancement based on shallow neural network, speech enhancement based on deep neural network. The most widely used are the single-channel speech enhancement method based on digital signal processing and the deep neural network speech enhancement method based on machine learning.

[0005] 1.2 Speech Enhancement with Extremely Low Signal-to-Noise Ratio

[0006] The signal-to-noise ratio (SNR) is a parameter for measuring the signal quality, which represents the ratio of the signal power to the noise power in the noisy signal. The lower the SNR, the greater the interference to the signal and the greater the difficulty of signal enhancement; on the contrary, the higher the SNR, the smaller the interference to the signal and the smaller the difficulty of signal enhancement.

[0007] Generally, speech enhancement with an input signal SNR > 0 dB is called high SNR speech enhancement; speech enhancement with an input signal -5 dB < SNR < 0 dB is called medium SNR speech enhancement; speech enhancement with an input signal -15 dB < SNR < -5 dB is called low SNR speech enhancement; speech enhancement with an input signal SNR < -15 dB is called speech enhancement with extremely low SNR.

[0008] While digital signal processing-based speech enhancement methods perform well under high and medium signal-to-noise ratio (SNR) conditions, their effectiveness drops rapidly, sometimes even becoming negligible, for low SNR input signals. Deep learning neural network-based speech enhancement methods can handle low SNR conditions, but they are only applicable to specific noise types and lack the broad applicability of digital signal processing-based methods. To date, however, there is almost no mature method capable of enhancing noisy speech under extremely low SNR conditions.

[0009] 1.3BGRU Network

[0010] A Bilaterally Gated Recurrent Unit (BGRU) is a type of recurrent neural network consisting of two GRU units. It processes time-series data from both forward and backward directions, simultaneously capturing both forward and backward information, thus enabling better understanding and prediction of time series data. Audio information is a type of time series, and therefore can be well processed by BGRU.

[0011] GRU is the basic unit of BGRU, consisting of update gates, reset gates, hidden states, and candidate states. Among them:

[0012] Update Gate:

[0013]

[0014] Reset Door:

[0015]

[0016] Update candidate status:

[0017]

[0018] Update hidden status:

[0019]

[0020] 1.43CL loss

[0021] Component Loss (CL), derived from white-box testing methods, allows for the measurement of speech enhancement performance from three main aspects: damage to clean speech components, removal of noise components, and naturalness of residual noise (hence the term 3CL). Assuming the noise components of noisy speech are... Speech components Voice enhancement mask The filtered noise components are obtained. Filtered speech components Ultimately enhanced voice We can obtain:

[0022]

[0023] Impairment of clean speech components:

[0024] ;

[0025] Noise component removal:

[0026] ;

[0027] The naturalness of the noise:

[0028] ;

[0029] Considering the different levels of importance of each part, weights are added.

[0030] The 3CL loss is obtained as follows:

[0031] .

[0032] Existing speech enhancement technologies, such as digital signal processing-based methods, can only handle noisy speech with a medium to high signal-to-noise ratio (SNR); machine learning-based methods can only handle noisy speech with a low SNR or higher. There is no mature method for handling noisy speech signals under extremely low SNR conditions. However, extremely low SNR scenarios are very common in applications such as drone sound acquisition and long-distance sound pickup. Therefore, this invention proposes a speech enhancement method with reference noise based on a bidirectional gated neural network for extremely low SNR conditions, aiming to improve speech quality and intelligibility under these conditions. Summary of the Invention

[0033] In view of this, the present invention aims to overcome the shortcomings of the above-mentioned problems in the prior art and proposes a speech enhancement method based on bidirectional gated neural network and reference noise.

[0034] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0035] The first aspect of this invention provides a speech enhancement method based on a bidirectional gated neural network and reference noise, comprising:

[0036] A speech enhancement model is constructed, including a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module;

[0037] The speech enhancement model is trained by an input module that receives four signals: noise ground truth, speech ground truth, reference noise, and noisy speech. The four noise signals are respectively processed by a short-time Fourier transform encoder. The encoded reference noise and noisy speech are processed by an enhancement network module to obtain noise estimates and speech estimates. The encoded speech ground truth, noise ground truth, noise estimates, and speech estimates are all fed into a 3CL loss module. The loss is calculated and fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module.

[0038] The reference noise and noisy speech are used as input signals. After performing short-time Fourier transform operations on each signal, they are input into the enhancement network module to obtain the speech estimation signal. After inverse short-time Fourier transform processing, the enhanced speech is obtained.

[0039] Furthermore, the enhanced network module processes the input signal by splitting it into three paths:

[0040] Path 1: The input encoded reference noise signal is processed by two-dimensional convolution and normalization operations and then input into the reference information extraction module. The reference information extraction module is repeated N times. The extracted reference information is normalized after linear mapping to obtain noise mask 1.

[0041] Channel 2: The input encoded noisy speech signal is processed by two-dimensional convolution and normalization operations and then input to the noise extraction module. The extracted signal is normalized after linear mapping to obtain noise mask 2.

[0042] Path 3: The input encoded noisy speech signal is processed by two-dimensional convolution and normalization operations and then input into the speech extraction module. The extracted signal is normalized after linear mapping to obtain the speech mask.

[0043] The memory information output by the reference information extraction module, which is repeated N times, is concatenated and convolved to serve as the initial memory of the noise, and then input into the noise extraction module and the speech extraction module.

[0044] The noise mask is obtained by averaging noise mask 1 and noise mask 2. The noise mask is then multiplied by the encoded noisy speech to obtain the noise estimate.

[0045] Speech estimation 1 is obtained by multiplying the speech mask by the noisy speech, speech estimation 2 is obtained by subtracting the noise estimation from the encoded noisy speech, and speech estimation 1 and speech estimation 2 are weighted and added together to obtain the speech estimation.

[0046] Noise estimation and speech estimation are outputs of the enhancement network module.

[0047] Furthermore, the reference information extraction module comprises four parts, wherein:

[0048] Part 1: Includes a bidirectional gated neural network layer and a linear layer. The input signal enters the bidirectional gated neural network layer, is calculated, and then enters the linear layer. After calculation, the linear layer adds the linear signal to the input signal to obtain the intermediate signal x1.

[0049] Part 2: The intermediate signal x1 passes through a bidirectional gated neural network layer and then enters a linear layer. After calculation, the intermediate signal x2 is obtained.

[0050] Part 3: The intermediate signal x2 is added to the intermediate signal x3 after passing through the self-attention mechanism and the convolutional layer;

[0051] Part 4: After the intermediate signal x3 passes through the activation layer, it is normalized to obtain the output signal;

[0052] In this process, the memory signals generated by the bidirectional gated neural network are spliced ​​together to obtain the memory output.

[0053] Furthermore, the speech extraction module comprises M+1 parts, wherein:

[0054] Part 1: The input signal enters the encoder module, and the result is divided into two paths. One path is used as the input for the next part, and the other path is added to the output of the next part and input to the decoder. The result obtained after calculation is the output signal.

[0055] Part 2: The input signal is obtained from the previous part. The result calculated by the encoder is divided into two paths. One path is left as the input to the next part, and the other path is added to the output of the next part and input to the decoder. After calculation, it is used as the output to provide to the previous part.

[0056] The same structure is repeated up to part M;

[0057] Part M+1: The input signal obtained from the Mth layer passes through the encoder, self-attention mechanism, and decoder in sequence to obtain the output of this layer, which is then returned to the Mth layer;

[0058] All encoders and decoders initialize their memory using externally provided memory signals.

[0059] Furthermore, the noise extraction module comprises M+1 parts, wherein:

[0060] Part 1: The input signal enters the encoder module, and the result is divided into two paths. One path is used as the input for the next part, and the other path is added to the output of the next part and input to the decoder. The result obtained after calculation is the output signal.

[0061] Part 2: The input signal is obtained from the previous part. The result calculated by the encoder is divided into two paths. One path is left as the input to the next part, and the other path is added to the output of the next part and input to the decoder. After calculation, it is used as the output to provide to the previous part.

[0062] The same structure is repeated up to part M;

[0063] Part M+1: The input signal obtained from the Mth layer passes through the encoder and decoder in sequence to obtain the output of this layer, which is then returned to the Mth layer;

[0064] All encoders and decoders initialize their memory using externally provided memory signals.

[0065] Furthermore, the encoder's structure specifically includes: the input signal sequentially passes through a bidirectional gated neural network module, a convolutional layer, an activation layer, and normalization to obtain the output; simultaneously, the bidirectional gated neural network is initialized using an externally provided memory signal.

[0066] Furthermore, the structure of the decoder specifically includes: the input signal sequentially passes through a bidirectional gated neural network module, a deconvolution layer, an activation layer, and normalization to obtain the output, while the bidirectional gated neural network is initialized using an externally provided memory signal.

[0067] A second aspect of the present invention provides a speech enhancement device based on a bidirectional gated neural network and reference noise, comprising:

[0068] The building unit is used to build the speech enhancement model, including a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, a decoder, and an output module;

[0069] The training unit is used to train the speech enhancement model. The input module receives four signals: noise ground truth, speech ground truth, reference noise, and noisy speech. The four noise signals are respectively processed by a short-time Fourier transform encoder. The encoded reference noise and noisy speech are processed by the enhancement network module to obtain noise estimation and speech estimation. The encoded speech ground truth, noise ground truth, noise estimation, and speech estimation are all sent to the 3CL loss module. After the loss is calculated, it is fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module.

[0070] The enhancement unit takes the reference noise and noisy speech as input signals, performs short-time Fourier transform operations on them respectively, and then inputs them into the enhancement network module to obtain the speech estimation signal. After inverse short-time Fourier transform processing, the enhanced speech is obtained.

[0071] A third aspect of the present invention provides an electronic device, including a processor and a memory communicatively connected to the processor and used to store processor-executable instructions, wherein the processor is used to execute the above-described speech enhancement method based on a bidirectional gated neural network and reference noise.

[0072] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described speech enhancement method based on a bidirectional gated neural network and reference noise.

[0073] Compared with existing technologies, the speech enhancement method based on bidirectional gated neural networks and reference noise described in this invention has the following advantages:

[0074] This invention introduces reference noise into the speech enhancement model to address the problem of ineffective identification of speech and noise components in noisy speech due to its extremely low signal-to-noise ratio. The method employs a reference information extraction module to extract key information from reference noise strongly correlated with the actual noise, and models the noise. Through a shared memory mechanism, the noise information extracted by the reference information extraction module is transferred to the speech extraction and noise extraction modules. The speech extraction module, based on its input noise memory, extracts speech components from the noisy speech more accurately; similarly, the noise extraction module, based on its input noise memory, extracts noise components from the noisy speech more accurately. Finally, by fusing the reference noise information extracted by the reference information module, the noise components extracted by the noise extraction module, and the speech components extracted by the speech module, noisy speech in scenarios with extremely low signal-to-noise ratios is enhanced, improving speech quality and intelligibility. Attached Figure Description

[0075] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0076] Figure 1 This is a schematic diagram of the speech enhancement model structure of the present invention;

[0077] Figure 2 This is a schematic diagram of the enhanced network module structure of the present invention;

[0078] Figure 3 This is a schematic diagram of the reference information extraction module structure of the present invention;

[0079] Figure 4 This is a schematic diagram of the speech extraction module structure of the present invention;

[0080] Figure 5 This is a schematic diagram of the noise extraction module structure of the present invention;

[0081] Figure 6 This is a schematic diagram of the encoder of the present invention;

[0082] Figure 7 This is a schematic diagram of the decoder of the present invention. Detailed Implementation

[0083] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0084] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0085] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0086] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0087] Example 1:

[0088] This invention proposes a speech enhancement method with reference noise based on BGRU. By collecting reference noise that is strongly correlated with the actual noise, the method extracts and fuses noise information from the reference noise and the noisy speech, enhances the noisy speech by fusing the noise, and transfers the memory of noise extracted from the reference noise to the speech enhancement part. This method aims to improve the intelligibility and speech quality of speech enhancement under extremely low signal-to-noise ratio conditions.

[0089] The speech enhancement method with reference noise based on BGRU of the present invention specifically includes:

[0090] A speech enhancement model is constructed, including a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module; such as Figure 1 As shown;

[0091] The model accepts four inputs:

[0092] Noise Truth Value: The noise component contained in noisy speech used for speech enhancement;

[0093] Speech truth: The speech components contained in noisy speech used for speech enhancement;

[0094] Reference noise: Acquisition noise that is strongly correlated with the true noise value;

[0095] Noisy speech: The source signal used to enhance processing.

[0096] Due to the characteristics of deep learning algorithms, the working process of speech enhancement methods can be divided into training and inference:

[0097] During training: Primarily used to train the enhancement network. The model accepts four signals: ground truth noise, ground truth speech, reference noise, and noisy speech. The ground truth noise and ground truth speech are the target signals for adjusting the enhancement network, while the reference noise and noisy speech are the input signals used by the enhancement network. The reference noise and noisy speech are processed by Short-Time Fourier Transform (SFT) and then input into the enhancement network to calculate noise estimates and speech estimates. The ground truth noise and ground truth speech are then processed by SFT and input into the 3CL loss module to calculate the combined errors of noise estimate versus ground truth noise, speech estimate versus ground truth speech, and residual noise. This error is used to optimize the enhancement network. By continuously training the enhancement network with a large amount of noisy speech with extremely low signal-to-noise ratio (SNR) along with its reference noise, ground truth noise, and ground truth speech, it develops strong speech enhancement capabilities and can stably enhance speech with extremely low SNR.

[0098] During inference: Primarily used to obtain enhanced speech using an enhancement network. The model accepts reference noise and noisy speech as input signals. The reference noise and noisy speech are each subjected to a short-time Fourier transform and then input into the enhancement network to obtain the speech estimation signal. After inverse short-time Fourier transform processing, the enhanced speech is obtained.

[0099] The above process describes the overall training and inference process of the speech enhancement model. The following mainly describes the core of the algorithm: the enhancement network module, such as... Figure 2 As shown:

[0100] The network consists of three parts: the reference part, the reinforcement part, and the memory transfer part.

[0101] Reference section: Extracting noise information from reference noise. First, the input reference noise is passed through a convolutional layer and a normalization layer. The result is then input into the reference information extraction module. Depending on the quality of the reference noise and the computing power of the hardware, the reference information extraction module can be repeated N times. Finally, the noise mask 1 is obtained by passing the noise through a linear layer and a normalization layer.

[0102] The structure of the reference noise information extraction module is as follows: Figure 3 As shown, the module can be divided into three parts:

[0103] Part 1: Composed of a BGRU layer and a linear layer. The input signal enters the BGRU layer, is calculated, and then enters the linear layer. After calculation, the linear layer adds the linear signal to the input signal to obtain the intermediate signal x1.

[0104] Part 2: The intermediate signal x1 passes through the BGRU layer and enters the linear layer, where it is calculated to obtain the intermediate signal x2.

[0105] Part 3: The intermediate signal x2 is added to the intermediate signal x3 after passing through the self-attention mechanism and the convolutional layer;

[0106] The intermediate signal x3 is output after passing through the activation layer and normalization.

[0107] Enhancement section: Separate speech and noise from noisy speech.

[0108] For the noise separation part: the input noisy speech signal is passed through a convolutional layer and a normalization layer, and then input into the noise extraction module. The result is passed through a linear layer and a normalization layer to obtain the noise mask 2.

[0109] The structure of the noise extraction module is as follows: Figure 5 As shown: The noise extraction module consists of M+1 parts, among which:

[0110] Part 1: The input signal enters the encoder module, and the result is divided into two paths. One path is used as the input for the next part, and the other path is added to the output of the next part and input to the decoder. The result obtained after calculation is the output signal.

[0111] Part 2: Similar to Part 1, the input signal is obtained from the previous part, and the result calculated by the encoder is divided into two paths. One path is left as the input to the next part, and the other path is added to the output of the next part and input to the decoder. After calculation, it is provided as the output to the previous part.

[0112] The same structure is repeated up to part M.

[0113] Part M+1: The input signal obtained from the Mth layer passes through the encoder and decoder in sequence to obtain the output of the layer, which is then returned to the Mth layer.

[0114] For the speech separation part: the input noisy speech signal is passed through a convolutional layer and a normalization layer, and then input into the speech extraction module. The result is passed through a linear layer and a normalization layer to obtain speech mask 1.

[0115] The structure of the speech extraction module is as follows: Figure 4 As shown: The speech extraction module consists of M+1 parts, among which:

[0116] Part 1: The input signal enters the encoder module, and the result is divided into two paths. One path is used as the input for the next part, and the other path is added to the output of the next part and input to the decoder. The result obtained after calculation is the output signal.

[0117] Part 2: Similar to Part 1, the input signal is obtained from the previous part, and the result calculated by the encoder is divided into two paths. One path is left as the input to the next part, and the other path is added to the output of the next part and input to the decoder. After calculation, it is provided as the output to the previous part.

[0118] The same structure is repeated up to part M.

[0119] Part M+1: The input signal obtained from the Mth layer passes through the encoder, self-attention mechanism, and decoder in sequence to obtain the output of this layer, which is then returned to the Mth layer.

[0120] All the encoders mentioned above have the same structure, as shown below. Figure 6 As shown: The input signal passes through the BGRU module, convolutional layer, activation layer, and normalization layer in sequence to obtain the output.

[0121] All the decoders mentioned above have the same structure, as follows: Figure 7 As shown: The input signal passes through the BGRU module, deconvolution layer, activation layer, and normalization layer in sequence to obtain the output.

[0122] Fusion of the reference part and the enhancement part: The noise estimate is obtained by averaging noise mask 1 and noise mask 2 and multiplying it with the noisy speech input; the speech estimate 1 is obtained by multiplying speech mask 1 with the noisy speech input; the speech estimate 2 is obtained by subtracting the noise estimate from the noisy speech input; the speech estimate is obtained by weighted summation of speech estimate 2 and speech estimate 1.

[0123] Memory transfer: In the reference information extraction module, the BGRU layer can generate a memory of the reference noise. The memory information generated by all N reference noise extraction modules is concatenated and processed by the convolutional layer to obtain the processed memory information of the noise, which is then distributed to all encoders and decoders for their memory initialization.

[0124] The various modules work together to obtain a complete BGRU-based speech enhancement network with reference noise, which achieves efficient enhancement of speech input with extremely low signal-to-noise ratio even with reference noise.

[0125] Example 2:

[0126] A speech enhancement device based on a bidirectional gated neural network and reference noise, comprising:

[0127] The building unit, used to construct the speech enhancement model, includes a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module;

[0128] The training unit is used to train the speech enhancement model. The input module receives four signals: noise ground truth, speech ground truth, reference noise, and noisy speech. The four noise signals are respectively processed by a short-time Fourier transform encoder. The encoded reference noise and noisy speech are processed by the enhancement network module to obtain noise estimation and speech estimation. The encoded speech ground truth, noise ground truth, noise estimation, and speech estimation are all sent to the 3CL loss module. After the loss is calculated, it is fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module.

[0129] The enhancement unit takes the reference noise and noisy speech as input signals, performs short-time Fourier transform operations on them respectively, and then inputs them into the enhancement network module to obtain the speech estimation signal. After inverse short-time Fourier transform processing, the enhanced speech is obtained.

[0130] Example 3:

[0131] An electronic device includes a processor and a memory communicatively connected to the processor and used to store processor-executable instructions, the processor being used to execute the aforementioned speech enhancement method based on a bidirectional gated neural network and reference noise.

[0132] Example 4:

[0133] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned speech enhancement method based on a bidirectional gated neural network and reference noise.

[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A speech enhancement method based on a bidirectional gated neural network and reference noise, characterized in that: include: A speech enhancement model is constructed, including a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module; The speech enhancement model is trained by an input module that receives four signals: ground truth noise, ground truth speech, reference noise, and noisy speech. These four signals are each processed by a short-time Fourier transform encoder. The encoded reference noise and noisy speech are then processed by an enhancement network module, which divides the input signal into three paths for further processing. Path 1: The input encoded reference noise signal, after undergoing two-dimensional convolution and normalization operations, is input to the reference information extraction module. This process is repeated N times, and the extracted reference information is normalized after linear mapping to obtain noise mask 1. Path 2: The input encoded noisy speech signal, after undergoing two-dimensional convolution and normalization operations, is input to the noise extraction module. The extracted signal is normalized after linear mapping to obtain noise mask 2. Path 3: The input encoded noisy speech signal, after undergoing two-dimensional convolution and normalization operations, is input to the speech extraction module. The extracted signal is normalized after linear mapping to obtain a speech mask. The memory signal output from the reference information extraction module, repeated N times, is then processed. The noise information, after being concatenated and convolved, serves as the initial memory for the noise and is input into the noise extraction module and the speech extraction module. The noise mask is obtained by averaging noise mask 1 and noise mask 2, and then multiplied by the encoded noisy speech to obtain a noise estimate. The speech mask is then multiplied by the noisy speech to obtain speech estimate 1, and the encoded noisy speech is subtracted from the noise estimate to obtain speech estimate 2. Speech estimate 1 and speech estimate 2 are then weighted and summed to obtain the final speech estimate. The noise estimate and speech estimate serve as the outputs of the enhancement network module. The encoded speech ground truth, noise ground truth, noise estimate, and speech estimate are all fed into the 3CL loss module, where the loss is calculated and fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module. The reference noise and noisy speech are used as input signals. After performing short-time Fourier transform operations on each signal, they are input into the enhancement network module to obtain the speech estimation signal. After inverse short-time Fourier transform processing, the enhanced speech is obtained.

2. The speech enhancement method based on a bidirectional gated neural network and reference noise according to claim 1, characterized in that: The reference information extraction module comprises four parts, including: Part 1: Includes a bidirectional gated neural network layer and a linear layer. The input signal enters the bidirectional gated neural network layer, is calculated, and then enters the linear layer. After calculation, the linear layer adds the linear signal to the input signal to obtain the intermediate signal x1. Part 2: The intermediate signal x1 passes through a bidirectional gated neural network layer and then enters a linear layer. After calculation, the intermediate signal x2 is obtained. Part 3: The intermediate signal x2 is added to the intermediate signal x3 after passing through the self-attention mechanism and the convolutional layer; Part 4: After the intermediate signal x3 passes through the activation layer, it is normalized to obtain the output signal; In this process, the memory signals generated by the bidirectional gated neural network are spliced ​​together to obtain the memory output.

3. The speech enhancement method based on a bidirectional gated neural network and reference noise according to claim 1, characterized in that: The speech extraction module comprises M+1 parts, wherein: Part 1: The input signal enters the encoder module, and the result is divided into two paths. One path is used as the input for the next part, and the other path is added to the output of the next part and input to the decoder. The result obtained after calculation is the output signal. Part 2: The input signal is obtained from the previous part. The result calculated by the encoder is divided into two paths. One path is left as the input to the next part, and the other path is added to the output of the next part and input to the decoder. After calculation, it is used as the output to provide to the previous part. The same structure is repeated up to part M; Part M+1: The input signal obtained from the Mth layer passes through the encoder, self-attention mechanism, and decoder in sequence to obtain the output of this layer, which is then returned to the Mth layer; All encoders and decoders initialize their memory using externally provided memory signals.

4. The speech enhancement method based on a bidirectional gated neural network and reference noise according to claim 1, characterized in that: The noise extraction module comprises M+1 parts, wherein: Part 1: The input signal enters the encoder module, and the result is divided into two paths. One path is used as the input for the next part, and the other path is added to the output of the next part and input to the decoder. The result obtained after calculation is the output signal. Part 2: The input signal is obtained from the previous part. The result calculated by the encoder is divided into two paths. One path is left as the input to the next part, and the other path is added to the output of the next part and input to the decoder. After calculation, it is used as the output to provide to the previous part. The same structure is repeated up to part M; Part M+1: The input signal obtained from the Mth layer passes through the encoder and decoder in sequence to obtain the output of this layer, which is then returned to the Mth layer; All encoders and decoders initialize their memory using externally provided memory signals.

5. A speech enhancement method based on a bidirectional gated neural network and reference noise according to any one of claims 3 or 4, characterized in that: The encoder's structure specifically includes: the input signal sequentially passes through a bidirectional gated neural network module, a convolutional layer, an activation layer, and normalization to obtain the output; at the same time, the bidirectional gated neural network is initialized using an externally provided memory signal.

6. A speech enhancement method based on a bidirectional gated neural network and reference noise according to any one of claims 3 or 4, characterized in that: The decoder's structure specifically includes: the input signal sequentially passes through a bidirectional gated neural network module, a deconvolution layer, an activation layer, and normalization to obtain the output; at the same time, the bidirectional gated neural network is initialized using an externally provided memory signal.

7. A speech enhancement device based on a bidirectional gated neural network and reference noise, characterized in that: include: The building unit, used to construct the speech enhancement model, includes a 4-way input module, a short-time Fourier transform encoder, a 3CL loss module, an enhancement network module, an inverse short-time Fourier transform decoder, and an output module; The training unit is used to train the speech enhancement model. The input module receives four signals: the ground truth noise value, the ground truth speech value, the reference noise, and the noisy speech. Each of the four signals is processed by a short-time Fourier transform encoder. The encoded reference noise and noisy speech are then processed by the enhancement network module, which divides the input signal into three paths for further processing. Path 1: The input encoded reference noise signal, after undergoing two-dimensional convolution and normalization operations, is input to the reference information extraction module. This process is repeated N times, and the extracted reference information is normalized after linear mapping to obtain noise mask 1. Path 2: The input encoded noisy speech signal, after undergoing two-dimensional convolution and normalization operations, is input to the noise extraction module. The extracted signal is normalized after linear mapping to obtain noise mask 2. Path 3: The input encoded noisy speech signal, after undergoing two-dimensional convolution and normalization operations, is input to the speech extraction module. The extracted signal is normalized after linear mapping to obtain a speech mask. The memory signal output from the reference information extraction module, repeated N times, is then processed. The noise information, after being concatenated and convolved, serves as the initial memory for the noise and is input into the noise extraction module and the speech extraction module. The noise mask is obtained by averaging noise mask 1 and noise mask 2, and then multiplied by the encoded noisy speech to obtain a noise estimate. The speech mask is then multiplied by the noisy speech to obtain speech estimate 1, and the encoded noisy speech is subtracted from the noise estimate to obtain speech estimate 2. Speech estimate 1 and speech estimate 2 are then weighted and summed to obtain the final speech estimate. The noise estimate and speech estimate serve as the outputs of the enhancement network module. The encoded speech ground truth, noise ground truth, noise estimate, and speech estimate are all fed into the 3CL loss module, where the loss is calculated and fed back to the enhancement network module to adjust its parameters and optimize the enhancement network module. The enhancement unit takes the reference noise and noisy speech as input signals, performs short-time Fourier transform operations on them respectively, and then inputs them into the enhancement network module to obtain the speech estimation signal. After inverse short-time Fourier transform processing, the enhanced speech is obtained.

8. An electronic device, comprising a processor and a memory communicatively connected to the processor and used to store instructions executed by the processor, characterized in that: The processor is used to execute the speech enhancement method based on bidirectional gated neural network and reference noise as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the speech enhancement method based on bidirectional gated neural network and reference noise as described in any one of claims 1-6.

Images