A two-stage speech noise reduction method, device, computer equipment, computer-readable storage medium and computer program product based on amplitude spectrum and complex spectrum

Abstract

The present application relates to the field of speech processing technology and discloses a two-stage speech denoising method, apparatus, computer device, computer-readable storage medium and computer program product based on amplitude spectrum and complex spectrum. The method comprises performing a preprocessing operation on the acquired original speech signal and dividing it into a number of time frames according to a preset interval; determining the amplitude, phase information and complex spectrum of each different frequency component; using the amplitude of each different frequency component to perform noise estimation, performing preliminary noise suppression on the original noisy speech to obtain a preliminary denoised amplitude spectrum; based on the preliminary denoised amplitude spectrum, combined with the phase information, converting it into a preliminary denoised complex spectrum; using the complex spectrum of the original noisy speech and the preliminary denoised complex spectrum to perform noise estimation, and converting it into a complex spectrum of the target speech signal. The present application has the effect of improving the denoising accuracy when processing noisy speech with a low signal-to-noise ratio or a mixture of multiple types of noise.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a two-stage speech noise reduction method, apparatus, computer equipment, computer-readable storage medium, and computer program product based on amplitude spectrum and complex spectrum. Background Art

[0002] The existing deep learning model based on amplitude spectrum estimates the amplitude spectrum mask through the model, and applies the amplitude spectrum mask to the original speech in combination with the voice transmission and receiving equipment to suppress the noise of the original noisy speech and restore the human voice.

[0003] Furthermore, since the system still uses the phase information of the original noisy speech when synthesizing the final noise-reduced speech, this method often cannot provide satisfactory noise reduction effects when processing noisy speech with low signal-to-noise ratio or mixed noise of multiple types.

[0004] Regarding the related technologies mentioned above, the inventors found that when processing noisy speech with low signal-to-noise ratio or mixed types of noise, the existing speech denoising methods only focus on amplitude spectrum estimation and use the phase information of the original noisy speech in the reconstruction process, resulting in low denoising accuracy. Summary of the Invention

[0005] In order to improve the noise reduction accuracy when processing noisy speech with low signal-to-noise ratio or mixed types of noise, the present application provides a two-stage speech noise reduction method, apparatus, computer equipment, computer-readable storage medium and computer program product based on amplitude spectrum and complex spectrum.

[0006] In a first aspect, the present application provides a two-stage speech denoising method based on amplitude spectrum and complex spectrum.

[0007] This application is achieved through the following technical solutions:

[0008] A two-stage speech noise reduction method based on amplitude spectrum and complex spectrum includes the following steps:

[0009] Obtaining the original speech signal;

[0010] Performing a preprocessing operation on the original speech signal to obtain an initial speech signal within a preset range;

[0011] Dividing the initial speech signal within a preset range according to a preset interval to obtain a plurality of time frames;

[0012] Decomposing the time frame into different frequency components by using a short-time Fourier transform method, and determining the amplitude and phase information of each different frequency component and the complex spectrum of each different frequency component;

[0013] Using the amplitudes of the different frequency components to perform noise estimation, obtain an amplitude spectrum mask, and apply the amplitude spectrum mask to the original speech signal to perform preliminary noise suppression on the original noisy speech to obtain a preliminary noise-reduced amplitude spectrum;

[0014] Based on the amplitude spectrum of the preliminary denoising, combined with the phase information, converting it into a complex spectrum of the preliminary denoising;

[0015] Determine the complex spectrum of the original noisy speech using the complex spectra of the different frequency components;

[0016] splicing the complex spectrum of the original noisy speech and the complex spectrum of the preliminary noise reduction to obtain a first complex spectrum;

[0017] performing noise estimation based on the first complex spectrum to obtain a second complex spectrum;

[0018] The second complex spectrum and the complex spectrum of the preliminary noise reduction are spliced ​​to obtain the complex spectrum of the target speech signal.

[0019] The complex spectrum of the target speech signal is converted into a time domain signal output by using the short-time inverse Fourier transform method.

[0020] In a preferred example, the present application may be further configured as follows: the step of using the amplitudes of the different frequency components to perform noise estimation and obtain an amplitude spectrum mask includes:

[0021] The non-DC part of each frequency component is used as the input of the N-layer amplitude spectrum encoder, and then passes through the M-layer amplitude spectrum GRU and the N-layer amplitude spectrum decoder in sequence, and outputs an amplitude spectrum mask with the same size as the input;

[0022] Among them, the number of input nodes of the GRU of the M-layer amplitude spectrum is Encoder n The output size of the N-layer amplitude spectrum decoder is the input of the amplitude spectrum decoder i+1 The output of the encoder with the magnitude spectrum i+1 The sum of the residuals of the output.

[0023] In a preferred example, the present application may be further configured as follows: the step of using the amplitudes of the different frequency components to perform noise estimation and obtain an amplitude spectrum mask includes:

[0024] The non-DC part of each frequency component is used as the input of the 4-layer amplitude spectrum encoder, and then passes through the 1-layer amplitude spectrum GRU and the 4-layer amplitude spectrum decoder in sequence to output the amplitude spectrum mask of size FFT_n / 2, where FFT_n represents the number of Fourier transform points of the short-time Fourier transform.

[0025] In a preferred example, the present application may be further configured as follows: performing noise estimation based on the first complex spectrum to obtain the second complex spectrum includes:

[0026] The first complex spectrum is processed by an N-layer complex spectrum encoder, an M-layer complex spectrum GRU, and an N-layer complex spectrum decoder, and outputs a second complex spectrum having the same size as the input;

[0027] Among them, the number of input nodes of the GRU of the M-layer complex spectrum is Encoder n The output size of the N-layer complex spectrum decoder is the complex spectrum Decoder i+1 The output of the Encoder with the complex spectrum i+1 The sum of the residuals of the output.

[0028] In a preferred example, the present application can be further configured as follows: the structure of the encoder includes a 5×3 depth convolution layer, a PReLU activation function layer, a 1×1 point convolution layer, a BatchNorm2d layer and a PReLU activation function layer connected in series.

[0029] In a preferred example, the present application can be further configured as follows: the structure of the decoder includes a 5×3 depth deconvolution layer, a PReLU activation function layer, a 1×1 point deconvolution layer, a BatchNorm2d layer and a PReLU activation function layer connected in series.

[0030] In a second aspect, the present application provides a two-stage speech noise reduction device based on amplitude spectrum and complex spectrum.

[0031] This application is achieved through the following technical solutions:

[0032] A two-stage speech noise reduction device based on amplitude spectrum and complex spectrum, comprising:

[0033] Original speech acquisition module, used to acquire original speech signals;

[0034] A preprocessing module, configured to perform a preprocessing operation on the original speech signal to obtain an initial speech signal within a preset range;

[0035] A division module, configured to divide an initial speech signal within a preset range according to a preset interval to obtain a plurality of time frames;

[0036] A short-time Fourier transform module is used to decompose the time frame into different frequency components using a short-time Fourier transform method, and determine the amplitude and phase information of each different frequency component and the complex spectrum of each different frequency component;

[0037] An amplitude spectrum estimation module is used to perform noise estimation using the amplitudes of different frequency components to obtain an amplitude spectrum mask, and apply the amplitude spectrum mask to the original speech signal to perform preliminary noise suppression on the original noisy speech to obtain a preliminary noise-reduced amplitude spectrum;

[0038] A preliminary denoised complex spectrum module, configured to convert the preliminary denoised amplitude spectrum into a preliminary denoised complex spectrum in combination with the phase information;

[0039] An original complex spectrum module, for determining the complex spectrum of the original noisy speech using the complex spectra of each different frequency component;

[0040] an optimization module, configured to concatenate the complex spectrum of the original noisy speech and the complex spectrum of the preliminary noise reduction to obtain a first complex spectrum;

[0041] a complex spectrum estimation module, performing noise estimation based on the first complex spectrum to obtain a second complex spectrum;

[0042] The denoising module is used to concatenate the second complex spectrum and the complex spectrum of the preliminary denoising to obtain the complex spectrum of the target speech signal, and convert the complex spectrum of the target speech signal into a time domain signal output through the short-time Fourier inverse transform method.

[0043] In a third aspect, the present application provides a computer device.

[0044] This application is achieved through the following technical solutions:

[0045] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements the steps of any one of the above-mentioned two-stage speech noise reduction methods based on amplitude spectrum and complex spectrum.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium.

[0047] This application is achieved through the following technical solutions:

[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned two-stage speech noise reduction methods based on amplitude spectrum and complex spectrum.

[0049] In a fifth aspect, the present application provides a computer program product.

[0050] This application is achieved through the following technical solutions:

[0051] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of any one of the above-mentioned two-stage speech noise reduction methods based on amplitude spectrum and complex spectrum.

[0052] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0053] Acquire the original speech signal and perform preprocessing operations on the original speech signal to ensure that the preprocessed initial speech signal is within an appropriate range; divide the processed initial speech signal according to preset intervals to obtain a number of time frames for subsequent speech noise reduction processing; use the short-time Fourier transform method to decompose the time frame into different frequency components, determine the amplitude, phase information and complex spectrum of each different frequency component to reflect the local characteristics of the signal; use the amplitude of each different frequency component to perform noise estimation to obtain an amplitude spectrum mask, and apply the amplitude spectrum mask to the original speech signal to perform preliminary noise suppression on the original noisy speech to obtain a preliminary noise-reduced amplitude spectrum to preliminarily filter out the noise in the speech signal; based on the preliminary noise-reduced amplitude spectrum, combined with the phase information, convert to obtain a preliminary noise-reduced complex spectrum to reflect the phase information of the speech signal at different frequencies; use the complex spectrum of each different frequency component to determine the complex spectrum of the original noisy speech to provide the original speech signal at Phase information at different frequencies; the complex spectrum of the original noisy speech and the complex spectrum of the preliminary denoising are spliced ​​to obtain a first complex spectrum to accurately reflect the phase information of the speech signal at different frequencies; the first complex spectrum is used to perform noise estimation to obtain a second complex spectrum, and the complex spectrum of the preliminary denoising is used to provide a reference guide for noise estimation, thereby improving the noise reduction effect while improving the efficiency of estimation; the second complex spectrum and the complex spectrum of the preliminary denoising are spliced ​​to obtain the complex spectrum of the target speech signal, and the complex spectrum of the target speech signal is converted into a time domain signal output through the short-time Fourier inverse transform method, thereby improving the noise reduction accuracy under complex environmental noise, and improving the noise reduction robustness of noisy speech in low signal-to-noise ratio or mixed scenes of multiple types of noise. It has higher adaptability and noise reduction performance, can better handle various complex environmental noises, and the human voice restoration of the denoised speech is higher and the listening experience is better, thereby improving the application effect of the denoised speech in voice communication, audio processing and speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The present invention provides an exemplary embodiment of a two-stage speech noise reduction method based on amplitude spectrum and complex spectrum.

[0055] Figure 2 A schematic structural diagram of an encoder for a two-stage speech denoising method based on amplitude spectrum and complex spectrum is provided as another exemplary embodiment of the present application.

[0056] Figure 3 A schematic structural diagram of a decoder of a two-stage speech noise reduction method based on amplitude spectrum and complex spectrum is provided as another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0057] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but as long as they are within the scope of the claims of the present application, they are protected by the patent law.

[0058] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0059] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0060] The embodiments of the present application are described in further detail below with reference to the accompanying drawings.

[0061] An embodiment of the present application provides a two-stage speech noise reduction method based on amplitude spectrum and complex spectrum, and the main steps of the method are described as follows.

[0062] Obtaining the original speech signal;

[0063] Performing a preprocessing operation on the original speech signal to obtain an initial speech signal within a preset range;

[0064] Dividing the initial speech signal within a preset range according to a preset interval to obtain a plurality of time frames;

[0065] Decomposing the time frame into different frequency components by using a short-time Fourier transform method, and determining the amplitude and phase information of each different frequency component and the complex spectrum of each different frequency component;

[0066] Using the amplitudes of the different frequency components to perform noise estimation, obtain an amplitude spectrum mask, and apply the amplitude spectrum mask to the original speech signal to perform preliminary noise suppression on the original noisy speech to obtain a preliminary noise-reduced amplitude spectrum;

[0067] Based on the amplitude spectrum of the preliminary denoising, combined with the phase information, converting it into a complex spectrum of the preliminary denoising;

[0068] Determine the complex spectrum of the original noisy speech using the complex spectra of the different frequency components;

[0069] splicing the complex spectrum of the original noisy speech and the complex spectrum of the preliminary noise reduction to obtain a first complex spectrum;

[0070] performing noise estimation based on the first complex spectrum to obtain a second complex spectrum;

[0071] The second complex spectrum and the complex spectrum of the preliminary noise reduction are spliced ​​to obtain the complex spectrum of the target speech signal, and the complex spectrum of the target speech signal is converted into a time domain signal output by the short-time Fourier inverse transform method.

[0072] Specifically, the original voice signal is collected from a voice source, such as a microphone or recording device, through an acquisition unit such as a microphone and an ADC hardware chip, and the voice analog signal in the environment is converted into a digital signal.

[0073] Perform some preprocessing operations on the original speech signal, such as removing DC offset, adjusting the sampling rate, etc., to ensure that the speech signal is within the preset range and obtain a speech signal that is more in line with the actual situation, which is beneficial to subsequent speech signal processing.

[0074] The initial speech signal within the preset range is divided into a plurality of time frames according to a preset interval, so as to divide the speech signal into short time frames, which facilitates subsequent speech signal processing. The preset interval can be 10ms-30ms. In this embodiment, the preset interval is 10ms.

[0075] The short-time Fourier transform (STFT) method is used to decompose the time frame into different frequency components, and the amplitude, phase information and complex spectrum of each different frequency component are determined to accurately reflect the local characteristics of the speech signal for noise reduction processing.

[0076] Subsequently, the amplitudes of the different frequency components are used to estimate noise, generating an amplitude spectrum mask. This mask is then applied to the original speech signal to perform preliminary noise suppression on the original noisy speech, resulting in a preliminary noise-reduced amplitude spectrum. Based on the amplitudes of the different frequency components, the absolute values ​​of the signal's spectral values ​​are calculated to obtain the amplitude spectrum for each component. This amplitude spectrum is then used to perform noise estimation and generate an amplitude spectrum mask. The amplitude spectrum mask is multiplied by the noise amplitude spectrum of the original speech signal and combined with the amplitude of the original speech signal to obtain the preliminary noise-reduced amplitude. This preliminary noise suppression is performed on the original noisy speech, generating an enhanced amplitude spectrum and performing preliminary noise reduction on the speech signal.

[0077] In applying the magnitude spectrum mask, the operation is performed on the basis of the magnitude spectrum.

[0078] Based on the amplitude spectrum of the preliminary denoised speech signal and the corresponding phase information, the coefficients of the complex exponential function at different frequencies of the preliminary denoised speech signal are determined and expressed in complex form to obtain the preliminary denoised complex spectrum, which more accurately reflects the phase information of the speech signal at different frequencies. The complex spectrum consists of two parts: real and imaginary. Complex number = real part + imaginary part i, where i is the basic unit of complex number. During calculation, the complex spectrum is first calculated using real-valued operations. Then, a cross-operation is performed to simulate the multiplication rules of complex numbers to complete the complex operations and model the relationship between the amplitude spectrum and phase spectrum.

[0079] Similarly, using the amplitude spectrum and phase spectrum of the original noisy speech, the phase spectrum of the original noisy signal is obtained by calculating the amplitude angle of the spectral value of the signal, and then using the amplitude spectrum and phase spectrum of the original noisy signal, the coefficients of the complex exponential function of the speech signal of the original noisy speech at different frequencies are determined, expressed in complex form, to obtain the complex spectrum of the original noisy speech.

[0080] Next, the complex spectrum of the original noisy speech and the complex spectrum of the preliminary noise reduction are spliced ​​and added to obtain a first complex spectrum to accurately reflect the phase information of the speech signal at different frequencies.

[0081] The first complex spectrum is used to perform noise estimation, and is used to optimize and adjust the complex spectrum of the first stage of preliminary noise reduction to obtain a second complex spectrum.

[0082] The second complex spectrum and the complex spectrum of the preliminary noise reduction are then concatenated and added to obtain the complex spectrum of the target speech signal, that is, the final optimized complex spectrum.

[0083] Finally, the complex spectrum of the target speech signal is converted into a time domain signal output through the inverse short-time Fourier transform method (ISTFT).

[0084] In the process of applying the complex spectrum, operations are performed on the basis of the complex spectrum.

[0085] The output signal is played back through the terminal unit to obtain the noise-reduced speech in the time domain. The terminal unit can be composed of any real-time conference communication device with network access to play the noise-reduced audio data.

[0086] In one embodiment, the step of using the amplitudes of different frequency components to perform noise estimation and obtain an amplitude spectrum mask includes:

[0087] The non-DC part of each frequency component is used as the input of the N-layer amplitude spectrum encoder, and then passes through the M-layer amplitude spectrum GRU and the N-layer amplitude spectrum decoder in sequence, outputting an amplitude spectrum mask with the same size as the input;

[0088] Among them, the number of input nodes of the GRU of the M-layer amplitude spectrum is Encoder n The output size of the M-layer amplitude spectrum GRU includes a fully connected layer, and the number of output nodes of the fully connected layer is consistent with the number of input nodes of the M-layer amplitude spectrum GRU; the input of the N-layer amplitude spectrum decoder is the amplitude spectrum Decoder i+1 The output of the encoder with the magnitude spectrum i+1 The sum of the residuals of the output.

[0089] In one embodiment, the step of performing noise estimation based on the first complex spectrum to obtain the second complex spectrum includes:

[0090] The first complex spectrum is processed by an N-layer complex spectrum encoder, an M-layer complex spectrum GRU, and an N-layer complex spectrum decoder, and outputs a second complex spectrum having the same size as the input;

[0091] Among them, the number of input nodes of the GRU of the M-layer complex spectrum is Encoder n The output size of the M-layer complex spectrum GRU includes a fully connected layer, and the number of output nodes of the fully connected layer is consistent with the number of input nodes of the M-layer complex spectrum GRU; the input of the N-layer complex spectrum decoder is the complex spectrum Decoder i+1 The output of the Encoder with the complex spectrum i+1 The sum of the residuals of the output.

[0092] Reference Figure 1 Specifically, after the short-time frame is processed by STFT, the frequency domain amplitude spectrum of the noisy speech is obtained, where the size of the non-DC part is FFT_n / 2, and FFT_n represents the number of short-time Fourier transform points of STFT.

[0093] The non-DC portion of the amplitude spectrum is taken as input and processed by an N-layer amplitude spectrum encoder, an M-layer amplitude spectrum GRU (Gated Recurrent Unit) neural network, and an N-layer amplitude spectrum decoder to produce an amplitude spectrum mask of the same size as the input. This combination of an N-layer amplitude spectrum encoder, an M-layer amplitude spectrum GRU, and an N-layer amplitude spectrum decoder is called a one-stage amplitude spectrum model.

[0094] The one-stage amplitude spectrum model, based on the Unet architecture, uses a full-band amplitude spectrum as input. The encoder, GRU, and decoder operate in amplitude form. The one-stage model outputs an amplitude spectrum mask, which is applied to the amplitude spectrum of the noisy speech to obtain the amplitude spectrum of the first-stage denoised speech. This spectrum is then combined with the original phase information to create the complex spectrum of the first-stage denoised speech. The concatenated complex spectrum of the noisy speech serves as input for the second stage.

[0095] In one embodiment, the step of using the amplitudes of different frequency components to perform noise estimation and obtain an amplitude spectrum mask includes:

[0096] The non-DC part of each frequency component is used as the input of the 4-layer amplitude spectrum encoder, and then passes through the 1-layer amplitude spectrum GRU and the 4-layer amplitude spectrum decoder in sequence to output the amplitude spectrum mask of size FFT_n / 2, where FFT_n represents the number of Fourier transform points of the short-time Fourier transform.

[0097] By preferably adopting N=4, M=1, a 4-layer amplitude spectrum encoder, a 1-layer amplitude spectrum GRU and a 4-layer amplitude spectrum decoder are designed; the number of input nodes of the amplitude spectrum GRU is Encoder n The output size of the decoder is P, and the number of hidden nodes is P; i , whose input is defined as the Decoder of the magnitude spectrum i+1 The output of the encoder with the magnitude spectrum i+1 The sum of the residuals of the output is used to improve the noise reduction efficiency while ensuring high noise reduction accuracy.

[0098] In one embodiment, the amplitude spectrum GRU includes a fully connected layer, the number of output nodes of the fully connected layer is consistent with the number of input nodes of the amplitude spectrum GRU, so that the signal processed by the amplitude spectrum GRU is dimensionalized through the fully connected layer, so that the transformed signal can be input into the subsequent amplitude spectrum decoder more quickly.

[0099] In one embodiment, the number of hidden nodes P of the GRU of the amplitude spectrum is preferably 128 to better solve the gradient vanishing problem in back propagation and improve the accuracy of the model.

[0100] In one embodiment, the number of FFT_n short-time Fourier transform points is 512, which can obtain a frequency domain amplitude spectrum of noisy speech that is more in line with actual conditions and improve the accuracy of the sample.

[0101] In one embodiment, the output channels of the amplitude spectrum encoder are both set to 48, and the output channels of the amplitude spectrum decoder are both set to 48.

[0102] By setting the parameters of the convolution layer, it is preferred that the number of output channels of Encoder1 to Encoder4 of the amplitude spectrum is all designed to be 48, and the number of output channels of Decoder1 to Decoder4 of the amplitude spectrum is all designed to be 48, so as to obtain a more accurate noise-reduced speech signal.

[0103] After a first-stage amplitude spectrum speech denoising, the complex spectrum of the first-stage denoised speech is concatenated with the complex spectrum of the original noisy speech (the original speech signal divided into short time frames) as input, making model training relatively simple. The model can receive near-clean audio (i.e., the residual sum of the complex spectrum of the initial denoised model and the complex spectrum of the original speech signal) as input, reducing the burden of model learning. The complex spectrum of the output signal of the first-stage model provides some learning guidance. After being processed by an N-layer complex spectrum encoder, an M-layer complex spectrum GRU, and an N-layer complex spectrum decoder, a second complex spectrum of the same size as the input is ultimately obtained. This N-layer complex spectrum encoder, M-layer complex spectrum GRU, and N-layer complex spectrum decoder are called a two-stage complex spectrum model.

[0104] The two-stage complex spectrum model uses the full-band complex spectrum as input based on the Unet structure. The operation mode of the encoder, GRU and decoder is to simulate the complex operation. During the calculation process, the real and imaginary parts of the complex spectrum are used as the two input channels, that is, the complex spectrum obtained by STFT processing, which contains two parts: real part and imaginary part. Complex number = real part + imaginary part i, where i is the basic unit of complex number. The complex spectrum first uses real-valued operations, and then simulates the multiplication rule of complex numbers through cross operations to complete the complex operation and realize the modeling of the relationship between the amplitude spectrum and the phase spectrum.

[0105] The output of the two-stage model is a complex spectrum. During the application process, the complex spectrum output by the two-stage model and the complex spectrum converted from the first-stage output are added together to obtain the complex spectrum of the target speech signal. The complex spectrum of the target speech signal is converted into a time domain signal through the short-time Fourier inverse transform method as the final output result.

[0106] In one embodiment, by preferably adopting N=4, M=1, a 4-layer complex spectrum encoder, a 1-layer complex spectrum GRU and a 4-layer complex spectrum decoder are designed; the number of input nodes of the complex spectrum GRU is Encoder n The output size of the complex spectrum is P, and the number of hidden nodes is P; i , whose input is defined as the Decoder of the complex spectrum i+1 The output of the Encoder with the complex spectrum i+1 The sum of the residuals of the output is used to improve the noise reduction efficiency while ensuring high noise reduction accuracy.

[0107] In one embodiment, the GRU of the complex spectrum includes a fully connected layer, the number of output nodes of the fully connected layer is consistent with the number of input nodes of the GRU of the complex spectrum, so as to perform dimensionality transformation on the signal processed by the GRU of the complex spectrum through the fully connected layer of the complex spectrum, so that the transformed signal can be input into the subsequent complex spectrum decoder Decoder more quickly.

[0108] In one embodiment, the output channels of the encoder of the complex spectrum are all set to 32, and the number of hidden nodes of the GRU of the complex spectrum is set to 32.

[0109] By setting the parameters of the convolutional layer, the number of output channels of Encoder1 to Encoder4 of the complex spectrum is optimized to 32, and the number of output channels of Decoder1 to Decoder4 of the complex spectrum is optimized to 32, so as to obtain a more accurate denoised speech signal in the two-stage denoising process.

[0110] In one embodiment, the number of hidden nodes P of the complex spectrum GRU is preferably 128 to better solve the gradient vanishing problem in back propagation and improve the accuracy of the model.

[0111] Reference Figure 2 In one embodiment, the structure of the encoder includes a 5×3 depth convolution layer, a PReLU activation function layer (non-linear activation layer), a 1×1 point convolution layer, a BatchNorm2d layer (BN layer) and a PReLU activation function layer connected in series.

[0112] In the amplitude form and complex form of the encoder, by designing each layer of the encoder to include a 5×3 depth convolution, PReLU activation function, 1×1 point convolution, BatchNorm2d function and PReLU activation function, the input signal is sequentially subjected to 5×3 depth convolution operation, PReLU activation function activation, 1×1 point convolution operation, BatchNorm2d function to perform batch normalization operation, and finally activated by the PReLU activation function to obtain the output signal. This scheme changes the conventional convolution into depth-separable convolution (depth convolution, point convolution, BN layer and nonlinear activation layer), and adopts the order of depth convolution first and then point convolution, which can reduce the number of model parameters and play the role of a lightweight model, so as to improve the noise reduction efficiency while ensuring high noise reduction accuracy.

[0113] Reference Figure 3 In one embodiment, the structure of the decoder includes a 5×3 depth deconvolution layer, a PReLU activation function layer, a 1×1 point deconvolution layer, a BatchNorm2d layer (BN layer) and a PReLU activation function layer connected in series.

[0114] In the amplitude form and complex form of Decoder, by designing each layer of decoder Decoder to a structure including 5×3 depth deconvolution, PReLU activation function, 1×1 point deconvolution, BatchNorm2d function and PReLU activation function, the input signal is sequentially subjected to 5×3 depth deconvolution operation, PReLU activation function activation, 1×1 point deconvolution operation, BatchNorm2d function to perform batch normalization operation, and finally activated by PReLU activation function. The output signal is obtained in the order of depth deconvolution first and then point deconvolution. The conventional deconvolution is changed to depth-separable deconvolution (depth deconvolution, point deconvolution, BN layer and nonlinear activation layer), which can reduce the number of model parameters and play the role of a lightweight model, so as to improve the noise reduction efficiency while ensuring high noise reduction accuracy.

[0115] A two-stage speech denoising method based on amplitude spectrum and complex spectrum adopts Unet structures based on amplitude and complex operations respectively. In the first stage, the input is the amplitude spectrum of the noisy speech, and the output is the amplitude mask; in the second stage, the input is the complex spectrum of the noisy speech and the complex spectrum of the speech that has undergone preliminary noise reduction in the first stage; the network structures of the two stages are similar. In the encoder part, both are implemented by combining deep convolution and point convolution; in the decoder part, both are implemented by point deconvolution and depth deconvolution.

[0116] During calculation, a calculation unit can be composed of a single chip microcomputer or a calculation chip with an operating system, which is used for calculating the noise reduction model.

[0117] The output signal can be a data transmission unit composed of a network system that can transmit data, transmitting the data completed by the calculation.

[0118] A two-stage speech denoising method based on the magnitude spectrum and complex spectrum utilizes a UNet structure that employs two-stage computations on the magnitude and complex spectrum. This method significantly improves the model's denoising performance under complex conditions, significantly enhancing the restoration and sound quality of human voices. Furthermore, by employing a specialized encoder and decoder implementation, the model's computational workload is effectively reduced, enabling real-time model execution on lower-performance platforms. Finally, the introduction of a multi-objective loss for magnitude and complex spectrum reduces the complexity of model learning, facilitating model training.

[0119] A two-stage speech noise reduction method based on amplitude spectrum and complex spectrum can improve the noise reduction accuracy in complex environmental noise and enhance the noise reduction robustness of noisy speech in low signal-to-noise ratio or mixed noise scenarios of multiple types. It has higher adaptability and noise reduction performance, can better handle various complex environmental noises, and the human voice restoration of the noise-reduced speech is higher and the listening experience is better, thereby improving the application effect of noise-reduced speech in voice communication, audio processing, speech recognition and other fields.

[0120] A two-stage speech noise reduction method based on amplitude spectrum and complex spectrum can be flexibly deployed in real-time conference communication equipment to achieve real-time noise reduction and suppression of speech.

[0121] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0122] The embodiment of the present application further provides a two-stage speech noise reduction device based on amplitude spectrum and complex spectrum, which corresponds one-to-one to the two-stage speech noise reduction method based on amplitude spectrum and complex spectrum in the above embodiment. The two-stage speech noise reduction device based on amplitude spectrum and complex spectrum includes:

[0123] Original speech acquisition module, used to acquire original speech signals;

[0124] A preprocessing module, configured to perform a preprocessing operation on the original speech signal to obtain an initial speech signal within a preset range;

[0125] A division module, configured to divide an initial speech signal within a preset range according to a preset interval to obtain a plurality of time frames;

[0126] A short-time Fourier transform module is used to decompose the time frame into different frequency components using a short-time Fourier transform method, and determine the amplitude and phase information of each different frequency component and the complex spectrum of each different frequency component;

[0127] An amplitude spectrum estimation module is used to perform noise estimation using the amplitudes of different frequency components to obtain an amplitude spectrum mask, and apply the amplitude spectrum mask to the original speech signal to perform preliminary noise suppression on the original noisy speech to obtain a preliminary noise-reduced amplitude spectrum;

[0128] A preliminary denoised complex spectrum module, configured to convert the preliminary denoised amplitude spectrum into a preliminary denoised complex spectrum in combination with the phase information;

[0129] An original complex spectrum module, for determining the complex spectrum of the original noisy speech using the complex spectra of each different frequency component;

[0130] an optimization module, configured to concatenate the complex spectrum of the original noisy speech and the complex spectrum of the preliminary noise reduction to obtain a first complex spectrum;

[0131] a complex spectrum estimation module, performing noise estimation based on the first complex spectrum to obtain a second complex spectrum;

[0132] The denoising module is used to concatenate the second complex spectrum and the complex spectrum of the preliminary denoising to obtain the complex spectrum of the target speech signal, and convert the complex spectrum of the target speech signal into a time domain signal output through the short-time Fourier inverse transform method.

[0133] For the specific definition of a two-stage speech noise reduction device based on amplitude spectrum and complex spectrum, please refer to the definition of a two-stage speech noise reduction method based on amplitude spectrum and complex spectrum above, which will not be repeated here.

[0134] Each module in the aforementioned dual-stage speech noise reduction device based on amplitude spectrum and complex spectrum can be implemented in whole or in part via software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0135] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When executed by the processor, the computer program implements any of the above-mentioned two-stage speech noise reduction methods based on amplitude spectrum and complex spectrum.

[0136] In one embodiment, a computer-readable storage medium is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the above-mentioned two-stage speech noise reduction methods based on amplitude spectrum and complex spectrum is implemented.

[0137] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, it implements any of the above-mentioned two-stage speech noise reduction methods based on amplitude spectrum and complex spectrum.

[0138] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0139] Those skilled in the art will clearly understand that for the sake of convenience and brevity in description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A two-stage speech noise reduction method based on amplitude spectrum and complex spectrum, characterized in that: The following steps are included: Obtaining the original speech signal; Performing a preprocessing operation on the original speech signal to obtain an initial speech signal within a preset range; Dividing the initial speech signal within a preset range according to a preset interval to obtain a plurality of time frames; Decomposing the time frame into different frequency components by using a short-time Fourier transform method, and determining the amplitude and phase information of each different frequency component and the complex spectrum of each different frequency component; The amplitude of each different frequency component is used to estimate the noise and obtain the amplitude spectrum mask, including: taking the non-DC part of each different frequency component as the input of the N-layer amplitude spectrum encoder, and then passing it through the M-layer amplitude spectrum GRU and the N-layer amplitude spectrum decoder in sequence, and outputting the amplitude spectrum mask with the same size as the input. Among them, the number of input nodes of the M-layer amplitude spectrum GRU is Encoder n The output size of the N-layer amplitude spectrum decoder is the input of the amplitude spectrum decoder i+1 The output of the encoder with the magnitude spectrum i+1 The sum of the residuals of the output; Applying the amplitude spectrum mask to the original speech signal to perform preliminary noise suppression on the original noisy speech to obtain a preliminary noise-reduced amplitude spectrum; Based on the amplitude spectrum of the preliminary denoising, combined with the phase information, converting it into a complex spectrum of the preliminary denoising; Determine the complex spectrum of the original noisy speech using the complex spectra of the different frequency components; splicing the complex spectrum of the original noisy speech and the complex spectrum of the preliminary noise reduction to obtain a first complex spectrum; performing noise estimation based on the first complex spectrum to obtain a second complex spectrum; The second complex spectrum and the complex spectrum of the preliminary noise reduction are spliced ​​to obtain the complex spectrum of the target speech signal. The complex spectrum of the target speech signal is converted into a time domain signal output by using the short-time inverse Fourier transform method.

2. The two-stage speech noise reduction method based on amplitude spectrum and complex spectrum according to claim 1 is characterized in that: The step of using the amplitudes of the different frequency components to perform noise estimation and obtain an amplitude spectrum mask includes: The non-DC part of each frequency component is used as the input of the 4-layer amplitude spectrum encoder, and then passes through the 1-layer amplitude spectrum GRU and the 4-layer amplitude spectrum decoder in sequence to output the amplitude spectrum mask of size FFT_n / 2, where FFT_n represents the number of Fourier transform points of the short-time Fourier transform.

3. The two-stage speech noise reduction method based on amplitude spectrum and complex spectrum according to claim 1 is characterized in that: The step of performing noise estimation based on the first complex spectrum to obtain a second complex spectrum includes: The first complex spectrum is processed by an N-layer complex spectrum encoder, an M-layer complex spectrum GRU, and an N-layer complex spectrum decoder, and outputs a second complex spectrum having the same size as the input; Among them, the number of input nodes of the GRU of the M-layer complex spectrum is Encoder n The output size of the N-layer complex spectrum decoder is the complex spectrum Decoder i+1 The output of the Encoder with the complex spectrum i+1 The sum of the residuals of the output.

4. The two-stage speech noise reduction method based on amplitude spectrum and complex spectrum according to claim 1 or 3, characterized in that: The structure of the encoder includes 5 3 depth convolution layer, PReLU activation function layer, 1 1-point convolutional layer, BatchNorm2d layer, and PReLU activation function layer.

5. The two-stage speech noise reduction method based on amplitude spectrum and complex spectrum according to claim 1 or 3, characterized in that: The structure of the decoder includes 5 3 deep deconvolution layers, PReLU activation function layers, 1 1-point deconvolution layer, BatchNorm2d layer, and PReLU activation function layer.

6. A two-stage speech noise reduction device based on amplitude spectrum and complex spectrum, characterized in that: include, Original speech acquisition module, used to acquire original speech signals; A preprocessing module, configured to perform a preprocessing operation on the original speech signal to obtain an initial speech signal within a preset range; A division module, configured to divide an initial speech signal within a preset range according to a preset interval to obtain a plurality of time frames; A short-time Fourier transform module is used to decompose the time frame into different frequency components using a short-time Fourier transform method, and determine the amplitude and phase information of each different frequency component and the complex spectrum of each different frequency component; The amplitude spectrum estimation module is used to use the amplitude of each different frequency component to perform noise estimation, obtain an amplitude spectrum mask, and apply the amplitude spectrum mask to the original speech signal to perform preliminary noise suppression on the original noisy speech to obtain a preliminary noise-reduced amplitude spectrum; specifically, it is used to use the non-DC part of each different frequency component as the input of the N-layer amplitude spectrum encoder, and then pass it through the M-layer amplitude spectrum GRU and the N-layer amplitude spectrum decoder in sequence, and output an amplitude spectrum mask with the same size as the input amount, wherein the number of input nodes of the M-layer amplitude spectrum GRU is Encoder n The output size of the N-layer amplitude spectrum decoder is the input of the amplitude spectrum decoder i+1 The output of the encoder with the magnitude spectrum i+1 The sum of the residuals of the output; A preliminary denoised complex spectrum module, configured to convert the preliminary denoised amplitude spectrum into a preliminary denoised complex spectrum in combination with the phase information; An original complex spectrum module, for determining the complex spectrum of the original noisy speech using the complex spectra of each different frequency component; an optimization module, configured to concatenate the complex spectrum of the original noisy speech and the complex spectrum of the preliminary noise reduction to obtain a first complex spectrum; a complex spectrum estimation module, performing noise estimation based on the first complex spectrum to obtain a second complex spectrum; The denoising module is used to concatenate the second complex spectrum and the complex spectrum of the preliminary denoising to obtain the complex spectrum of the target speech signal, and convert the complex spectrum of the target speech signal into a time domain signal output through the short-time Fourier inverse transform method.

7. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, characterized in that The invention comprises a computer program, which implements the steps of the method according to any one of claims 1 to 5 when the computer program is executed by a processor.

Images