A speech denoising method and system based on complex spectrum

Abstract

This invention discloses a speech denoising method and system based on complex spectrum. The method includes real-time acquisition of the original speech signal, converting the original speech signal into a frequency domain complex form to obtain a first complex spectrum, extracting the non-DC portion of the first complex spectrum to obtain a second complex spectrum, estimating noise on the second complex spectrum using a preset complex operation model to obtain a first complex mask. The preset complex operation model includes a preset N-layer complex form Encoder, a preset M-layer complex form RNN, and a preset N-layer complex form Decoder. Based on the real and imaginary parts of the first complex mask and the amplitude and phase spectra of the original speech signal, a denoising spectrum transformation is performed to obtain a third complex spectrum, which is then converted into a time domain form to obtain a denoised signal, thus achieving speech denoising. This embodiment achieves real-time noise reduction and suppression, clearly restores the speech signal, reduces model computation, and improves the denoising effect.

Description

Technical Field

[0001] This invention relates to the field of speech denoising, and more particularly to a speech denoising method and system based on complex spectrum. Background Technology

[0002] In real-time conferencing, broadcasting, and other scenarios, microphones receive a large amount of ambient noise when they receive voice signals. As a result, when the voice signals are sent to the receiving terminal or playback device, the voice information often contains a lot of noise. Therefore, voice noise reduction is crucial in the voice transmission process when the voice signal is acquired. Voice noise reduction is an essential part of the audio processing process and can remove the noise contained in the audio signal.

[0003] In existing technical solutions, voice signals are transmitted and acquired through voice transceivers, and noise reduction is performed based on the amplitude spectrum of the acquired voice signals to suppress noise. However, noise reduction is only performed on the amplitude spectrum, ignoring the phase information of the voice signals. As a result, even the theoretically optimal solution cannot fully restore the ideal clean voice signal, and the output audio quality cannot be guaranteed. The processing effect is poor for voice signals with low signal-to-noise ratio. At the same time, the computing power of real-time voice noise reduction systems is limited, and it is usually difficult to meet the requirements of running large-scale deep learning noise reduction models in real time, which has obvious limitations. Summary of the Invention

[0004] This invention provides a speech denoising method and system based on complex spectrum, which achieves real-time noise reduction and suppression, clearly restores speech signals, reduces model computation, and improves noise reduction effect.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide a speech denoising method based on complex spectrum, comprising:

[0006] The original speech signal is acquired in real time and converted into a complex form in the frequency domain to obtain a first complex spectrum. The non-DC part of the first complex spectrum is extracted to obtain a second complex spectrum.

[0007] The noise of the second complex number spectrum is estimated by using a preset complex number operation model to obtain the first complex number mask; wherein, the preset complex number operation model includes a preset N-layer complex number form Encoder, a preset M-layer complex number form RNN and a preset N-layer complex number form Decoder;

[0008] Based on the real and imaginary parts of the first complex mask, the amplitude spectrum and phase spectrum of the original speech signal, a noise reduction spectrum transformation is performed to obtain the third complex spectrum. The third complex spectrum is then transformed into a time-domain form to obtain the noise-reduced signal, thus achieving speech noise reduction.

[0009] Implementing this embodiment of the invention involves acquiring the original speech signal in real time and converting it into a complex frequency domain form to obtain a first complex spectrum. The non-DC portion of the first complex spectrum is extracted to obtain a second complex spectrum. A preset complex operation model (a complex encoder, a complex RNN, and a complex decoder) is used to estimate noise in the second complex spectrum to obtain a first complex mask. The preset complex operation model includes a preset N-layer complex encoder, a preset M-layer complex RNN, and a preset N-layer complex decoder. The Unet structure (Encoder+RNN+Decoder) employing complex operations significantly improves the noise reduction effect of the complex operation model, especially at low signal-to-noise ratios, greatly enhancing the ability to reproduce human voices. The implementation using complex encoders and decoders significantly reduces the computational load of the model and the complexity of model learning. Based on the real and imaginary parts of the first complex mask and the amplitude and phase spectra of the original speech signal, a denoising spectrum transformation is performed to obtain a third complex spectrum. This third complex spectrum is then converted into a time-domain form to obtain the denoised signal, thus achieving speech denoising. The denoising spectrum transformation using the real and imaginary parts of the first complex mask takes into account both the amplitude and phase spectra of the original speech signal, avoiding the neglect of phase information during original speech signal processing. This results in the restoration of an ideal, clean speech signal, improving the denoising effect. It can process noisy speech signals in real time and can be deployed in real-time conferencing communication equipment for real-time noise reduction and suppression of speech signals.

[0010] As a preferred approach, noise estimation is performed on the first real part and the first imaginary part using a preset complex number operation model to obtain the first complex number mask, specifically:

[0011] The second complex spectrum is input into a pre-defined N-layer complex form Encoder and convolutionally encoded to obtain the fourth complex spectrum;

[0012] The fourth complex spectrum is input into a preset M-layer complex RNN and cross-neural network operation is performed to obtain the fifth complex spectrum.

[0013] The fifth complex spectrum is input into a pre-defined N-layer complex form decoder for deconvolution decoding to obtain the first complex mask.

[0014] By implementing the embodiments of the present invention, a complex number operation model is established through a complex number form of Encoder, RNN and Decoder, noise estimation is performed, and a complex number mask is obtained. By utilizing complex number operations with a not complicated structure, the complexity of model learning is reduced, enabling the complex number operation model to run in real time on a lower performance platform.

[0015] As a preferred approach, the second complex spectrum is input into a pre-defined N-layer complex-form Encoder for convolutional encoding to obtain the fourth complex spectrum, specifically as follows:

[0016] The spectrum of the second complex number is split into a first real part and a first imaginary part;

[0017] By performing cross-convolution operations on the first real part and the first imaginary part through depthwise convolution and pointwise convolution, the real part convolution and the imaginary part convolution are obtained. The real part convolution and the imaginary part convolution are then subjected to nonlinear activation of the neural network to obtain the spectrum of the fourth complex number.

[0018] The number of input channels, output channels, and groups in a depthwise convolution are the same as the number of input channels in the current layer of the depthwise convolution, and the number of output channels in a pointwise convolution is the same as the number of output channels in the current layer of the pointwise convolution.

[0019] As a preferred approach, cross-convolution operations are performed on the first real part and the first imaginary part using depthwise convolution and pointwise convolution to obtain real part convolution and imaginary part convolution. Then, the real part convolution and imaginary part convolution are subjected to nonlinear activation of the neural network to obtain the fourth complex number spectrum, specifically:

[0020] The first real part is input into the depthwise convolution to perform depthwise convolution operation, and the first real part depthwise convolution and the first imaginary part depthwise convolution are obtained. The first real part depthwise convolution and the first imaginary part depthwise convolution are then input into the real part pointwise convolution to perform pointwise convolution operation, and the real part convolution is obtained. The depthwise convolution includes the real part depthwise convolution and the imaginary part depthwise convolution.

[0021] The first imaginary part is input into a depthwise convolution to perform a depthwise convolution operation, resulting in a second real part depthwise convolution and a second imaginary part depthwise convolution. The second real part depthwise convolution and the second imaginary part depthwise convolution are then input into an imaginary part pointwise convolution to perform a pointwise convolution operation, resulting in an imaginary part convolution.

[0022] The real part convolution and the imaginary part convolution are concatenated to obtain the sixth complex spectrum. The sixth complex spectrum is then passed through a BN layer and a nonlinear activation layer to obtain the fourth complex spectrum.

[0023] As a preferred approach, the fourth complex spectrum is input into a pre-defined M-layer complex RNN for cross-neural network operations to obtain the fifth complex spectrum, specifically as follows:

[0024] The spectrum of the fourth complex number is split into a second real part and a second imaginary part;

[0025] The second real part is input into the real part RNN and the imaginary part RNN to perform RNN neural network operations, thereby obtaining the first real part RNN and the first imaginary part RNN, and the real part RNN is obtained based on the first real part RNN and the first imaginary part RNN.

[0026] The second imaginary part is input into the real part RNN and the imaginary part RNN, and RNN neural network operation is performed to obtain the second real part RNN and the second imaginary part RNN. Based on the second real part RNN and the second imaginary part RNN, the imaginary part RNN is obtained.

[0027] Based on the fully connected layers in the pre-defined M-layer complex RNN, the real part RNN and the imaginary part RNN are mapped to the output to obtain the fifth complex spectrum.

[0028] As a preferred approach, the fifth complex spectrum is input into a pre-defined N-layer complex form decoder for deconvolution decoding to obtain the first complex mask, specifically:

[0029] The spectrum of the fifth complex number is split into the third real part and the third imaginary part;

[0030] By performing cross-deconvolution operations on the third real part and the third imaginary part through point deconvolution and depth deconvolution, the real part deconvolution and the imaginary part deconvolution are obtained. The real part deconvolution and the imaginary part deconvolution are then subjected to nonlinear activation of the neural network to obtain the first complex mask.

[0031] In this context, the number of input channels for point deconvolution is the same as the number of input channels for the current layer of point deconvolution, the number of output channels for point deconvolution is the same as the number of output channels for the current layer of point deconvolution, and the number of input channels, output channels, and groups for depth deconvolution are the same as the number of output channels for the current layer of depth deconvolution.

[0032] As a preferred approach, cross-deconvolution operations are performed on the third real part and the third imaginary part using point deconvolution and depthwise deconvolution to obtain real part deconvolution and imaginary part deconvolution. Then, the real part deconvolution and imaginary part deconvolution are subjected to nonlinear activation of the neural network to obtain the first complex mask, specifically:

[0033] The third real part is input into the point deconvolution to perform point deconvolution operation, obtaining the first real part point deconvolution and the second real part point deconvolution. The first real part point deconvolution and the second real part point deconvolution are then cross-inputted into the real part depth deconvolution and the imaginary part depth deconvolution to perform depth deconvolution operation, obtaining the real part deconvolution. Here, point deconvolution includes real part point de-depth convolution and imaginary part point de-depth convolution.

[0034] The third imaginary part is input into the point deconvolution and the point deconvolution operation is performed to obtain the first imaginary part point deconvolution and the second imaginary part point deconvolution. The first imaginary part point deconvolution and the second imaginary part point deconvolution are then cross-inputted into the real part depth deconvolution and the imaginary part depth deconvolution and the depth deconvolution operation is performed to obtain the imaginary part deconvolution.

[0035] The real part deconvolution and the imaginary part deconvolution are concatenated to obtain the seventh complex spectrum. The seventh complex spectrum is then passed through a BN layer and a nonlinear activation layer to obtain the first complex mask.

[0036] As a preferred approach, based on the first complex mask, the amplitude spectrum and phase spectrum of the original speech signal, a denoising spectrum transformation is performed to obtain a third complex spectrum. This third complex spectrum is then converted into a time-domain form to obtain the denoised signal, thus achieving speech denoising. Specifically:

[0037] Based on the real and imaginary parts of the first complex mask, obtain the real part mask and the imaginary part mask;

[0038] The real part mask and the imaginary part mask are offset to obtain the amplitude spectrum mask and the phase spectrum offset. Based on the amplitude spectrum mask, the phase spectrum offset, and the amplitude spectrum and phase spectrum of the original speech signal, the noise reduction amplitude and phase are calculated to obtain the first amplitude spectrum and the first phase spectrum.

[0039] The first phase spectrum and the first amplitude spectrum are transformed into real and imaginary parts to obtain the fourth real part and the fourth imaginary part. Based on the fourth real part and the fourth imaginary part, the third complex spectrum is obtained.

[0040] According to ISTFT, the third complex spectrum is transformed into a time-domain form to obtain the noise-reduced signal, thus achieving speech noise reduction.

[0041] By implementing the embodiments of the present invention, the ordinary noise reduction model ignores the phase information of the speech signal and the computing power requirement exceeds the real-time requirements of the system. Through offset calculation and noise reduction amplitude and phase calculation, the first amplitude spectrum and the first phase spectrum after noise reduction are accurately calculated. By converting the spectrum from virtual to real, the spectrum after noise reduction, namely the third complex spectrum, is calculated. The entire speech noise reduction method is lightweight. While meeting the real-time requirements of the system, noise can be effectively suppressed under the condition of low signal-to-noise ratio.

[0042] As a preferred approach, the real part mask and the imaginary part mask are offset to obtain the amplitude spectrum mask and the phase spectrum offset. Then, based on the amplitude spectrum mask, the phase spectrum offset, and the amplitude and phase spectra of the original speech signal, noise reduction amplitude and phase calculations are performed to obtain the first amplitude spectrum and the first phase spectrum, specifically:

[0043] The amplitude spectrum mask is obtained by performing amplitude normalization transformation on the real part mask and the imaginary part mask. The amplitude normalization transformation formula is as follows:

[0044]

[0045] Among them, mags mask For amplitude spectrum mask, real mask For the real part mask, imag mask For the imaginary part mask:

[0046] The real part mask and the imaginary part mask are transformed by phase bias to obtain the phase spectrum bias. The phase bias transformation formula is as follows:

[0047]

[0048] Among them, phase bias For phase spectrum bias;

[0049] Based on the amplitude spectrum and amplitude spectrum mask of the original speech signal, the noise reduction amplitude is calculated to obtain the first amplitude spectrum; the formula for calculating the noise reduction amplitude is as follows:

[0050] mags denoisy =mags noisy ·mags mask

[0051] Among them, mags denoisy For the first amplitude spectrum, mags noisy The amplitude spectrum of the original speech signal;

[0052] Based on the phase spectrum and phase spectrum offset of the original speech signal, noise reduction phase calculation is performed to obtain the first phase spectrum; the noise reduction phase calculation formula is as follows:

[0053] phase denoisy =phase noisy +phase bias

[0054] Among them, phase denoisy For the first phase spectrum, phase noisy This is the phase spectrum of the original speech signal.

[0055] To address the same technical problem, embodiments of the present invention also provide a speech denoising system based on complex spectrum, comprising: a data acquisition unit, a calculation unit, a data transmission unit, and a terminal unit; wherein, the calculation unit is used to execute a speech denoising method based on complex spectrum;

[0056] The acquisition unit is used to convert the analog speech signal into a digital signal, obtain the original speech signal, and transmit it to the computing unit.

[0057] The data transmission unit is used to transmit the noise-reduced signal obtained by the computing unit to the terminal unit;

[0058] The terminal unit is used to play the audio data of the noise-reduced signal. Attached Figure Description

[0059] Figure 1 : A schematic flowchart of an embodiment of a speech denoising method based on complex spectrum provided by the present invention;

[0060] Figure 2 : A flowchart of the complex form noise reduction process of an embodiment of the speech noise reduction method based on complex spectrum provided by the present invention;

[0061] Figure 3 : A flowchart illustrating the computation of a complex form Encoder, representing an embodiment of a speech denoising method based on complex spectrum provided by the present invention;

[0062] Figure 4 : A flowchart of the computation of a complex form RNN, which is an embodiment of a speech denoising method based on complex spectrum provided by the present invention;

[0063] Figure 5 : A flowchart illustrating the complex form of a decoder, representing an embodiment of a speech denoising method based on complex spectrum provided by the present invention;

[0064] Figure 6 : A denoising spectrum transformation diagram of an embodiment of a speech denoising method based on complex spectrum provided by the present invention;

[0065] Figure 7 : A schematic diagram of an embodiment of a speech noise reduction system based on complex spectrum provided by the present invention;

[0066] Figure 8 : A unit structure diagram of an embodiment of a speech noise reduction system based on complex spectrum provided by the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] Example 1

[0069] Please refer to Figure 1 This is a flowchart illustrating a speech denoising method based on complex spectrum provided by an embodiment of the present invention. The speech denoising method in this embodiment processes the complex spectrum of the original speech signal, taking into account both the amplitude and phase spectra of the original speech signal, to achieve real-time noise suppression, clearly restore the speech signal, reduce model computation, and improve denoising effect. The speech denoising method includes steps 101 to 104, each step as follows:

[0070] Step 101: Acquire the original speech signal in real time, convert the original speech signal into a complex form in the frequency domain to obtain the first complex spectrum, extract the non-DC part of the first complex spectrum to obtain the second complex spectrum.

[0071] In this embodiment, the noisy speech, i.e. the original speech signal, can be processed by STFT to obtain a complex form in the frequency domain, i.e., the first complex spectrum. Each frequency point in the complex spectrum is represented by a complex number, containing its own real and imaginary parts. The part with a frequency of 0Hz is the DC part, and the part with a frequency other than 0Hz is the non-DC part. The size of the non-DC part is FFT_n / 2, where FFT_n is the number of Fourier transform points of the STFT. As an example of this embodiment, the number of points is preferably 512. The non-DC part in the first complex spectrum is extracted, i.e., the second complex spectrum, and then the second complex spectrum is used as the input of a preset complex operation model.

[0072] Step 102: Using a preset complex number operation model, perform noise estimation on the second complex number spectrum to obtain the first complex number mask; wherein, the preset complex number operation model includes a preset N-layer complex form Encoder, a preset M-layer complex form RNN, and a preset N-layer complex form Decoder.

[0073] In this embodiment, noise estimation is performed using a preset complex number operation model, i.e., complex number form operations, to ultimately achieve noise reduction. The complex number form noise reduction process is as follows: Figure 2As shown, in real-time speech denoising, the original speech signal is acquired in real time and converted to the frequency domain. Based on the real and imaginary parts of the original speech signal in the frequency domain, noise estimation is performed using a preset complex number operation model to obtain masks for the real and imaginary parts. These masks are then combined to form a complex mask, which is applied to the complex spectrum of the original speech. Finally, the complex spectrum is restored to a time-domain signal, and the denoised speech is output. The complex number operation model, based on the Unet structure, uses the complex spectrum as input and modifies the real-number encoder, RNN, and decoder into complex-number encoders, RNNs, and decoders according to complex number operations. The complex spectrum of the non-DC portion is used as input to the complex number operation model. After passing through N layers of complex-number encoders, M layers of complex-number RNNs, and N layers of complex-number decoders, a complex mask of the same size as the input is obtained, where N and M are both integers greater than 1. As an example in this embodiment, N is preferably 4, and M is preferably 2. The number of input channels for the complex form of Encoderri is equal to the number of output channels for the complex form of Encoderri-1. Specifically, the number of input channels for the complex form of Encoder1 is 2, corresponding to the real and imaginary parts of the complex spectrum. As an example of this embodiment, the number of output channels for complex forms of Encoder1 to Encoder4 is preferably 16, 32, 64, and 64. The number of input channels for the complex form of Decoderri is the sum of the number of output channels for the complex form of Decoderri+1 and the number of output channels for the complex form of Encoderri+1. As an example of this embodiment, the number of output channels for complex forms of Decoder1 to Decoder4 is preferably 2, 16, 32, and 64. The number of input nodes of the complex form RNN1 is equal to the output size of Encodern, and the number of hidden nodes is P. As an example of this embodiment, the number of hidden nodes is preferably 128. The number of input nodes and the number of hidden nodes of the complex form RNN2 are the same as the number of hidden nodes of the complex form RNN1. At the same time, the last layer of the complex form RNN contains a fully connected layer, wherein the number of output nodes of the fully connected layer is the same as the number of input nodes of the first layer of the complex form RNN.

[0074] Optionally, step 102 specifically includes steps 1021 to 1023, each of which is as follows:

[0075] Step 1021: Input the second complex spectrum into a pre-defined N-layer complex form Encoder and perform convolutional encoding to obtain the fourth complex spectrum.

[0076] Step 1021 specifically involves: splitting the second complex spectrum into a first real part and a first imaginary part; performing cross-convolution operations on the first real part and the first imaginary part through depthwise convolution and pointwise convolution to obtain real part convolution and imaginary part convolution, and performing nonlinear activation of the neural network on the real part convolution and imaginary part convolution to obtain the fourth complex spectrum; wherein the number of input channels, the number of output channels, and the number of groups of the depthwise convolution are the same as the number of input channels of the current layer of the depthwise convolution, and the number of output channels of the pointwise convolution are the same as the number of output channels of the current layer of the pointwise convolution.

[0077] Optionally, the first real part is input into a depthwise convolution to perform depthwise convolution operations, obtaining a first real part depthwise convolution and a first imaginary part depthwise convolution. The first real part depthwise convolution and the first imaginary part depthwise convolution are then input into a real part pointwise convolution to perform pointwise convolution operations, obtaining a real part convolution. Herein, the depthwise convolution includes a real part depthwise convolution and an imaginary part depthwise convolution. The first imaginary part is input into a depthwise convolution to perform depthwise convolution operations, obtaining a second real part depthwise convolution and a second imaginary part depthwise convolution. The second real part depthwise convolution and the second imaginary part depthwise convolution are then input into an imaginary part pointwise convolution to perform pointwise convolution operations, obtaining an imaginary part convolution. The real part convolution and the imaginary part convolution are concatenated to obtain a sixth complex spectrum. The sixth complex spectrum is then passed through a BN layer and a nonlinear activation layer to obtain a fourth complex spectrum.

[0078] In this embodiment, the computation process of the complex form of the Encoder is as follows: Figure 3As shown, in a pre-defined N-layer complex encoder, the convolutional unit consists of real part depthwise convolution, real part pointwise convolution, imaginary part depthwise convolution, imaginary part pointwise convolution, BN layer, and nonlinear activation layer. The order of depthwise convolution followed by pointwise convolution is adopted. The number of input channels, the number of output channels, and the number of convolution groups of the depthwise convolution are all consistent with the number of input channels of the current layer. The number of output channels of the pointwise convolution is consistent with the number of output channels of the current layer. As an example of this embodiment, the number of groups is preferably 1. Using the second complex spectrum as the input part of the complex form encoder, the input part is split into real and imaginary inputs on the channel, namely the first real part and the first imaginary part. In the cross-input real-part depthwise convolution and imaginary-part depthwise convolution, the first real part is input to the real-part depthwise convolution, labeled 1, resulting in the first real-part depthwise convolution 1'; the first real part is input to the imaginary-part depthwise convolution, labeled 2, resulting in the first imaginary-part depthwise convolution 2'; and the first imaginary part is input to the real-part depthwise convolution, labeled 3, resulting in the second real-part depthwise convolution 3'. The input is an imaginary depthwise convolution, labeled 4, which yields the second imaginary depthwise convolution 4'. Then, the corresponding real and imaginary point convolutions are input separately, resulting in 1”, 2”, 3”, and 4”. The real convolution output is 1”+4”, and the imaginary convolution output is 2”-3”. The real and imaginary convolutions are then concatenated on the channel, and then passed through a BN layer and a nonlinear activation layer to obtain the output of the complex form encoder. After calculation by an N-layer complex form encoder, the fourth complex spectrum is obtained. As an example of this embodiment, the kernel size of the real part depthwise convolution and the imaginary part depthwise convolution is preferably (5, 2), the stride is preferably (2, 1), the convolution group is preferably the same as the number of its input channels, and its output channels are the same as the input channels. The kernel size of the real part pointwise convolution and the imaginary part pointwise convolution is preferably (1, 1), the stride is preferably (1, 1), the convolution group is preferably 1, and its output channels are the same as the output channels of this layer. The activation function of the nonlinear activation layer is preferably the PReLU function, with the following formula:

[0079]

[0080] Step 1022: Input the fourth complex spectrum into a preset M-layer complex RNN and perform cross-neural network operations to obtain the fifth complex spectrum.

[0081] Step 1022 specifically involves: splitting the fourth complex spectrum into a second real part and a second imaginary part; inputting the second real part into the real part RNN and the imaginary part RNN, performing RNN neural network operations to obtain the first real part RNN and the first imaginary part RNN, and obtaining the real part RNN based on the first real part RNN and the first imaginary part RNN; inputting the second imaginary part into the real part RNN and the imaginary part RNN, performing RNN neural network operations to obtain the second real part RNN and the second imaginary part RNN, and obtaining the imaginary part RNN based on the second real part RNN and the second imaginary part RNN; and mapping the outputs of the real part RNN and the imaginary part RNN according to the fully connected layers in the preset M-layer complex form RNN to obtain the fifth complex spectrum.

[0082] In this embodiment, the computation process of the complex form of the RNN is as follows: Figure 4 As shown, the output of the complex form encoder is used as the input of the complex form RNN. The fourth complex spectrum is then used as the input part of the complex form RNN. The input part is split into real and imaginary inputs on the channel, namely the second real part and the second imaginary part. The real and imaginary inputs are cross-inputted into the real and imaginary RNNs. The second real part is input into the real RNN, labeled 1, thus obtaining the first real RNN (1'). The second real part is input into the imaginary RNN, labeled 2, thus obtaining the first imaginary RNN (2'). The second imaginary part is input into the real RNN, labeled 3, thus obtaining the... The real part RNN (3') is input to the second imaginary part RNN, which is labeled as 4, thus obtaining the second imaginary part RNN (4'). The output of the real part RNN is 1'+4', and the output of the imaginary part RNN is 2'-3'. The fully connected layer exists only in the last layer of the complex form RNN and is used to map the final output to the same size as the input of the first layer of the complex form RNN. After calculation by the M layers of complex form RNN, the fifth complex spectrum is obtained. As an example of this embodiment, the RNN is preferably LSTM, and the hidden nodes of both the real part RNN and the imaginary part RNN are preferably 128.

[0083] Step 1023: Input the fifth complex spectrum into a preset N-layer complex form Decoder, perform deconvolution decoding, and obtain the first complex mask.

[0084] Step 1023 specifically involves: splitting the spectrum of the fifth complex number into a third real part and a third imaginary part; performing cross-deconvolution operations on the third real part and the third imaginary part through point deconvolution and depth deconvolution to obtain real part deconvolution and imaginary part deconvolution, and performing nonlinear activation of the neural network on the real part deconvolution and imaginary part deconvolution to obtain the first complex number mask; wherein, the number of input channels of the point deconvolution is the same as the number of input channels of the current layer of the point deconvolution, the number of output channels of the point deconvolution is the same as the number of output channels of the current layer of the point deconvolution, and the number of input channels, output channels, and groups of the depth deconvolution are the same as the number of output channels of the current layer of the depth deconvolution.

[0085] Optionally, the third real part is input into the point deconvolution layer for point deconvolution operation to obtain the first real part point deconvolution layer and the second real part point deconvolution layer. The first real part point deconvolution layer and the second real part point deconvolution layer are then cross-inputted into the real part depth deconvolution layer and the imaginary part depth deconvolution layer for depth deconvolution operation to obtain the real part deconvolution layer. Herein, the point deconvolution layer includes the real part point de-depth convolution layer and the imaginary part point de-depth convolution layer. The third imaginary part is input into the point deconvolution layer for point deconvolution operation to obtain the first imaginary part point deconvolution layer and the second imaginary part point deconvolution layer. The first imaginary part point deconvolution layer and the second imaginary part point deconvolution layer are then cross-inputted into the real part depth deconvolution layer and the imaginary part depth deconvolution layer for depth deconvolution operation to obtain the imaginary part deconvolution layer. The real part deconvolution layer and the imaginary part deconvolution layer are concatenated to obtain the seventh complex spectrum. The seventh complex spectrum is then passed through a BN layer and a nonlinear activation layer to obtain the first complex mask.

[0086] In this embodiment, the computation process of the complex form of the Decoder is as follows: Figure 5As shown, in the preset N-layer complex form Decoder, the deconvolution unit consists of real part point deconvolution, real part depth deconvolution, imaginary part point deconvolution, imaginary part depth deconvolution, BN layer, and nonlinear activation layer. The order is point deconvolution followed by depth deconvolution. The number of input channels and output channels of the point deconvolution are the same as the current layer, and the number of groups is 1. The number of input channels, output channels, and deconvolution groups of the depth deconvolution are all the same as the current layer's output channels. Using the output of the complex form RNN as the input of the complex form Decoder, the fifth complex spectrum becomes the complex form Decoder. The input part of er is split into real and imaginary inputs on the channel, namely the third real part and the third imaginary part. The real and imaginary inputs are respectively deconvolved at the real part and the imaginary part to obtain the real part deconvolution result (first real part deconvolution 1 and second real part deconvolution 2) and the imaginary part deconvolution result (second imaginary part deconvolution 3 and second imaginary part deconvolution 4). As an example of this embodiment, the kernel size of the real and imaginary point deconvolutions is preferably (1,1), the stride is preferably (1,1), and the deconvolution group is preferably 1. Its input channel is consistent with the input channel of the layer, and its output channel is consistent with the output channel of the layer. The real part deconvolution result and the imaginary part deconvolution result are cross-inputted into the real part depth deconvolution and the imaginary part depth deconvolution, and the convolution results are 1', 2', 3', and 4'. The real part deconvolution output is the result of 1'+4', and the imaginary part deconvolution output is the result of 2'-3'. As an example of this embodiment, the kernel size of the real part depthwise deconvolution and the imaginary part depthwise deconvolution is preferably (5, 2), the stride is preferably (2, 1), the number of deconvolution groups, the number of input channels, and the number of output channels are consistent, preferably consistent with the number of output channels of this layer. The outputs of the real part deconvolution and the imaginary part deconvolution are then concatenated on the channels. Finally, the output corresponding to the complex form decoder is obtained after passing through a BN layer and a nonlinear activation layer. The first complex mask is obtained through calculation by N layers of complex form decoders. As an example of this embodiment, the activation function of the nonlinear activation layer is preferably the PReLU function, with the following formula:

[0087]

[0088] Step 103: Based on the real and imaginary parts of the first complex mask, the amplitude spectrum and phase spectrum of the original speech signal, perform noise reduction spectrum transformation to obtain the third complex spectrum, and convert the third complex spectrum into time domain form to obtain the noise reduction signal, thereby achieving speech noise reduction.

[0089] In this embodiment, the noise reduction spectrum conversion, such as Figure 6As shown, the mask output by the complex number operation model is a complex mask, specifically including a real part mask and an imaginary part mask. The real part mask and the imaginary part mask are converted into amplitude spectrum mask and phase spectrum bias, and then applied to the amplitude spectrum and phase spectrum of the noisy speech. They are then converted back to real part and imaginary part form, and finally converted into time-domain denoised speech through ISTFT.

[0090] Optionally, step 103 specifically includes steps 1031 to 1034, each of which is as follows:

[0091] Step 1031: Obtain the real part mask and the imaginary part mask based on the real part and the imaginary part of the first complex mask.

[0092] In this embodiment, the complex mask is the output of Decoder1, that is, the output calculated by N layers of complex form Decoder, which is the first complex mask, including the real part mask and the imaginary part mask.

[0093] Step 1032: Perform offset calculations on the real part mask and the imaginary part mask to obtain the amplitude spectrum mask and the phase spectrum offset. Based on the amplitude spectrum mask, the phase spectrum offset, and the amplitude spectrum and phase spectrum of the original speech signal, perform noise reduction amplitude and phase calculations to obtain the first amplitude spectrum and the first phase spectrum.

[0094] Optionally, step 1032 specifically involves: performing amplitude normalization transformation on the real part mask and the imaginary part mask to obtain the amplitude spectrum mask, wherein the amplitude normalization transformation formula is as follows:

[0095]

[0096] Among them, mags mask For amplitude spectrum mask, real mask For the real part mask, imag mask For the imaginary part mask;

[0097] The real part mask and the imaginary part mask are transformed by phase bias to obtain the phase spectrum bias. The phase bias transformation formula is as follows:

[0098]

[0099] Among them, phase bias For phase spectrum bias;

[0100] Based on the amplitude spectrum and amplitude spectrum mask of the original speech signal, the noise reduction amplitude is calculated to obtain the first amplitude spectrum; the formula for calculating the noise reduction amplitude is as follows:

[0101] mags denoisy =mags noisy ·mags mask

[0102] Among them, mags denoisy For the first amplitude spectrum, mags noisy The amplitude spectrum of the original speech signal;

[0103] Based on the phase spectrum and phase spectrum offset of the original speech signal, noise reduction phase calculation is performed to obtain the first phase spectrum; the noise reduction phase calculation formula is as follows:

[0104] phase denoisy =phase noisy +phase bias

[0105] Among them, phase denoisy For the first phase spectrum, phase noisy This is the phase spectrum of the original speech signal.

[0106] In this embodiment, the real part mask and the imaginary part mask are first converted into an amplitude spectrum mask and a phase spectrum bias, and the amplitude spectrum mask is normalized to ensure that the value range of the amplitude spectrum mask is [0, 1]. Then, the calculated normalized amplitude spectrum mask and phase spectrum bias are applied to the amplitude spectrum and phase spectrum of the noisy speech to obtain the amplitude spectrum (first amplitude spectrum) and phase spectrum (first phase spectrum) of the denoised speech.

[0107] Step 1033: Perform a spectrum transformation between the first phase spectrum and the first amplitude spectrum to obtain the fourth real part and the fourth imaginary part. Based on the fourth real part and the fourth imaginary part, obtain the third complex spectrum.

[0108] In this embodiment, the amplitude spectrum (first amplitude spectrum) and phase spectrum (first phase spectrum) of the denoised speech are converted into real and imaginary parts, namely the fourth real part and the fourth imaginary part, by a spectrum transformation. The spectrum transformation formula is as follows:

[0109]

[0110] Among them, real denoisy For the fourth real part, imag denoisy It is the fourth imaginary part.

[0111] Step 1034: Based on ISTFT, the third complex spectrum is converted into a time-domain form to obtain the noise-reduced signal and achieve speech noise reduction.

[0112] The system acquires the original speech signal in real time and converts it into a complex frequency domain form to obtain a first complex spectrum. The non-DC portion of the first complex spectrum is extracted to obtain a second complex spectrum. A pre-defined complex operation model (complex Encoder, complex RNN, and complex Decoder) is used to estimate noise in the second complex spectrum to obtain a first complex mask. The pre-defined complex operation model includes a pre-defined N-layer complex Encoder, a pre-defined M-layer complex RNN, and a pre-defined N-layer complex Decoder. The Unet structure (Encoder+RNN+Decoder) using complex operations significantly improves the noise reduction effect of the complex operation model, especially at low signal-to-noise ratios, greatly enhancing the ability to reproduce human voice. The use of complex Encoder and Decoder implementations significantly reduces the computational load of the model and the complexity of model learning. Based on the real and imaginary parts of the first complex mask and the amplitude and phase spectra of the original speech signal, a denoising spectrum transformation is performed to obtain a third complex spectrum. This third complex spectrum is then converted into a time-domain form to obtain the denoised signal, thus achieving speech denoising. The denoising spectrum transformation using the real and imaginary parts of the first complex mask takes into account both the amplitude and phase spectra of the original speech signal, avoiding the neglect of phase information during original speech signal processing. This results in the restoration of an ideal, clean speech signal, improving the denoising effect. It can process noisy speech signals in real time and can be deployed in real-time conferencing communication equipment for real-time noise reduction and suppression of speech signals.

[0113] Example 2

[0114] Accordingly, see Figure 7 , Figure 7 This is a structural diagram of a second embodiment of a speech noise reduction system based on complex spectrum provided by the present invention. Figure 7 As shown, the speech noise reduction system based on complex spectrum includes: acquisition unit 701, calculation unit 702, data transmission unit 703 and terminal unit 704.

[0115] The acquisition unit 701 is used to convert the analog speech signal into a digital signal, obtain the original speech signal, and transmit it to the computing unit 702.

[0116] In this embodiment, the unit structure of the speech denoising system based on complex spectrum is as follows: Figure 8 As shown, the acquisition unit 701 includes, but is not limited to, a microphone and an ADC hardware chip, used to convert analog voice signals in the environment into digital signals.

[0117] The computing unit 702 is used to execute a speech denoising method based on complex spectrum.

[0118] In this embodiment, the computing unit 702 includes, but is not limited to, a single-chip microcomputer or a computing chip with an operating system, which is used for the calculation of the noise reduction model, to obtain the original speech signal from the acquisition unit 701 in real time, obtain the noise reduction signal, and input the noise reduction signal into the data transmission unit 703.

[0119] The data transmission unit 703 is used to transmit the noise-reduced signal obtained by the computing unit to the terminal unit;

[0120] In this embodiment, the data transmission unit 703 includes, but is not limited to, a network system capable of transmitting data for transmitting the calculated data.

[0121] The terminal unit 704 is used to play the audio data of the noise-reduced signal.

[0122] In this embodiment, the terminal unit 704 includes, but is not limited to, any real-time conferencing communication device with network access, used to play noise-reduced audio data.

[0123] It should be noted that this system can be directly applied to microphones to improve the final output audio quality. At the same time, the voice noise reduction method is not limited to various hardware devices and can be extended to the PC, existing independently as a noise reduction algorithm on the PC, for example, as an independent noise reduction algorithm in meetings.

[0124] This invention proposes a lightweight speech denoising system based on complex spectrum. While meeting real-time requirements, it effectively suppresses noise under low signal-to-noise ratio (SNR) conditions and restores clear human voice. The proposed complex spectrum-based speech denoising method is designed to be lightweight and can be deployed in real-time conferencing communication devices for real-time speech denoising. The Unet structure, employing complex operations, significantly improves the model's denoising performance, especially in restoring human voice at low SNR. Secondly, the special Encoder and Decoder implementations drastically reduce the model's computational load, enabling real-time operation on lower-performance platforms. Finally, the complex mask conversion to amplitude and phase spectra reduces the complexity of model learning, facilitating model learning. The entire system can be deployed in real-time conferencing communication devices for real-time speech denoising and suppression.

[0125] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method of speech noise reduction based on complex spectrum, characterized by, include: The original speech signal is acquired in real time and converted into a complex form in the frequency domain to obtain a first complex spectrum. The non-DC part of the first complex spectrum is extracted to obtain a second complex spectrum. A first complex mask is obtained by noise estimation of the second complex spectrum through a preset complex number operation model; wherein, the preset complex number operation model includes a preset N-layer complex form Encoder, a preset M-layer complex form RNN, and a preset N-layer complex form Decoder; Based on the real and imaginary parts of the first complex mask and the amplitude and phase spectra of the original speech signal, a noise reduction spectrum transformation is performed to obtain a third complex spectrum, and the third complex spectrum is converted into a time-domain form to obtain a noise-reduced signal, thereby achieving speech noise reduction. The step of obtaining the first complex mask by performing noise estimation on the second complex spectrum through a preset complex number operation model is as follows: The second complex spectrum is input into the preset N-layer complex form Encoder for convolutional encoding to obtain the fourth complex spectrum; The fourth complex spectrum is input into the preset M-layer complex RNN for cross-neural network operation to obtain the fifth complex spectrum. Specifically, the cross-neural network operation involves: splitting the fourth complex spectrum into a second real part and a second imaginary part; inputting the second real part into the real part RNN and the imaginary part RNN for RNN neural network operation to obtain a first real part RNN and a first imaginary part RNN, and obtaining a real part RNN based on the first real part RNN and the first imaginary part RNN; inputting the second imaginary part into the real part RNN and the imaginary part RNN for RNN neural network operation to obtain a second real part RNN and a second imaginary part RNN, and obtaining an imaginary part RNN based on the second real part RNN and the second imaginary part RNN; and mapping the outputs of the real part RNN and the imaginary part RNN using the fully connected layers in the preset M-layer complex RNN to obtain the fifth complex spectrum. The fifth complex number spectrum is input into the preset N-layer complex number form Decoder for deconvolution decoding to obtain the first complex number mask.

2. The complex spectrum based speech denoising method as claimed in claim 1, wherein, The step of inputting the second complex spectrum into the preset N-layer complex form Encoder for convolutional encoding to obtain the fourth complex spectrum specifically involves: The spectrum of the second complex number is split into a first real part and a first imaginary part; By performing cross-convolution operations on the first real part and the first imaginary part through depthwise convolution and pointwise convolution, the real part convolution and the imaginary part convolution are obtained. The real part convolution and the imaginary part convolution are then subjected to nonlinear activation of the neural network to obtain the spectrum of the fourth complex number. The number of input channels, output channels, and groups of the depthwise convolution are the same as the number of input channels of the current layer of the depthwise convolution, and the number of output channels of the pointwise convolution is the same as the number of output channels of the current layer of the pointwise convolution.

3. The complex spectrum based speech denoising method of claim 2, wherein, The process involves performing cross-convolution operations on the first real part and the first imaginary part using depthwise convolution and pointwise convolution to obtain real part convolution and imaginary part convolution. Then, the real part convolution and the imaginary part convolution are subjected to nonlinear activation of the neural network to obtain the fourth complex number spectrum. Specifically: The first real part is input into the depthwise convolution to perform depthwise convolution operation, thereby obtaining the first real part depthwise convolution and the first imaginary part depthwise convolution. The first real part depthwise convolution and the first imaginary part depthwise convolution are then input into the real part pointwise convolution to perform pointwise convolution operation, thereby obtaining the real part convolution. The depthwise convolution includes the real part depthwise convolution and the imaginary part depthwise convolution. The first imaginary part is input into the depthwise convolution, and the depthwise convolution operation is performed to obtain the second real part depthwise convolution and the second imaginary part depthwise convolution. The second real part depthwise convolution and the second imaginary part depthwise convolution are then input into the imaginary part pointwise convolution, and the pointwise convolution operation is performed to obtain the imaginary part convolution. The real part convolution and the imaginary part convolution are concatenated to obtain the sixth complex spectrum. The sixth complex spectrum is then passed through a BN layer and a nonlinear activation layer to obtain the fourth complex spectrum.

4. The complex spectrum based speech denoising method as claimed in claim 1, wherein, The step of inputting the fifth complex spectrum into the preset N-layer complex form Decoder for deconvolution decoding to obtain the first complex mask is specifically as follows: The spectrum of the fifth complex number is split into a third real part and a third imaginary part; By performing point deconvolution and depth deconvolution on the third real part and the third imaginary part, we obtain real part deconvolution and imaginary part deconvolution. Then, we perform neural network nonlinear activation on the real part deconvolution and the imaginary part deconvolution to obtain the first complex mask. Wherein, the number of input channels of the point deconvolution is the same as the number of input channels of the current layer of the point deconvolution, the number of output channels of the point deconvolution is the same as the number of output channels of the current layer of the point deconvolution, and the number of input channels, the number of output channels, and the number of groups of the depth deconvolution are the same as the number of output channels of the current layer of the depth deconvolution.

5. The complex spectrum based speech denoising method of claim 4, wherein, The method involves performing cross-deconvolution operations on the third real part and the third imaginary part using point deconvolution and depth deconvolution to obtain real part deconvolution and imaginary part deconvolution. Then, the real part deconvolution and the imaginary part deconvolution are subjected to nonlinear activation of a neural network to obtain the first complex mask. Specifically: The third real part is input into the point deconvolution to perform point deconvolution operation, obtaining the first real part point deconvolution and the second real part point deconvolution. The first real part point deconvolution and the second real part point deconvolution are then cross-inputted into the real part depth deconvolution and the imaginary part depth deconvolution to perform depth deconvolution operation, obtaining the real part deconvolution. The point deconvolution includes real part point de-depth convolution and imaginary part point de-depth convolution. The third imaginary part is input into the point deconvolution, and the point deconvolution operation is performed to obtain the first imaginary part point deconvolution and the second imaginary part point deconvolution. The first imaginary part point deconvolution and the second imaginary part point deconvolution are cross-inputted into the real part depth deconvolution and the imaginary part depth deconvolution, and the depth deconvolution operation is performed to obtain the imaginary part deconvolution. The real part deconvolution and the imaginary part deconvolution are concatenated to obtain the seventh complex spectrum. The seventh complex spectrum is then passed through a BN layer and a nonlinear activation layer to obtain the first complex mask.

6. The complex spectrum based speech denoising method as claimed in claim 1, wherein, The process involves performing a noise reduction spectrum transformation based on the first complex mask, the amplitude spectrum, and the phase spectrum of the original speech signal to obtain a third complex spectrum, and then converting the third complex spectrum into a time-domain form to obtain a noise-reduced signal, thereby achieving speech noise reduction. Specifically: Based on the real and imaginary parts of the first complex mask, obtain the real part mask and the imaginary part mask; The real part mask and the imaginary part mask are offset to obtain the amplitude spectrum mask and the phase spectrum offset. Based on the amplitude spectrum mask, the phase spectrum offset, and the amplitude spectrum and phase spectrum of the original speech signal, the noise reduction amplitude and phase are calculated to obtain the first amplitude spectrum and the first phase spectrum. The first phase spectrum and the first amplitude spectrum are subjected to a spectrum real-to-virtual conversion to obtain a fourth real part and a fourth imaginary part. Based on the fourth real part and the fourth imaginary part, the third complex spectrum is obtained. According to ISTFT, the third complex spectrum is converted into a time-domain form to obtain a noise-reduced signal, thereby achieving speech noise reduction.

7. The complex spectrum based speech denoising method as claimed in claim 6, wherein, The process involves offsetting the real and imaginary part masks to obtain an amplitude spectrum mask and a phase spectrum offset. Then, based on the amplitude spectrum mask, the phase spectrum offset, and the amplitude and phase spectra of the original speech signal, noise reduction amplitude and phase calculations are performed to obtain a first amplitude spectrum and a first phase spectrum. Specifically: The amplitude spectrum mask is obtained by performing amplitude normalization transformation on the real part mask and the imaginary part mask, wherein the amplitude normalization transformation formula is as follows: wherein, is the magnitude spectrum mask, is the real part mask, is the imaginary part mask; The phase spectrum bias is obtained by performing a phase bias transformation on the real part mask and the imaginary part mask, wherein the phase bias transformation is performed by the following formula: wherein is the phase spectrum bias; Based on the amplitude spectrum of the original speech signal and the amplitude spectrum mask, the noise reduction amplitude is calculated to obtain the first amplitude spectrum; wherein, the noise reduction amplitude calculation formula is as follows: in, This is the first amplitude spectrum. The amplitude spectrum of the original speech signal; Based on the phase spectrum of the original speech signal and the phase spectrum bias, a noise reduction phase calculation is performed to obtain the first phase spectrum; wherein, the noise reduction phase calculation formula is as follows: in, This is the first phase spectrum. The phase spectrum of the original speech signal.

8. A speech noise reduction system based on complex spectrum, characterized in that, include: The system comprises an acquisition unit, a computing unit, a data transmission unit, and a terminal unit; wherein the computing unit executes the speech denoising method based on complex spectrum as described in any one of claims 1 to 7. The acquisition unit is used to convert the analog speech signal into a digital signal to obtain the original speech signal, and then transmit it to the computing unit. The data transmission unit is used to transmit the noise-reduced signal obtained by the computing unit to the terminal unit; The terminal unit is used to play the audio data of the noise-reduced signal.

Images