A multi-channel noise suppression method based on two-stage deep complex network

Abstract

The present invention discloses a multi-channel noise suppression method based on a two-stage deep complex network, comprising constructing a deep complex gated convolutional attention network to perform correlation modeling on complex spectra; utilizing a two-stage deep complex network to enhance multi-channel noisy signals; in the beamforming stage, performing amplitude and phase enhancement on the multi-channel complex spectrum to obtain a coarsely denoised single-channel complex spectrum; and in the post-filtering stage, performing refinement on the coarsely denoised single-channel complex spectrum to further suppress noise. The method of the present invention divides the noise suppression task into two stages, beamforming and post-filtering, through a two-stage enhancement strategy, effectively avoiding the problems of residual noise and distortion of the target acoustic signal. In addition, by adopting a complex network structure in both stages to model the complex spectrum, the amplitude and phase information of the signal are simultaneously enhanced, phase distortion is reduced, and the quality and naturalness of the signal are improved.

Description

Technical Field

[0001] The present invention relates to the field of sound signal processing, and in particular to a multi-channel noise suppression method based on a two-stage deep complex network. Background Art

[0002] In modern voice communication, automatic speech recognition (ASR), and acoustic event detection applications, clear target sound signal quality is crucial. However, in real-world scenarios, the target sound signal collected by a microphone array is often interfered with by noise, resulting in a decrease in signal quality and clarity. Noise suppression technology aims to reduce the impact of noise on the target sound signal and extract the clean target sound from the noisy signal collected by the microphone, which is a mixture of target sound and noise. Multi-channel noise suppression methods utilize the spatial information obtained by a microphone array composed of multiple microphones to selectively enhance or suppress signals in certain directions. This can improve the quality of the target sound signal to a certain extent and has gradually become a hot topic of current research.

[0003] In recent years, multi-channel noise suppression methods based on deep learning have made significant progress in performance, which can be mainly divided into two types: one is an end-to-end method based on deep neural networks; the other is a method combining beamforming with deep neural networks.

[0004] End-to-end noise suppression methods based on deep neural networks train the network to estimate the mapping function from a mask or noisy signal to the target signal, thereby obtaining an enhanced signal. While end-to-end methods demonstrate good noise reduction capabilities, they do not impose a distortion-free constraint, and nonlinear distortion is inevitably introduced during the noise reduction process. Combining beamforming with deep neural networks uses a neural network to estimate beamforming weights or the mask used, and then performs spatial filtering through beamforming to obtain an enhanced signal. This method effectively suppresses noise from other directions through directional filtering without destroying the original structure of the target signal, eliminating or reducing the problem of nonlinear distortion.

[0005] The complex spectrum can be decomposed into real and imaginary parts in Cartesian coordinates and into amplitude and phase in polar coordinates. Amplitude determines the intensity of the sound signal, while phase describes the temporal relationship and structure of the sound waveform. Together, they influence the quality and naturalness of the sound. However, modeling the correlation information in the complex spectrum is difficult in multi-channel noise suppression, and improper modeling can lead to loss of signal phase information. Existing multi-channel noise suppression methods simply enhance the amplitude of the noisy signal or are limited to reconstructing phase information from the complex spectrum of the noisy signal using real-valued neural network models. These methods fail to capture the complex time-frequency relationships in the noisy signal. This, to a certain extent, limits the application of multi-channel noise suppression algorithms in complex noise environments. In the presence of severe noise interference, the estimated phase of the target acoustic signal will exhibit significant deviations. If the phase information is inaccurate, the target acoustic signal will still appear unnatural or blurred, even if the amplitude spectrum is fully restored. Furthermore, statistical information such as the spectrum or mask in multi-channel signals is difficult to accurately estimate, resulting in a significant amount of residual noise after filtering. Summary of the Invention

[0006] The purpose of the present invention is to provide a multi-channel noise suppression method based on a two-stage deep complex network, which can reasonably model the correlation information in the complex spectrum of the multi-channel noisy signal, and adopt a two-stage enhancement strategy to realize the amplitude and phase enhancement of the signal, restore the phase information of the target sound signal, solve the problem of phase distortion, reduce the distortion of the target sound signal, suppress the residual noise in the beamforming stage, and effectively improve the signal quality.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A multi-channel noise suppression method based on a two-stage deep complex network includes the following steps:

[0009] S1: Use short-time Fourier transform to transform the multi-channel noisy signal received by the microphone array into the time-frequency domain and perform spectrum compression to obtain a compressed multi-channel noisy complex spectrum, and select one of the channels as the reference channel;

[0010] S2: Constructing a deep complex gated convolutional attention network, wherein the deep complex gated convolutional attention network includes a plurality of complex encoder modules, a plurality of complex decoder modules, a complex Conformer module, and a plurality of encoder-decoder attention modules; the plurality of complex encoder modules, the complex Conformer modules, and the plurality of complex decoder modules are sequentially connected, and each complex encoder module is skip-connected to the corresponding complex decoder module through the encoder-decoder attention module;

[0011] S3: Using a deep complex gated convolutional attention network to build a two-stage deep complex network consisting of a beamforming stage and a post-filtering stage;

[0012] S4: The compressed multi-channel noisy complex spectrum is taken as input and passed through the beamforming stage consisting of a deep complex gated convolutional attention network, a complex mask generator and a minimum variance distortionless response beamformer to obtain a coarsely denoised single-channel complex spectrum;

[0013] S5: The coarsely denoised single-channel complex spectrum and the noisy complex spectrum of the reference channel are taken as input, and the final enhanced complex spectrum is obtained through a post-filtering stage consisting of a deep complex gated convolutional attention network and a complex mask generator;

[0014] S6: The enhanced complex spectrum is subjected to spectrum decompression and inverse short-time Fourier transform to obtain the enhanced target signal in the time domain.

[0015] Furthermore, in step S1, the short-time Fourier transform is converted into the time-frequency domain and spectrum compression is performed. The spectrum compression method is expressed as:

[0016]

[0017] Among them, Y ori represents the complex spectrum after short-time Fourier transform, |Y ori | and represent the amplitude and phase angle of the complex spectrum after short-time Fourier transform, respectively.

[0018] Furthermore, the complex encoder module includes gated convolution, complex batch normalization and PreLU activation function; the complex decoder module includes gated deconvolution, complex batch normalization and PreLU activation function.

[0019] Furthermore, the complex number Conformer module includes two Conformer modules, namely the real part (r) and the imaginary part (i) processing parts, where each Conformer module includes a convolution module, a multi-head self-attention module, a feedforward module and layer normalization connected in sequence.

[0020] Furthermore, the encoding and decoding attention module is connected between each complex decoder module and the corresponding complex encoder module, takes the feature information from the complex decoder module and the corresponding complex encoder module as input, fuses them to obtain high-dimensional features, and splices the fused high-dimensional features with the feature information input by the complex decoder module as output.

[0021] Furthermore, the two-stage deep complex network has a first stage of beamforming, which performs coarse denoising on multi-channel noisy input to obtain a coarsely denoised single-channel complex spectrum; and a second stage of post-filtering, which further enhances the output of the first stage, suppresses residual noise, and obtains a final enhanced complex spectrum.

[0022] Furthermore, the beamforming stage includes a deep complex gated convolutional attention network, a complex mask generator and a minimum variance distortionless response beamformer connected in sequence, wherein the deep complex gated convolutional attention network is used to model the multi-channel noisy complex spectrum; the complex mask generator is used to generate a set of complex ideal ratio masks, so as to obtain the weight coefficients of the minimum variance distortionless response beamformer; the minimum variance distortionless response beamformer uses the weight coefficients to suppress the multi-channel noisy complex spectrum to obtain a coarsely denoised single-channel complex spectrum.

[0023] Furthermore, the post-filtering stage includes a deep complex gated convolutional attention network and a complex mask generator connected in sequence, wherein the deep complex gated convolutional attention network is used to model the single-channel complex spectrum of the coarse denoising; the complex mask generator is used to generate a complex ideal ratio mask, and the single-channel complex spectrum of the coarse denoising is multiplied with it to obtain the final enhanced complex spectrum.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) The present invention constructs a deep complex gated convolutional attention network. By introducing a complex Conformer module, a complex encoder, and a complex decoder module, the correlation between the real and imaginary parts of the complex spectrum of the noisy signal is modeled, which can simultaneously enhance the amplitude and phase information, thereby effectively reducing phase distortion. In addition, the encoder-decoder attention module is introduced to enhance the complex decoder's ability to perceive detailed features, thereby improving the quality of the reconstructed signal.

[0026] (2) The present invention adopts a two-stage enhancement strategy to decompose the noise suppression task into two steps: the first stage is the beamforming stage, which combines the deep complex gated convolutional attention network with the beamforming technology to perform rough noise suppression, remove the main noise and retain the original structure of the target sound signal; the second stage is the post-filtering stage, which refines the signal output from the first stage, removes the residual noise from the beamforming stage and further enhances the quality of the target signal. This staged processing method not only retains the advantage of the combination of beamforming and deep neural network in causing little damage to the original structure of the target sound signal, but also solves the problem that the output of beamforming contains a large amount of residual noise, thereby improving the effect of noise suppression and reducing the distortion of the target sound signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of the process of the multi-channel noise suppression method based on the two-stage deep complex network of the present invention;

[0028] Figure 2 Schematic diagram of the structure of the deep complex gated convolutional attention network in the embodiment;

[0029] Figure 3 Schematic diagram of the structure of the complex Conformer module in the embodiment;

[0030] Figure 4 Schematic diagram of the structure of a two-stage deep complex network in an embodiment;

[0031] Figure 5 This is a spectrum diagram after two-stage noise reduction in the embodiment;

[0032] Among them, (a) spectrum of clean speech, (b) spectrum of noisy speech, (c) spectrum after denoising in the beamforming stage, and (d) spectrum after denoising in the beamforming stage + post-filtering stage;

[0033] Figure 6 Graph showing the evaluation index results of two-stage noise reduction at different signal-to-noise ratios in the embodiment;

[0034] Among them, (a) PESQ evaluation index, (b) STOI evaluation index. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments and drawings. Obviously, the embodiments described are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] Example:

[0037] A multi-channel noise suppression method based on a two-stage deep complex network, the flow chart is as follows Figure 1 As shown, in this embodiment, a speech signal is selected as the target signal and factory noise is selected as the noise signal. The method of the present invention is also applicable when the target signal is a non-speech signal. The suppression method includes the following steps:

[0038] S1: Use short-time Fourier transform to transform the multi-channel time-domain noisy signal y(t) received by the microphone array composed of P microphones into [y1(t),…,y P (t)] T Transformed into the time-frequency domain, it is expressed as:

[0039] Y(t,f)=S(t,f)+N(t,f)=d(f)S(t,f)+N(t,f)

[0040] Where Y(t,f)=[Y1(t,f),…,Y P (t,f)] T represents the signal received by the microphone array at time frame t∈{1,…,T} and frequency f∈{1,…,F}, S(t,f) represents the target sound signal component received by the microphone array, N(t,f) represents the additive noise component, d(t,f) represents the relative transfer function of the signal, and S(t,f) represents the target sound of the reference channel. Usually, the first channel is selected as the reference channel;

[0041] The signal transformed into the time-frequency domain is spectrally compressed using a compression rate of 0.5. The compressed complex spectrum is expressed as:

[0042]

[0043] Among them, Y represents the compressed complex spectrum, Y ori represents the complex spectrum after short-time Fourier transform, Y ori | and represent the amplitude and phase angle of the complex spectrum after short-time Fourier transform, respectively.

[0044] S2: Construct a deep complex gated convolutional attention network, such as Figure 2 As shown, the deep complex gated convolutional attention network includes several complex encoder modules, several complex decoder modules, complex Conformer modules and encoding and decoding attention modules; several complex encoder modules, complex Conformer modules and several complex decoder modules are connected in sequence, and each complex encoder module is jump-connected to the corresponding complex decoder module through the encoding and decoding attention module.

[0045] like Figure 2 As shown, each complex encoder module includes gated convolution, complex batch normalization, and PreLU activation function for extracting high-level features and downsampling. The gated convolution is divided into two processing parts: the real part (r) and the imaginary part (i). The input complex spectrum is also divided into two parts: the real part and the imaginary part. In the complex encoder module, except for the first layer of complex convolution gated units, the kernel size is set to (2,5), and the kernel size of the remaining layers is set to (2,3). The step size of each layer of complex convolution gated units is set to (1,2). Through layer-by-layer convolution operations, the complex encoder gradually compresses the input spectrum signal into a more compact feature representation, removes redundant information, and retains key features. The number of channels of all complex encoder modules is set to 64.

[0046] The encoding and decoding attention module fuses the detailed feature information from the encoding layer and the deep information from the decoding layer and then inputs it into the next complex decoder to enhance the information interaction between the encoder and decoder, strengthen the decoder's ability to perceive the signal details, and thus enhance the quality of the reconstructed signal. The fused feature information is expressed as:

[0047]

[0048] in, and represents the output of the i-th layer complex encoder and the corresponding complex decoder, represents a convolution kernel, and Represent the convolution kernels of the i-th encoding layer and decoding layer respectively. and Concatenate them together and use them as the input of the next complex decoder.

[0049] like Figure 2 As shown, each complex decoder module includes gated deconvolution, complex batch normalization, and PreLU activation function for feature processing and upsampling. The gated deconvolution is divided into two processing parts: the real part (r) and the imaginary part (i). The input complex spectrum is also divided into two parts: the real part and the imaginary part. The complex encoder gradually compresses the input spectrum signal into a more compact feature representation, removes redundant information, and retains key features. In the complex decoder, except for the last layer of complex deconvolution gated units, the kernel size is set to (2, 5). The kernel size of the remaining layers is set to (2, 3). The step size of each layer of complex deconvolution gated units is set to (1, 2). The complex decoder gradually restores the high-resolution complex spectrum through layer-by-layer deconvolution. Except for the last complex decoder, the number of channels of the other complex decoders is set to 2. The number of channels of all other complex decoders is set to 64.

[0050] like Figure 3 As shown, the complex number Conformer module includes two Conformer modules, which are the real part (r) and the imaginary part (i) processing parts. The input complex spectrum is also divided into two parts, the real part and the imaginary part. Each Conformer module includes a convolution module, a multi-head self-attention module, a feedforward module and a layer normalization. The complex number Conformer module is used to simultaneously process the real part and the imaginary part of the input complex spectrum under the complex number calculation rules, model the correlation between the amplitude and phase of the signal, and enhance the model's ability to capture long-distance dependencies. The output Y of the complex number Conformer module C for:

[0051] Y C =(C r (Xr )-C i (X i ))+j(C i (X r )+C r (X i ))

[0052] Among them, X r and X i Represent the real and imaginary parts of the input complex spectrum, C r (·) and C i (·) represents the real and imaginary processing parts of the Conformer module respectively.

[0053] S3: Construct a two-stage deep complex network, which consists of two stages. The first stage is the beamforming stage, which performs coarse denoising on the multi-channel noisy input to obtain a coarsely denoised single-channel complex spectrum. The second stage is the post-filtering stage, which further enhances the output of the first stage, suppresses residual noise, and obtains the final enhanced complex spectrum.

[0054] like Figure 4 As shown, the beamforming stage consists of a deep complex gated convolutional attention network, a complex mask generator, and a minimum variance distortion-free response beamformer connected sequentially.

[0055] S4: Deep complex gated convolutional attention network extracts spatial-temporal-frequency features from multi-channel input, which are input to the complex mask generator to obtain a set of estimated complex ideal ratio masks The spatial covariance matrix of the target signal is calculated through this set of masks and the spatial covariance matrix of the noise signal

[0056]

[0057] Among them, Y(t,f) represents the noisy signal; by calculating the covariance matrix of the target signal The principal components of the guide vector

[0058] Using the estimated spatial covariance matrix of the noise signal and the guide vector The weights of the minimum variance distortionless response beamformer are calculated. The calculation process is expressed as:

[0059]

[0060] Multiplying the multi-channel noisy complex spectrum with the beamformer weight coefficient, the coarse noise reduction output of the beamforming stage is obtained as:

[0061]

[0062] like Figure 4 As shown, the post-filtering stage consists of a deep complex convolutional attention network and a complex mask generator.

[0063] S5: The single-channel signal after coarse noise reduction by the beamforming network is concatenated with the noisy signal of the reference channel as the input of the post-filtering stage. The deep complex gated convolutional attention network extracts the time-frequency features from the input and inputs them into the complex mask generator to obtain the estimated complex ideal ratio mask. Will Single-channel coarse noise reduction output with BF stage Multiply them together to get the final enhanced complex spectrum:

[0064]

[0065] S6: The final enhanced complex spectrum is subjected to spectrum decompression and inverse short-time Fourier transform to obtain the enhanced target signal in the time domain.

[0066] In this embodiment, the spectrum after two-stage noise reduction is as follows: Figure 5 As shown. Figure 5 As can be seen from the spectrum of clean speech Figure 5 (a) The target signal, and the spectrum of the noisy speech Figure 5 In (b), the target signal is drowned by noise, but after the beamforming stage, the noise can be effectively suppressed, e.g. Figure 5 As shown in (c), the residual noise is further suppressed after the post-filtering stage, as shown in Figure 5 As shown in (d), the target signal is better recovered from the noise.

[0067] In this embodiment, the Perceptual Evaluation Speech Quality (PESQ) is used to evaluate the quality of the denoised signal to measure the denoising performance of the two-stage deep complex network.

[0068] For the speech signal in this embodiment, the intelligibility of the enhanced speech signal is evaluated by calculating the Short-Time Objective Intelligibility (STOI). The PESQ index ranges from -0.5 to 4.5, with higher scores indicating better speech quality. The STOI index ranges from 0 to 1, with higher values ​​indicating higher speech intelligibility and clarity.

[0069] This embodiment performs two-stage noise reduction on noisy signals with signal-to-noise ratios of -10, -5, 0, 5, and 10 dB. The evaluation index results of the two-stage noise reduction at different signal-to-noise ratios are shown in the figure below. Figure 6 As shown. Figure 6It can be seen from the figure that the two-stage enhancement strategy makes the PESQ and STOI evaluation indicators of the noisy signal continuously improved. After the beamforming stage, Figure 6 (a) PESQ and Figure 6 The STOI index of (b) is significantly improved. The post-pass filtering stage can further improve the scores of the two indicators, and the improvement is relatively large, which shows that the noise reduction performance of this embodiment is good and can effectively improve the quality and intelligibility of the signal.

Claims

1. A multi-channel noise suppression method based on a two-stage deep complex network, characterized in that: The following steps are involved: S1: Use short-time Fourier transform to transform the multi-channel noisy signal received by the microphone array into the time-frequency domain and perform spectrum compression to obtain a compressed multi-channel noisy complex spectrum, and select one of the channels as the reference channel; S2: Constructing a deep complex gated convolutional attention network, wherein the deep complex gated convolutional attention network includes a plurality of complex encoder modules, a plurality of complex decoder modules, a complex Conformer module, and a plurality of encoder-decoder attention modules; the plurality of complex encoder modules, the complex Conformer modules, and the plurality of complex decoder modules are sequentially connected, and each complex encoder module is skip-connected to the corresponding complex decoder module through the encoder-decoder attention module; S3: Using a deep complex gated convolutional attention network to build a two-stage deep complex network consisting of a beamforming stage and a post-filtering stage; The beamforming stage includes a deep complex gated convolutional attention network, a complex mask generator, and a minimum variance distortionless response beamformer connected in sequence, wherein the deep complex gated convolutional attention network is used to model the multi-channel noisy complex spectrum; the complex mask generator is used to generate a set of complex ideal ratio masks to obtain the weight coefficients of the minimum variance distortionless response beamformer; the minimum variance distortionless response beamformer uses the weight coefficients to suppress the multi-channel noisy complex spectrum to obtain a coarsely denoised single-channel complex spectrum; S4: The compressed multi-channel noisy complex spectrum is taken as input and passed through the beamforming stage consisting of a deep complex gated convolutional attention network, a complex mask generator and a minimum variance distortionless response beamformer to obtain a coarsely denoised single-channel complex spectrum; S5: The coarsely denoised single-channel complex spectrum and the noisy complex spectrum of the reference channel are taken as input, and the final enhanced complex spectrum is obtained through a post-filtering stage consisting of a deep complex gated convolutional attention network and a complex mask generator; S6: The enhanced complex spectrum is subjected to spectrum decompression and inverse short-time Fourier transform to obtain the enhanced target signal in the time domain.

2. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that In step S1, the short-time Fourier transform is converted to the time-frequency domain and spectrum compression is performed. The spectrum compression method is expressed as: in, Y ori represents the complex spectrum after short-time Fourier transform, ∣ Y ori ∣ and represent the amplitude and phase angle of the complex spectrum after short-time Fourier transform, respectively.

3. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that The complex encoder module in step S2 includes gated convolution, complex batch normalization and PreLU activation function; the complex decoder module includes gated deconvolution, complex batch normalization and PreLU activation function.

4. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 3, characterized in that: In the complex encoder module described in step S2, except for the first layer of complex convolution gating units, the kernel size is set to (2, 5), and the kernel sizes of the remaining layers are set to (2, 3). The step size of the complex convolution gating units in each layer is set to (1, 2). Through layer-by-layer convolution operations, the number of channels of all complex encoder modules is set to 64.

5. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that: The complex number Conformer module in step S2 includes two Conformer modules, which are respectively the real part r and the imaginary part i. Each Conformer module includes a convolution module, a multi-head self-attention module, a feedforward module and layer normalization connected in sequence.

6. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that: The encoding and decoding attention module in step S2 is connected between each complex decoder module and the corresponding complex encoder module, takes the feature information from the complex decoder module and the corresponding complex encoder module as input, fuses them to obtain high-dimensional features, and splices the fused high-dimensional features with the feature information input by the complex decoder module as output.

7. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that The two-stage deep complex network described in step S3 has a first stage of beamforming, which performs coarse noise reduction on the multi-channel noisy input to obtain a coarsely denoised single-channel complex spectrum. The second stage is a post-filtering stage, which further enhances the output of the first stage, suppresses residual noise, and obtains the final enhanced complex spectrum.

8. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that: The post-filtering stage described in step S3 includes a deep complex gated convolutional attention network and a complex mask generator connected in sequence, wherein the deep complex gated convolutional attention network is used to model the single-channel complex spectrum of the coarse denoising; the complex mask generator is used to generate a complex ideal ratio mask, and the single-channel complex spectrum of the coarse denoising is multiplied with it to obtain the final enhanced complex spectrum.

Images