Multi-channel noise suppression method based on two-stage deep complex network

Abstract

The invention discloses a multi-channel noise suppression method based on a two-stage deep complex network, and the method comprises the steps: constructing a deep complex gating convolution attention network, and carrying out the correlation modeling of a complex frequency spectrum; enhancing a multi-channel noisy signal by using a two-stage deep complex network; in the beam forming stage, amplitude and phase enhancement is carried out on the multi-channel complex spectrum, and a single-channel complex spectrum with coarse noise reduction is obtained; and in the post-filtering stage, refining processing is carried out on the single-channel complex frequency spectrum subjected to coarse noise reduction, and noise is further suppressed. According to the method, a noise suppression task is divided into a beam forming stage and a post-filtering stage through a two-stage enhancement strategy, the problems of noise residue and target sound signal distortion are effectively avoided, in addition, a complex frequency spectrum is modeled by adopting a complex network structure in the two-stage network, and the noise suppression efficiency is improved. Meanwhile, the amplitude and phase information of the signal are enhanced, the phase distortion is reduced, and the quality and naturalness of the signal are improved.

Description

Technical Field

[0001] The present invention relates to the field of sound signal processing, and in particular to a multi-channel noise suppression method based on a two-stage deep complex network. Background Art

[0002] In modern voice communication, automatic speech recognition (ASR), and acoustic event detection, clear target sound signal quality is crucial. However, in actual scenarios, the target sound signal collected by the microphone array is usually interfered by noise, resulting in a decrease in signal quality and clarity. Noise suppression technology aims to reduce the impact of noise on the target sound signal and extract the clean target sound from the noisy signal collected by the microphone. The multi-channel noise suppression method uses the spatial information obtained by a microphone array composed of multiple microphones to selectively enhance or suppress signals in certain directions. It can improve the quality of the target sound signal to a certain extent and has gradually become a hot topic of current research.

[0003] In recent years, multi-channel noise suppression methods based on deep learning have made significant progress in performance, which can be mainly divided into two types: one is an end-to-end method based on deep neural networks; the other is a method combining beamforming with deep neural networks.

[0004] The end-to-end noise suppression method based on deep neural network is to estimate the mapping function from mask or noisy signal to target signal by training the network to obtain enhanced signal. Although the end-to-end method shows good noise reduction ability, it does not impose distortion-free constraints, and nonlinear distortion will inevitably be introduced in the noise reduction process. The method of combining beamforming with deep neural network is to use neural network to estimate the weights of beamforming or the mask used, and then implement spatial filtering through beamforming to obtain enhanced signal. This method can effectively suppress noise from other directions through directional filtering without destroying the original structure of the target signal, and can eliminate or reduce the problem of nonlinear distortion.

[0005] The complex spectrum can be decomposed into real and imaginary parts in Cartesian coordinates and into amplitude and phase in polar coordinates. The amplitude determines the strength of the sound signal, while the phase describes the temporal relationship of the sound waveform and the structure of the waveform. The two together affect the quality and naturalness of the sound. However, it is difficult to model the correlation information in the complex spectrum in multi-channel noise suppression, and unreasonable modeling will lead to the loss of signal phase information. The above-mentioned multi-channel noise suppression methods only enhance the amplitude of the noisy signal or are limited to using real-valued neural network models to reconstruct phase information from the complex spectrum of the noisy signal, which makes it difficult to capture the complex time-frequency relationship in the noisy signal. This limits the application of multi-channel noise suppression algorithms in complex noise environments to a certain extent. In the case of severe noise interference, the estimated phase of the target sound signal will have obvious deviations. If the phase information is inaccurate, even if the amplitude spectrum is completely restored, the target sound signal will still appear unnatural or blurred. In addition, statistical information such as spectrum or mask in multi-channel signals is difficult to accurately estimate, and there will still be a lot of noise residual after filtering. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-channel noise suppression method based on a two-stage deep complex network, which can reasonably model the correlation information in the complex spectrum of the multi-channel noisy signal, and adopt a two-stage enhancement strategy to realize the amplitude and phase enhancement of the signal, restore the phase information of the target sound signal, solve the problem of phase distortion, reduce the distortion of the target sound signal, suppress the residual noise in the beamforming stage, and effectively improve the quality of the signal.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A multi-channel noise suppression method based on a two-stage deep complex network comprises the following steps:

[0009] S1: Use short-time Fourier transform to transform the multi-channel noisy signal received by the microphone array into the time-frequency domain, and perform spectrum compression to obtain a compressed multi-channel noisy complex spectrum, and select one of the channels as the reference channel;

[0010] S2: construct a deep complex gated convolutional attention network, wherein the deep complex gated convolutional attention network includes a plurality of complex encoder modules, a plurality of complex decoder modules, a complex Conformer module, and a plurality of encoder-decoder attention modules; the plurality of complex encoder modules, the complex Conformer modules, and the plurality of complex decoder modules are sequentially connected, and each complex encoder module is jump-connected to the corresponding complex decoder module through the encoder-decoder attention module;

[0011] S3: Using a deep complex gated convolutional attention network to build a two-stage deep complex network consisting of a beamforming stage and a post-filtering stage;

[0012] S4: The compressed multi-channel noisy complex spectrum is taken as input and passed through the beamforming stage consisting of a deep complex gated convolutional attention network, a complex mask generator and a minimum variance distortionless response beamformer to obtain a coarsely denoised single-channel complex spectrum;

[0013] S5: The coarsely denoised single-channel complex spectrum and the noisy complex spectrum of the reference channel are taken as input, and the final enhanced complex spectrum is obtained after a post-filtering stage consisting of a deep complex gated convolutional attention network and a complex mask generator;

[0014] S6: The enhanced complex spectrum is subjected to spectrum decompression and short-time inverse Fourier transform to obtain a target signal enhanced in the time domain.

[0015] Furthermore, in step S1, the short-time Fourier transform is converted into the time-frequency domain and spectrum compression is performed, and the spectrum compression method is expressed as:

[0016]

[0017] Among them, Y ori represents the complex spectrum after short-time Fourier transform, |Y ori | and Respectively represent the amplitude and phase angle of the complex spectrum after short-time Fourier transform.

[0018] Furthermore, the complex encoder module includes gated convolution, complex batch normalization and PreLU activation function; the complex decoder module includes gated deconvolution, complex batch normalization and PreLU activation function.

[0019] Furthermore, the complex number Conformer module includes two Conformer modules, which are two processing parts of the real part (r) and the imaginary part (i), respectively, wherein each Conformer module includes a convolution module, a multi-head self-attention module, a feedforward module and layer normalization connected in sequence.

[0020] Furthermore, the encoding and decoding attention module is connected between each complex decoder module and the corresponding complex encoder module, takes the feature information from the complex decoder module and the corresponding complex encoder module as input, fuses them to obtain high-dimensional features, and concatenates the fused high-dimensional features with the feature information input by the complex decoder module as output.

[0021] Furthermore, the two-stage deep complex network has a first stage being a beamforming stage, which performs coarse denoising on multi-channel noisy input to obtain a coarsely denoised single-channel complex spectrum, and a second stage being a post-filtering stage, which further enhances the output of the first stage, suppresses residual noise, and obtains a final enhanced complex spectrum.

[0022] Furthermore, the beamforming stage includes a deep complex gated convolutional attention network, a complex mask generator and a minimum variance distortion-free response beamformer connected in sequence, wherein the deep complex gated convolutional attention network is used to model the multi-channel noisy complex spectrum; the complex mask generator is used to generate a set of complex ideal ratio masks, so as to obtain the weight coefficients of the minimum variance distortion-free response beamformer; the minimum variance distortion-free response beamformer uses the weight coefficients to suppress the multi-channel noisy complex spectrum to obtain a coarsely denoised single-channel complex spectrum.

[0023] Furthermore, the post-filtering stage includes a deep complex gated convolutional attention network and a complex mask generator connected in sequence, wherein the deep complex gated convolutional attention network is used to model the single-channel complex spectrum of the coarse denoising; the complex mask generator is used to generate a complex ideal ratio mask, and the single-channel complex spectrum of the coarse denoising is multiplied with it to obtain the final enhanced complex spectrum.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] (1) The present invention constructs a deep complex gated convolutional attention network. By introducing a complex Conformer module, a complex encoder and a complex decoder module, the correlation between the real and imaginary parts of the complex spectrum of the noisy signal is modeled, which can simultaneously enhance the amplitude and phase information, thereby effectively reducing the phase distortion. The encoder-decoder attention module is introduced to enhance the complex decoder's perception of detail features and enhance the quality of the reconstructed signal.

[0026] (2) The present invention adopts a two-stage enhancement strategy to decompose the noise suppression task into two steps: the first stage is the beamforming stage, which combines the deep complex gated convolutional attention network with the beamforming technology to perform rough noise suppression, remove the main noise and retain the original structure of the target sound signal; the second stage is the post-filtering stage, which refines the signal output from the first stage, removes the residual noise in the beamforming stage and further enhances the quality of the target signal. This staged processing method not only retains the advantage of the combination of beamforming and deep neural network in terms of less damage to the original structure of the target sound signal, but also solves the problem that the output of beamforming contains a large amount of residual noise, thereby improving the effect of noise suppression and reducing the distortion of the target sound signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a multi-channel noise suppression method based on a two-stage deep complex network of the present invention;

[0028] Figure 2 Schematic diagram of the structure of a deep complex gated convolutional attention network in an embodiment;

[0029] Figure 3 is a schematic diagram of the structure of a complex Conformer module in an embodiment;

[0030] Figure 4 It is a schematic diagram of the structure of a two-stage deep complex network in an embodiment;

[0031] Figure 5 This is a spectrum diagram after two-stage noise reduction in the embodiment;

[0032] Among them, (a) the spectrum of clean speech, (b) the spectrum of noisy speech, (c) the spectrum after denoising in the beamforming stage, and (d) the spectrum after denoising in the beamforming stage + post-filtering stage;

[0033] Figure 6 This is a graph showing the evaluation index results of two-stage noise reduction at different signal-to-noise ratios in the embodiment;

[0034] Among them, (a) PESQ evaluation index, (b) STOI evaluation index. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in combination with the embodiments and drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by the technicians in this field without making creative work are within the scope of protection of the present invention.

[0036] Example:

[0037] A multi-channel noise suppression method based on a two-stage deep complex network, the flow chart is as follows Figure 1 As shown, in this embodiment, a speech signal is selected as the target signal, and factory noise is selected as the noise signal. The method of the present invention is also applicable to the case where the target signal is a non-speech signal; the suppression method comprises the following steps:

[0038] S1: Use short-time Fourier transform to transform the multi-channel time-domain noisy signal y(t) = [y1(t),…,y P (t)] T Transformed to the time-frequency domain, it is expressed as:

[0039] Y(t,f)=S(t,f)+N(t,f)=d(f)S(t,f)+N(t,f)

[0040] Where Y(t,f)=[Y1(t,f),…,Y P (t,f)] T represents the signal received by the microphone array at time frame t∈{1,…,T} and frequency f∈{1,…,F}, S(t,f) represents the target sound signal component received by the microphone array, N(t,f) represents the additive noise component, d(t,f) represents the relative transfer function of the signal, S(t,f) represents the target sound of the reference channel, and the first channel is usually selected as the reference channel;

[0041] The signal transformed into the time-frequency domain is compressed with a compression rate of 0.5. The compressed complex spectrum is expressed as:

[0042]

[0043] Among them, Y represents the compressed complex spectrum, Y ori represents the complex spectrum after short-time Fourier transform, Y ori | and Respectively represent the amplitude and phase angle of the complex spectrum after short-time Fourier transform.

[0044] S2: Construct a deep complex gated convolutional attention network, such as Figure 2 As shown, the deep complex gated convolutional attention network includes several complex encoder modules, several complex decoder modules, complex Conformer modules and codec attention modules; several complex encoder modules, complex Conformer modules and several complex decoder modules are connected in sequence, and each complex encoder module is jump-connected to the corresponding complex decoder module through the codec attention module.

[0045] like Figure 2 As shown, each complex encoder module includes gated convolution, complex batch normalization and PreLU activation function, which are used to extract high-level features and downsampling. The gated convolution is divided into two processing parts: real part (r) and imaginary part (i), and the input complex spectrum is also divided into two parts: real part and imaginary part. In the complex encoder module, except for the first layer of complex convolution gated unit, the kernel size is set to (2,5), and the kernel size of the remaining layers is set to (2,3). The step size of each layer of complex convolution gated unit is set to (1,2). Through layer-by-layer convolution operations, the complex encoder gradually compresses the input spectrum signal into a more compact feature representation, removes redundant information, and retains key features. The number of channels of all complex encoder modules is set to 64.

[0046] The encoding and decoding attention module fuses the detail feature information from the encoding layer and the deep information from the decoding layer and then inputs it into the next complex decoder to enhance the information interaction between the encoder and the decoder, strengthen the decoder's ability to perceive the signal detail features, and thus enhance the quality of the reconstructed signal. The fused feature information is expressed as:

[0047]

[0048] in, and represents the output of the i-th layer complex encoder and the corresponding complex decoder, represents a convolution kernel, and Represent the convolution kernels of the i-th encoding layer and decoding layer respectively. and Concatenated together as the input of the next complex decoder.

[0049] like Figure 2 As shown, each complex decoder module includes gated deconvolution, complex batch normalization and PreLU activation function for feature processing and upsampling. The gated deconvolution is divided into two processing parts: real part (r) and imaginary part (i), and the input complex spectrum is also divided into two parts: real part and imaginary part. The complex encoder gradually compresses the input spectrum signal into a more compact feature representation, removes redundant information, and retains key features; except for the last layer of complex deconvolution gating unit in the complex decoder, the kernel size of the remaining layers is set to (2,3), and the step size of each layer of complex deconvolution gating unit is set to (1,2). The complex decoder gradually restores the high-resolution complex spectrum by deconvolution layer by layer. Except for the last complex decoder whose channel number is set to 2, the channel number of other complex decoders is set to 64.

[0050] like Figure 3 As shown, the complex number Conformer module includes two Conformer modules, which are the real part (r) and the imaginary part (i) processing parts. The input complex spectrum is also divided into two parts, the real part and the imaginary part. Each Conformer module includes a convolution module, a multi-head self-attention module, a feedforward module and a layer normalization. The complex number Conformer module is used to simultaneously process the real part and the imaginary part of the input complex spectrum under the complex number calculation rules, model the correlation between the amplitude and phase of the signal, and enhance the model's ability to capture long-distance dependencies. The output Y of the complex number Conformer module C for:

[0051] Y C =(C r (Xr )-C i (X i ))+j(C i (X r )+C r (X i ))

[0052] Among them, X r and X i Represent the real and imaginary parts of the input complex spectrum, C r (·) and C i (·) represent the real and imaginary processing parts of the Conformer module respectively.

[0053] S3: Construct a two-stage deep complex network, which includes two stages. The first stage is the beamforming stage, which performs coarse denoising on the multi-channel noisy input to obtain a coarsely denoised single-channel complex spectrum. The second stage is the post-filtering stage, which further enhances the output of the first stage, suppresses residual noise, and obtains the final enhanced complex spectrum.

[0054] like Figure 4 As shown, the beamforming stage consists of a deep complex gated convolutional attention network, a complex mask generator, and a minimum variance distortion-free response beamformer connected sequentially.

[0055] S4: A deep complex gated convolutional attention network extracts spatial-temporal-frequency features from multi-channel inputs and feeds them into a complex mask generator to obtain a set of estimated complex ideal ratio masks. The spatial covariance matrix of the target signal is calculated through this set of masks and the spatial covariance matrix of the noise signal

[0056]

[0057] Among them, Y(t,f) represents the noisy signal; by calculating the covariance matrix of the target signal The principal components of the guide vector

[0058] Using the estimated spatial covariance matrix of the noise signal and the guide vector The weights of the minimum variance distortion-free response beamformer are calculated. The calculation process is expressed as:

[0059]

[0060] Multiplying the multi-channel noisy complex spectrum with the beamformer weight coefficient, the coarse noise reduction output of the beamforming stage is obtained as:

[0061]

[0062] like Figure 4 As shown, the post-filtering stage consists of a deep complex convolutional attention network and a complex mask generator.

[0063] S5: The single-channel signal after coarse denoising by the beamforming network is concatenated with the noisy signal of the reference channel as the input of the post-filtering stage. The deep complex gated convolutional attention network extracts the time-frequency features from the input and inputs them into the complex mask generator to obtain the estimated complex ideal ratio mask. Will Single channel coarse noise reduction output with BF stage Multiply them together to get the final enhanced complex spectrum:

[0064]

[0065] S6: The final enhanced complex spectrum is subjected to spectrum decompression and short-time inverse Fourier transform to obtain a target signal enhanced in the time domain.

[0066] In this embodiment, the spectrum after two-stage noise reduction is as follows: Figure 5 As shown. Figure 5 As can be seen from the spectrum of clean speech Figure 5 (a) The target signal, and the spectrum of the noisy speech Figure 5 In (b), the target signal is submerged by noise, but after the beamforming stage, the noise can be effectively suppressed. Figure 5 As shown in (c), the residual noise is further suppressed after the post-filtering stage, as shown in Figure 5 As shown in (d), the target signal is better recovered from the noise.

[0067] In this embodiment, the Perceptual Evaluation of Speech Quality (PESQ) is used to evaluate the quality of the denoised signal to measure the denoising performance of the two-stage deep complex network.

[0068] For the speech signal in the embodiment, the intelligibility of the enhanced speech signal is evaluated by calculating the short-time objective intelligibility (STOI). The PESQ index value ranges from -0.5 to 4.5, and the higher the score, the better the speech quality. The STOI index value ranges from 0 to 1, and the larger the value, the higher the speech intelligibility and the clearer the speech.

[0069] In this embodiment, two-stage noise reduction is performed on noisy signals with signal-to-noise ratios of -10, -5, 0, 5, and 10 dB. The evaluation index result diagram of the two-stage noise reduction under different signal-to-noise ratios is as follows: Figure 6 As shown. Figure 6It can be seen that the two-stage enhancement strategy makes the PESQ and STOI evaluation indicators of noisy signals continuously improved. After the beamforming stage, Figure 6 (a) PESQ and Figure 6 The STOI index of (b) is significantly improved. The scores of the two indicators can be further improved by the post-filtering stage, and the improvement is relatively large, which shows that the noise reduction performance of this embodiment is good and can effectively improve the quality and intelligibility of the signal.

Claims

1. A multi-channel noise suppression method based on a two-stage deep complex network, characterized in that: The following steps are involved: S1: Use short-time Fourier transform to transform the multi-channel noisy signal received by the microphone array into the time-frequency domain, and perform spectrum compression to obtain a compressed multi-channel noisy complex spectrum, and select one of the channels as the reference channel; S2: construct a deep complex gated convolutional attention network, wherein the deep complex gated convolutional attention network includes a plurality of complex encoder modules, a plurality of complex decoder modules, a complex Conformer module, and a plurality of encoder-decoder attention modules; the plurality of complex encoder modules, the complex Conformer modules, and the plurality of complex decoder modules are sequentially connected, and each complex encoder module is jump-connected to the corresponding complex decoder module through the encoder-decoder attention module; S3: Using a deep complex gated convolutional attention network to build a two-stage deep complex network consisting of a beamforming stage and a post-filtering stage; S4: The compressed multi-channel noisy complex spectrum is taken as input and passed through the beamforming stage consisting of a deep complex gated convolutional attention network, a complex mask generator and a minimum variance distortionless response beamformer to obtain a coarsely denoised single-channel complex spectrum; S5: The coarsely denoised single-channel complex spectrum and the noisy complex spectrum of the reference channel are taken as input, and the final enhanced complex spectrum is obtained after a post-filtering stage consisting of a deep complex gated convolutional attention network and a complex mask generator; S6: The enhanced complex spectrum is subjected to spectrum decompression and short-time inverse Fourier transform to obtain a target signal enhanced in the time domain.

2. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1 is characterized in that: In step S1, the short-time Fourier transform is converted into the time-frequency domain and the spectrum is compressed. The spectrum compression method is expressed as: Among them, Y ori represents the complex spectrum after short-time Fourier transform, |Y ori ∣ and Respectively represent the amplitude and phase angle of the complex spectrum after short-time Fourier transform.

3. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1 is characterized in that: The complex encoder module in step S2 includes gated convolution, complex batch normalization and PreLU activation function; the complex decoder module includes gated deconvolution, complex batch normalization and PreLU activation function.

4. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 3 is characterized in that: In the complex encoder module described in step S2, except for the first layer of complex convolution gating units, the kernel size of the remaining layers is set to (2, 3), and the step size of each layer of complex convolution gating units is set to (1, 2). Through layer-by-layer convolution operations, the number of channels of all complex encoder modules is set to 64.

5. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that: The complex number Conformer module in step S2 includes two Conformer modules, which are two processing parts of the real part r and the imaginary part i, respectively, wherein each Conformer module includes a convolution module, a multi-head self-attention module, a feedforward module and layer normalization connected in sequence.

6. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that: The encoding and decoding attention module in step S2 is connected between each complex decoder module and the corresponding complex encoder module, takes the feature information from the complex decoder module and the corresponding complex encoder module as input, fuses them to obtain high-dimensional features, and concatenates the fused high-dimensional features with the feature information input by the complex decoder module as output.

7. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that: The two-stage deep complex network described in step S3 has a first stage of a beamforming stage, which performs coarse denoising on multi-channel noisy input to obtain a coarsely denoised single-channel complex spectrum, and a second stage of a post-filtering stage, which further enhances the output of the first stage, suppresses residual noise, and obtains a final enhanced complex spectrum.

8. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that: The beamforming stage in step S3 includes a deep complex gated convolutional attention network, a complex mask generator and a minimum variance distortion-free response beamformer connected in sequence, wherein the deep complex gated convolutional attention network is used to model the multi-channel noisy complex spectrum; the complex mask generator is used to generate a set of complex ideal ratio masks, so as to obtain the weight coefficients of the minimum variance distortion-free response beamformer; the minimum variance distortion-free response beamformer uses the weight coefficients to suppress the multi-channel noisy complex spectrum and obtain a single-channel complex spectrum with coarse noise reduction.

9. The multi-channel noise suppression method based on a two-stage deep complex network according to claim 1, characterized in that: The post-filtering stage in step S3 includes a deep complex gated convolutional attention network and a complex mask generator connected in sequence, wherein the deep complex gated convolutional attention network is used to model the single-channel complex spectrum of the coarse denoising; the complex mask generator is used to generate a complex ideal ratio mask, and the single-channel complex spectrum of the coarse denoising is multiplied with it to obtain the final enhanced complex spectrum.

Images