Full-band speech enhancement method based on spectral compression and self-attention neural network

Abstract

The present invention discloses a full-band speech enhancement method based on spectrum compression and self-attention neural network. The method comprises the following steps: using a clear speech data set, a noise data set, and a room impulse response data set to synthesize simulated noisy reverberation speech; performing short-time Fourier transform on the simulated noisy reverberation speech data and the corresponding clean reverberation speech data to obtain short-time Fourier spectra; using the short-time Fourier spectra to train the weights of the SCM-DPARN model; performing short-time Fourier transform on the noisy reverberation speech signal to be enhanced to obtain a short-time Fourier spectrum; inputting the short-time Fourier spectrum of the noisy reverberation speech to be enhanced into the trained SCM-DPARN model to obtain a short-time Fourier spectrum of the enhanced speech; and finally performing an inverse short-time Fourier transform on the short-time Fourier spectrum of the enhanced speech to obtain a time domain signal of the enhanced speech. The present invention models the global information in the frequency domain, so that the model has better noise suppression and speech quality preservation capabilities.

Description

Technical Field

[0001] The present invention belongs to the field of speech enhancement, and is mainly aimed at the noise suppression problem of full-band speech. Background Art

[0002] The goal of speech enhancement is to recover pure speech components from noise-contaminated speech signals, improving their audibility and clarity. It is a crucial front-end technology in many systems, such as automatic speech recognition and online conferencing terminals. Deep neural network-based speech enhancement methods have demonstrated significant superiority in suppressing various non-stationary noises. However, the vast majority of research on speech enhancement systems has been conducted under wideband (16kHz) conditions, and further research is needed in ultra-wideband (32kHz) and full-band (48kHz) scenarios.

[0003] There are already some studies on full-band speech enhancement solutions (Lv, Shubo, et al. "S-DCCRN: Super Wide Band DCCRN with learnable complex feature for speech enhancement." 2021). The main challenges of full-band speech enhancement are threefold: First, if all frequency points are retained for modeling, the network structure will be very heavy and computationally intensive, making it difficult to apply to real-time speech enhancement systems; second, if the same computing resources are allocated to each frequency point, the high frequency (5-24kHz) will receive more computing resources than the low frequency (0-5kHz), and human speech components are mainly concentrated in the low frequency band, especially the harmonic components. This allocation method will make it difficult to properly process low-frequency information, which will hinder the modeling of the entire speech; third, the processing of the high-frequency part itself is also more difficult, because the energy of high frequencies is usually sparse and the signal-to-noise ratio is often low, making it difficult for the network to effectively learn at high frequencies.

[0004] Methods based on deep neural networks are generally divided into two categories: time-frequency domain methods and time-domain methods. Although time-domain methods have recently gained increasing attention, real-time applications still primarily focus on time-frequency domain methods. In time-frequency domain methods, the signal is first subjected to a short-time Fourier transform, and the short-time Fourier spectrum is then input into the network for processing. Training objectives include mask prediction and direct mapping. Early researchers believed that it was unnecessary to estimate the phase of the clean signal spectrum, only the amplitude needed to be reconstructed. This approach limited the performance of speech enhancement models. A series of works have integrated phase estimation into the network, including the use of phase-sensitive masks, the utilization of complex-scale masks, information exchange between amplitude and phase spectra, the allocation of real and imaginary spectrum estimates to two independent decoders, and the simulation of complex networks.

[0005] The complex recurrent network (CRN) architecture for complex spectrum processing consists of an encoder, two decoders, and a recursive processing block in between. The U-net encoder-decoder structure can reduce spectral resolution and extract high-dimensional local features, making recursive networks suitable for sequence modeling. The dual-path recurrent neural network (DPRNN) is a time-domain separation network that segments long sequence inputs into segments and applies intra-block and inter-block recurrent neural networks along two consecutive paths to process local and global information, respectively. The dual-path complex recurrent network (DPCRN) combines these two approaches, using intra-block and inter-block recurrent neural networks as the processing blocks in the CRN instead of the original recursive network. The intra-block recurrent neural network models along the entire frequency axis, coupling the relationships between frequency points, while the inter-block recurrent neural network processes the correlations between consecutive frames.

[0006] In recent years, attention mechanism has been an active research field. The starting point of the attention model is that when encoding, the network should learn to pay attention to different parts of the input sequence in a targeted manner. The MHA (Multi-head attention) network is a representative work in the field of natural language processing, and many studies have integrated this architecture into the speech processing network. Sepformer (Subakan, Cem, et al. "Attention is all you need in speech separation." ICASS 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021) replaced the recursive neural network in DPRNN with the MHA network, and performed well in the speech separation task. A prominent advantage of the MHA network is that it can capture the relationship of long sequences and has parallel computing capabilities, avoiding the problem of low computational efficiency of recursive neural networks. Summary of the Invention

[0007] In order to effectively process full-band information, the present invention combines a set of learnable spectrum compression and inverse compression transformations on the basis of the DPCRN structure, and uses a multi-head self-attention mechanism in the frequency domain to replace the intra-block RNN ​​(Recurrent neural network) in the original DPCRN to construct the relationship between frequency points in the full frequency band.

[0008] The technical solution adopted in the present invention is:

[0009] The full-band speech enhancement method based on spectrum compression and self-attention neural network includes the following steps:

[0010] Step 1: Use the clear speech dataset, the noise dataset, and the room impulse response dataset to synthesize simulated noisy reverberant speech;

[0011] Step 2: performing short-time Fourier transform on the simulated noisy reverberation speech data and the corresponding pure reverberation speech data to obtain short-time Fourier spectra of the two;

[0012] Step 3: Construct an SCM-DPARN model. This model compresses the spectrum using a spectrum compression mapping module (SCM) and then inputs it into a dual-path attention recurrent neural network (DPARN) that uses a self-attention mechanism in the frequency domain. The SCM-DPARN model weights are trained using the short-time Fourier spectrum obtained in step 2.

[0013] Step 4, performing short-time Fourier transform on the noisy reverberation speech signal to be enhanced to obtain a short-time Fourier spectrum;

[0014] Step 5: Input the short-time Fourier spectrum obtained in step 4 into the SCM-DPARN model trained in step 3 to obtain the short-time Fourier spectrum of the enhanced speech;

[0015] Step 6: Perform inverse short-time Fourier transform on the short-time Fourier spectrum of the enhanced speech to obtain a time domain signal of the enhanced speech.

[0016] This invention uses spectrum compression transformation to effectively compress high-frequency information. At the same time, it replaces the frequency-domain recursive neural network with a multi-head self-attention network to model global information in the frequency domain, giving the model better noise suppression and voice quality preservation capabilities. Specific advantages are as follows:

[0017] (1) The method of the present invention can effectively perform full-band speech enhancement in complex noise environments, such as rapid knocking sounds, baby crying, table and chair dragging sounds, and low signal-to-noise ratio environments, using a lightweight network model.

[0018] (2) The low-frequency fixed and high-frequency learnable compression transformation can effectively overcome the obstacles of full-band speech processing.

[0019] (3) The multi-head self-attention mechanism has powerful sequence modeling capabilities, can efficiently process global information on the frequency axis, and can perform parallel operations, facilitating the implementation of real-time speech enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the method processing in the training stage and the enhancement stage of the present invention.

[0021] Figure 2 This is a schematic diagram of the SCM (spectral compression mapping) module used in this invention. The horizontal axis represents the original spectrum, and the vertical axis represents the compressed spectrum. "Intact" indicates that the corresponding portion of the spectrum is uncompressed, while "Compressed" indicates that the corresponding portion of the spectrum is compressed. "Fixed" refers to fixed parameters, and "Learnable" refers to learnable parameters. The curve in the figure is the initialization curve, and the logarithmic compression portion in the upper right corner is implemented by a triangular filter.

[0022] Figure 3 This is the overall framework of the SCM-DPARN model. DPARN (Dual-path attention-recurrent network) refers to a dual-path attention recurrent neural network. SCM (spectral compression mapping) refers to the spectral compression mapping module, iSCM (inverse spectral compression mapping) refers to the inverse spectral compression mapping module, Encoder refers to the encoder, Real Decoder refers to the complex spectrum real part decoder, Imaginary Decoder refers to the complex spectrum imaginary part decoder, Skip connection refers to the skip transfer operation, and Couple refers to the integration of the real and imaginary parts into the complex spectrum.

[0023] Figure 4 (a) and (b) are the specific block diagrams of the encoder and decoder, respectively. Conv output X refers to the output of the Xth convolution calculation layer, Conv2D refers to the 2D convolution layer, BN (Batch normalization) refers to batch normalization, and TransConv2D is the 2D deconvolution layer.

[0024] Figure 5(a) is a specific block diagram of the processing module of DPARN. MHAblock refers to the multi-head attention network block, FC refers to the fully connected layer, and IN refers to instance normalization. Intra block refers to the intra-frame processing module, and Inter block refers to the inter-frame processing module. F', T, and C refer to the frequency dimension, time dimension, and local feature vector dimension after SCM and encoder compression, respectively. Permute represents the transpose operation of the tensor. The arrow indicates the direction in which the network processes sequence information. (b) is a specific block diagram of the multi-head attention network block. Linear refers to the linear layer, Scaled Dot-Product Attention refers to the scaled dot product attention calculation, Add refers to addition, Norm refers to normalization processing, and Feed Forward is a feedforward neural network. DETAILED DESCRIPTION

[0025] This embodiment provides a full-band speech enhancement method based on spectrum compression and self-attention neural network. Figure 1 As shown, the following steps are included:

[0026] Step 1: Use the clear speech dataset, the noise dataset, and the room impulse response dataset to synthesize simulated noisy reverberant speech;

[0027] Step 2: Perform short-time Fourier transform on the simulated noisy reverberation speech data and the corresponding clean reverberation speech data to obtain their short-time Fourier spectra;

[0028] Step 3: Construct an SCM-DPARN model. This model compresses the spectrum using the SCM and inputs the complex spectrum augmentation network (DPARN) using a self-attention mechanism in the frequency domain. The SCM is initialized with specific parameters. The SCM-DPARN model weights are trained using the short-time Fourier spectrum obtained in step 2.

[0029] Step 4, performing short-time Fourier transform on the noisy reverberation speech signal to be enhanced to obtain a short-time Fourier spectrum;

[0030] Step 5: Input the short-time Fourier spectrum obtained in step 4 into the SCM-DPARN model trained in step 3, and output the short-time Fourier spectrum of the enhanced speech;

[0031] Step 6: Perform inverse short-time Fourier transform on the short-time Fourier spectrum of the enhanced speech to obtain a time domain signal of the enhanced speech.

[0032] 1. SCM transformation and inverse transformation.

[0033] For full-band speech enhancement tasks, if the clarity of the spectrum is maintained, directly expanding the use of broadband speech enhancement models and increasing the frequency dimension of the network is a poor strategy. On the one hand, it may increase the amount of computation by three times or even more than three times (depending on the structure of the network), greatly increasing the computational burden of the hardware; on the other hand, since the energy distribution of speech is relatively sparse at high frequencies, and the computing resources are heavily tilted towards high frequencies, it will make network learning more difficult. Therefore, the present invention uses an SCM method to effectively extract spectral information and compress the F-dimensional spectrum into a lower-dimensional (F C The spectrum transformation formula of SCM is borrowed from the transformation formula of Mel spectrum. It retains all frequency information in the frequency band below 5kHz and performs logarithmic mapping on the frequency in the frequency band of 5kHz-24kHz:

[0034]

[0035] Among them, the original frequency band and the compressed frequency band are f and f respectively c .

[0036] According to the transformation formula (1), a triangular filter bank is used to obtain a spectrum compression transformation matrix. In order to further adopt a transformation mode that is beneficial to the learning of the network, a fully connected layer is used to learn this transformation matrix and initialize it with the spectrum compression transformation matrix mentioned above. In order to ensure that the low-frequency information is processed sufficiently finely, the weights of the low-frequency part (below 5kHz, corresponding to the frequency features of the first K dimensions) are set to fixed, and the high-frequency part (5kHz-24kHz, corresponding to the frequency features of the last F C The weights of the frequency features of the K-dimensional part are set to be learnable. The settings of the SCM module and the curve corresponding to its initialization value are as follows Figure 2 iSCM (inverse spectral compression mapping) is also implemented through a learnable fully connected layer, but it does not require specific initialization and has no fixed weights.

[0037] 2. SCM-DPARN model and its training.

[0038] The block diagram of the SCM-DPARN model is as follows Figure 3As shown, the proposed system consists of a codec and a processing module. The SCM and iSCM layers are used before the encoder and after the real and imaginary part decoder, respectively. Between the codec and decoder, DPARN extracts pure speech information from the input spectral features. Building on the DPCRN architecture, DPARN models the frequency and time paths separately. When modeling frequency information, DPARN uses a multi-head self-attention mechanism instead of a recurrent network to address the coupling relationships between different frequency points. Compared to recurrent networks, multi-head self-attention networks offer stronger sequence processing capabilities, better consider global information, and more effectively integrate global frequency domain information. In the temporal processing stage, DPARN uses a recurrent neural network to model the relationships between spectral frames. Recurrent neural networks can handle temporal relationships with minimal computational cost, enabling real-time speech enhancement. A key innovation of the present invention lies in this processing strategy: using a self-attention network to model the entire spectrum within a frame and a recurrent network to model the temporal dependencies between the same frequency point across different frames.

[0039] The input of the SCM-DPARN model is the short-time Fourier spectrum X of the noisy reverberation signal, and its output is the estimated pure speech

[0040]

[0041] Represents the neural network SCM-DPARN. Training is achieved by minimizing the loss function L using stochastic gradient descent:

[0042]

[0043] 1) Model parameter setting

[0044] The sampling rate of all audio data is 48kHz, the window length of short-time Fourier transform is 25ms, and the overlap rate between frames is 50%. The full-band spectrum dimension F of the input network is 601, and the spectrum dimension F after SCM compression is 601. C The dimension K for the low-frequency band (below 5kHz) is 256, and the dimension K for the low-frequency band (below 5kHz) is 125. To achieve batch processing and improve computational efficiency, this embodiment fixes the duration of each input speech segment to 8 seconds during training. The real and imaginary parts of the complex spectrum are spliced ​​together as two different channels and input to the network. That is, the network input tensor shape is (F, T, 2), where T represents the number of frames.

[0045] In the encoder part, this embodiment uses 5 two-dimensional convolutional layers (Conv2D), whose feature channel number (filters), convolution kernel size (Kernel-size), and stride (Stride) are {16, 32, 48, 64, 80}, {(2, 5), (2, 3), (2, 3), (2, 3), (2, 1)}, {(1, 2), (1, 1), (1, 1), (1, 1), (1, 1)}, respectively. The first number represents the frequency dimension parameter, and the second number represents the time dimension parameter. Each Conv2D is followed by BN and PReLU activation functions. The specific framework of the encoder is as follows Figure 4 Figure (a) shows

[0046] The shape of the tensor output by the encoder to the DPARN processing module is (F′, T, C). In this embodiment, since the number of channels of the last convolutional layer of the encoder is 80, C = 80. On the frequency path, two layers of MHA are first used, followed by a fully connected layer with 128 units and a layer of IN (Instance normalization). The tensor is then transposed. On the time path, one layer of LSTM (Long short-term memory network) is first used, followed by a fully connected layer with 128 units and a layer of IN, and then transposed again. The specific diagram of the DPARN processing module is as follows Figure 5 (a) of Figure 1, the specific diagram of MHA is as follows Figure 5 Figure (b).

[0047] The tensor input by the DPARN processing module to the two independent real and imaginary part decoders is also (F′, T, C). The decoder uses 5 deconvolution layers (ConvTrans2D), whose feature channel numbers, convolution kernel sizes, and strides are {64, 48, 32, 16, 1}, {(2, 1), (2, 3), (2, 3), (2, 3), (2, 5)}, {(1, 1), (1, 1), (1, 1), (1, 1), (1, 2)} respectively. The first number represents the parameters of the frequency dimension, and the second number represents the parameters of the time dimension. Each ConvTrans2D is followed by BN and PReLU activation functions. The specific framework of the decoder is as follows. Figure 4 Figure (b).

[0048] 2) Loss function and training strategy.

[0049] The loss function used in this invention is the compressed MSE (Mean square error) function. A spectrogram can be written as Then the spectrogram after energy compression is defined as where θ S represents the complex phase angle, γ represents the compression coefficient, which is is a compression symbol. Then its actual part and the imaginary part It can be further written as

[0050]

[0051] Similarly, the energy-compressed estimated spectrogram can be defined as and

[0052] The loss function used in this embodiment is defined as:

[0053]

[0054]

[0055]

[0056] During training, the Adam optimizer is used, and the learning rate warm-up method is used to update the learning rate α:

[0057]

[0058] Where ψ is the number of training steps, and Ψ is the number of warmup steps, which in this example is 40,000. The learning rate warmup method is crucial for training MHA networks. It adjusts the learning rate from a very small value, first increasing it and then decreasing it. This prevents the network weights from fluctuating too much at the beginning, allowing it to find a suitable convergence direction.

[0059] The batch size of training is set to 6, and the maximum training cycle is 100 epochs.

[0060] 3. Dataset and evaluation metrics

[0061] The performance of the SCM-DPARN model was first tested using the VCTK-DEMAND dataset (Valentini-Botinhao, C., Wang, X., Takaki, S., & Yamagishi, J. (2016, September). Investigating RNN-based speech enhancement methods for noise-robust Text-to-Speech. In SSW. pp. 146-152.). For the speech part, it selected 28 speakers from the VCTK (Veaux, Christophe;Yamagishi, Junichi;MacDonald, Kirsten.(2017).CSTR VCTK Corpus:English Multi-speaker Corpus for CSTR VoiceCloning Toolkit,[sound].University of Edinburgh.The Centre for SpeechTechnology Research(CSTR).) speech dataset as the training set, and another 2 speakers as the test set; for the noise part, it selected 8 categories from DEMAND (Thiemann,J.,Ito,N.,&Vincent,E.(2013,June).The Diverse Environments Multi-channel Acoustic NoiseDatabase(DEMAND):A database of multichannel environmental noise recordings.InProceedings of Meetings on Acoustics ICA2013(Vol.19,No.1,p.035081).AcousticalSociety of Real noise from the United States and two types of generated noise were used for training, and five types of demodulated noise were selected for testing. The noisy speech SNRs (signal-to-noise ratios) for training data were {0dB, 5dB, 10dB, 15dB}, and for testing data were {2.5dB, 7.5dB, 12.5dB, and 17.5dB}. The training dataset was approximately 10 hours long.The corresponding objective evaluation indicators include PESQ (perceptual evaluation of speech quality), STOI (shorter-time objective intelligibility), and SI-SDR (scale-invariant signal-to-distortion ratios). The compared models include RNNoise (Valin, JM (2018, August). A hybrid DSP / deep learning approach to real-time full-band speech enhancement. In 2018IEEE 20th international workshop on multimedia signal processing (MMSP) (pp. 1-5). IEEE.), PerceptNet (Valin, JM, Isik, U., Phansalkar, N., Giri, R., Helwani, K., & Krishnaswamy, A. (2020). A perceptually-motivated approach for low-complexity, real-time enhancement of fullband speech.), and DeepFilterNet (. H., Rosenkranz, T., & Maier, A. (2021). DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-BandAudio based on Deep Filtering), S-DCCRN (Lv, S., Fu, Y., Xing, M., Sun, J., Xie, L., Huang, J.,... & Yu, T. (2021). S-DCCRN: Super Wide Band DCCRN with learnable complexfeature for speech enhancement.).

[0062] The performance of the SCM-DPARN model was further tested on the training and blind test datasets of DNS4 (Dubey, H., Gopal, V., Cutler, R., Aazami, A., Matusevych, S., Braun, S., ... & Aichner, R. (2022). ICASSP 2022 Deep Noise Suppression Challenge.). The DNS4 training set includes English, French, German, Italian, Russian, Spanish, emotional speech, and singing. The dataset contains approximately 70,000 audio samples (with a total duration of approximately 600 hours). This example uses 90% of the samples as a training set and 10% as a validation set. The room impulse responses (RIRs) required for reverberation generation come from openSLR26 and openSLR28 (Ko, T., Peddinti, V., Povey, D., Seltzer, ML, & Khudanpur, S. (2017, March). A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) pp. 5220-5224.). The clear speech is randomly convolved with the RIR and then added with noise to obtain a noisy reverberation signal. The signal-to-noise ratio (SNR) is randomly selected in the range of [-5dB, 15dB] with an interval of 1 dB. Because the blind test dataset consists of real, noisy speech recordings with no corresponding clean speech, the non-intrusive DNSMOS P.835 and DNSMOS P.808 metrics are used. BAK, SIG, and OVRL represent the noise suppression score, speech quality score, and overall quality score, respectively.The compared model is NsNet2 (Xia, Y., Braun, S., Reddy, CK, Dubey, H., Cutler, R., & Tashev, I. (2020, May). Weighted speech distortion losses for neural-network-based real-time speech enhancement. In ICASSP 2020-2020IEEEInternational Conference on Acoustics, Speech and Signal Processing(ICASSP)pp.871-875.).

[0063] 4. Experimental results.

[0064] The experimental results on the VCTK-DEMAND dataset are shown in Table 1. The present invention achieves the best scores in all indicators compared with other models.

[0065] Table 1. Results on the VCTK-DEMAND dataset.

[0066] Models Para.(M) PESQ STOI SI-SDR Noisy - 1.97 92.1 8.41 RNNoise 0.06 2.29 - - PerceptNet 8 2.73 - - DeepFilterNet 1.8 2.81 - 16.63 S-DCCRN 2.34 2.84 94 - SCM-DPARN 0.89 2.92 94.2 18.28

[0067] The results on the DNS4 dataset are as follows. The present invention has significant performance advantages over the baseline model NsNet2.

[0068] Table 2. DNS4 dataset results

[0069]

Claims

1. A full-band speech enhancement method based on spectrum compression and self-attention neural network, characterized in that: The method comprises the following steps: Step 1: Use the clear speech dataset, the noise dataset, and the room impulse response dataset to synthesize simulated noisy reverberant speech; Step 2: performing short-time Fourier transform on the simulated noisy reverberation speech data and the corresponding pure reverberation speech data to obtain short-time Fourier spectra of the two; Step 3: Construct an SCM-DPARN model. This model compresses the spectrum using a spectrum compression mapping module (SCM) and then inputs it into a dual-path attention recurrent neural network (DPARN) that uses a self-attention mechanism in the frequency domain. The SCM-DPARN model weights are trained using the short-time Fourier spectrum obtained in step 2. Step 4, performing short-time Fourier transform on the noisy reverberation speech signal to be enhanced to obtain a short-time Fourier spectrum; Step 5: Input the short-time Fourier spectrum obtained in step 4 into the SCM-DPARN model trained in step 3 to obtain the short-time Fourier spectrum of the enhanced speech; Step 6: Perform inverse short-time Fourier transform on the short-time Fourier spectrum of the enhanced speech to obtain a time domain signal of the enhanced speech.

2. The full-band speech enhancement method based on spectrum compression and self-attention neural network according to claim 1, characterized in that: In step 3, the spectrum compression mapping module SCM completely retains the low-frequency information in the 0kHz-5kHz frequency band and compresses the information in the 5kHz-24kHz frequency band. The compression matrix is ​​set as a learnable parameter and initialized using a logarithmic transformation curve. The formula of the compression curve is as follows: Among them, f and f c They are the original frequency band and the compressed frequency band respectively.

3. The full-band speech enhancement method based on spectrum compression and self-attention neural network according to claim 2, characterized in that: In step 3, the dual-path attention recurrent neural network DPARN consists of a set of codecs and an enhancement processing network, wherein the enhancement processing network uses a multi-head self-attention mechanism in the frequency domain to model the relationship between frequency points, and then uses a recursive neural network in the time domain to model the relationship between each frame of the spectrum.

Images