A speech enhancement method based on convolutional self-attention coding structure

Abstract

This invention discloses a speech enhancement method based on a convolutional self-attention coding structure. During the training phase, clean and noisy speech signals from the training set are subjected to short-time Fourier transforms, and an ideal proportional masking matrix is ​​calculated as the training label. The amplitude spectrum of the noisy speech is used as the input feature of the convolutional self-attention coding structure for training. The convolutional self-attention coding structure consists of an encoder, a decoder, residual dense blocks, and a self-attention coding module. During the testing phase, the convolutional self-attention coding structure maps the amplitude spectrum of the noisy test speech to a masking value matrix. Based on the masking values, the amplitude spectrum of the clean speech is reconstructed, and combined with the phase spectrum of the original noisy speech, the enhanced speech signal is obtained. Simulation tests under different acoustic environments show that, based on different objective speech quality evaluation methods, this invention significantly improves the objective evaluation index of noisy speech and has strong generalization ability for unknown acoustic environments.

Description

Technical Field

[0001] This invention relates to the field of speech signal denoising, and more particularly to a speech enhancement method based on a convolutional self-attention coding structure. Background Technology

[0002] Voice is the most convenient and widely used interaction method, with extensive applications in daily life. People have an urgent need to obtain high-quality voice signals. The purpose of voice enhancement technology is to extract useful voice signals from voice signals that are interfered with by noise, and to suppress and reduce noise interference. In recent years, learning-based methods have achieved voice enhancement performance far exceeding that of traditional methods. Depending on the training objective, these algorithms can be divided into three categories: mask-based time-frequency domain algorithms, mapping-based time-frequency domain algorithms, and end-to-end algorithms based on time-domain waveforms.

[0003] In recent years, neural networks have attracted much attention from researchers, and algorithms based on neural networks have achieved excellent performance. Numerous neural network architectures, such as Deep Neural Networks (DNNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Attention Networks, have been applied to speech enhancement. However, this research area still requires extensive and in-depth work to meet the growing demand for high-quality speech applications.

[0004] The spectrograms of speech signals exhibit temporal correlation. While RNNs theoretically can establish long-range temporal correlations, due to information transmission capacity limitations and the vanishing gradient problem, they can practically only establish short-range correlations. Self-attention coding structures based on multi-head self-attention mechanisms offer advantages over RNNs. Self-attention, a variant of attention mechanisms, effectively captures internal data correlations and dynamically generates weights based on all input information, thus fully modeling the long-range temporal correlations of speech signals. This invention combines a self-attention encoder to construct a convolutional-self-attention coding structure, achieving a significant improvement in objective evaluation metrics for speech enhancement tasks and demonstrating strong generalization ability. Summary of the Invention

[0005] Technical Problem: This invention addresses the problems existing in current speech enhancement techniques by proposing a speech enhancement method based on a convolutional self-attention coding structure, leveraging the powerful fitting ability of neural networks and the superior performance of attention mechanisms. Simulation tests under different acoustic environments demonstrate that the algorithm of this invention can significantly improve the objective evaluation index of noisy speech and has strong generalization ability for unknown acoustic environments.

[0006] Technical solution: The speech enhancement method based on convolutional self-attention coding structure described in this invention includes the following steps:

[0007] Step 1: Mix the single-channel speech signal to be trained with various additive noises at signal-to-noise ratios of 0, 5, and 10 dB to generate noisy speech, and combine it with clean speech to form a training dataset.

[0008] Step 2: Take the noisy speech and clean speech in the training dataset obtained in Step 1, and obtain the amplitude spectra of the noisy speech and clean speech respectively through frame segmentation, windowing and short-time Fourier transform. Then, calculate the ideal proportional masking matrix based on the amplitude spectra of the noisy speech and the corresponding clean speech.

[0009] Step 3: Use the amplitude spectrogram of the noisy speech and the ideal scale masking matrix obtained in Step 2 as the input features and labels of the convolutional self-attention coding structure, respectively, and train the convolutional self-attention coding structure based on the forward propagation and backward propagation algorithms.

[0010] Step 4: Mix the single-channel clean speech signal to be tested with various additive noises at arbitrary signal-to-noise ratios to obtain the test dataset. Then, through frame segmentation, windowing, and short-time Fourier transform, obtain the amplitude spectrum and phase spectrum of the noisy test speech.

[0011] Step 5: The amplitude spectrum of the test speech obtained in Step 4 is used as the input feature of the convolutional self-attention coding structure to estimate the ideal proportional masking matrix output. The amplitude spectrum of the enhanced speech is then multiplied element-wise with the amplitude spectrum of the noisy test speech obtained in Step 4. The phase spectrum of the enhanced speech signal is obtained using the phase spectrum of the noisy test speech obtained in Step 4. The time-domain waveform is obtained by inverse short-time Fourier transform and overlapping addition, thus realizing speech enhancement.

[0012] in:

[0013] The convolutional self-attention coding structure is an encoder-decoder structure based on a convolutional neural network. It adds a self-attention coding module between the convolutional layer and the deconvolutional layer, and uses residual dense blocks in the feature splicing path between the convolutional layer and the deconvolutional layer.

[0014] The convolutional self-attention coding structure uses a self-attention coding module between the convolutional and deconvolutional layers to capture the dependencies within the features. It uses two self-attention encoders, each containing eight multi-head self-attention units.

[0015] The convolutional self-attention encoding structure uses residual dense blocks to further process the output features of the convolutional layers in the feature concatenation path. Each residual dense block contains two cascaded convolutional blocks. The input of the residual dense block and the two convolutional blocks are connected by a feedback-free dense connection. The second convolutional block is followed by a convolutional layer with a kernel size of 1×1 to maintain the same size as the input feature of the residual dense block. Finally, residual connections are used to perform feature concatenation with the input features of the residual dense block to obtain the output features of the residual dense block.

[0016] The convolutional self-attention encoding structure uses a mean squared error loss function with ideally proportioned masking, and its formula is:

[0017]

[0018]

[0019] Where loss represents the loss function of the convolutional self-attention coding structure, and (f, k) is the f-th frequency value of the k-th frame of the speech signal after Fourier transform. IRM(f,k) is the ideal proportional masking matrix estimate obtained by mapping the amplitude spectrum of the noisy speech signal through a convolutional self-attention coding structure. IRM(f,k) is the ideal proportional masking value, and S(f,k) and N(f,k) are the spectra of the clean speech signal and the noisy signal, respectively.

[0020] Beneficial Effects: Compared with existing technologies, the significant advantages of this invention are: This invention extracts the amplitude spectrum of noisy speech signals as input features, and trains a convolutional self-attention coding structure using a suitable loss function to obtain a masking-based speech enhancement algorithm. Experimental results using this network to enhance noisy test speech under different acoustic environments demonstrate that the proposed speech enhancement method based on a convolutional self-attention coding structure can significantly improve the intelligibility and speech quality of noisy speech, and exhibits good robustness to unknown acoustic environments (unknown signal-to-noise ratio and noise type). Attached Figure Description

[0021] Figure 1 This is a flowchart of the speech enhancement system of the present invention;

[0022] Figure 2 This is a diagram showing the overall structure of the convolutional neural network of this invention;

[0023] Figure 3 This is a structural diagram of the residual dense block in the neural network of this invention;

[0024] Figure 4 This is a structural diagram of the self-attention encoding module in the neural network of this invention. Detailed Implementation

[0025] like Figure 1 As shown, the speech enhancement method based on a convolutional self-attention coding structure provided in this embodiment includes the following steps:

[0026] Step 1: Mix the single-channel clean speech signal to be trained with various additive noises at signal-to-noise ratios of 0, 5, and 10 dB to generate noisy speech. This invention uses open-source speech databases to train the network. The clean speech is from the NOIZEUS dataset, which contains 30 carefully selected sentences from the IEEE sentence dataset, read by three males and three females. The noise signal comes from the AURORA noise database, and the selected training noise types include crowd conversation noise, car noise, restaurant noise, street noise, and train noise.

[0027] Step 2: Perform frame segmentation, windowing, and short-time Fourier transform on the noisy speech and its corresponding clean speech in the training dataset obtained in Step 1, respectively, to obtain the amplitude spectra of the noisy speech and the clean speech. Then, calculate the ideal proportional masking matrix based on the amplitude spectra of the noisy speech and its corresponding clean speech. The speech sampling frequency is 8kHz, the frame length is 32ms, resulting in 256 sample points per frame. The frame shift is 8ms, and the FFT count is 256 (the same number of sample points as the frame length). The Hanning window is used, and the window function is...

[0028]

[0029] Frame segmentation and windowing operations can be expressed by the following formula:

[0030] x(k, n) = x(n + kL)w H (n)

[0031] y(k, n) = y(n + kL)w H (n)

[0032] Where x(n) and y(n) represent clean and noisy speech, respectively, w H (n) represents a Hanning window with a window length of N (N is 256 here), L is the number of sample points for frame shift (64 here); x(k,n) and y(k,n) are the clean speech and noisy speech of the kth frame after framing and windowing operations.

[0033] The short-time Fourier transform expression is:

[0034]

[0035]

[0036] Where X(f,k) and Y(f,k) are the spectra of the clean speech and the noisy speech in the kth frame at frequency point f, respectively.

[0037] The amplitude spectra |X(f,k)| and |Y(f,k)| of the noisy and clean speech are arranged according to frame order and frequency order, and then divided into several amplitude spectrum matrices with fixed frame lengths, which are the corresponding amplitude spectra. In this invention, the amplitude spectra of the noisy and clean speech are divided into 129×129 matrices, with their dimensions representing time frames and frequency points, respectively. The resulting matrices are the amplitude spectra. An ideal proportional masking matrix is ​​calculated based on the spectra of the noisy and corresponding clean speech. Similar to the above processing of the amplitude spectra, the obtained ideal proportional masking matrix is ​​divided into a 129×129 matrix, and then compared with the amplitude spectrum... Figure 1 One-to-one correspondence.

[0038] The ideal scaling mask expression is:

[0039]

[0040] Where IRM(f,k) is the ideal proportional masking value at the k-th frame and the f-th frequency point, and S(f,k) and N(f,k) are the spectra of the clean speech signal and the noise signal, respectively.

[0041] Step 3: Use the noisy speech amplitude spectrogram obtained in Step 2 as the input feature of the convolutional self-attention coding structure, and use the corresponding ideal scale masking matrix as the training target to train the convolutional self-attention coding structure.

[0042] This invention proposes a convolutional self-attention coding structure, the structure of which is given below. In fact, the structure of the convolutional self-attention coding structure proposed in this invention is not limited to the network structure of this embodiment; minor modifications also fall within the scope of the convolutional self-attention coding structure.

[0043] The convolutional self-attention encoding structure is an encoder-decoder structure based on U-Net.

[0044] The encoder contains three cascaded convolutional blocks. These blocks achieve a 4x downsampling by using strides of 1 and 2 and increasing the number of output channels, thereby reducing the size of the feature maps and increasing their channels. Each convolutional block includes the following operations: input convolutional layer, batch normalization, dropping 50% of neurons, and ReLU activation. The three convolutional layers have 16, 32, and 64 output channels, and kernel sizes of 7×7, 5×5, and 3×3, respectively. The latter two convolutional layers use a stride of 2 for downsampling.

[0045] The decoder consists of cascaded three deconvolutional blocks and two convolutional blocks. The three deconvolutional blocks employ a stride symmetrical to the encoder's convolutional blocks and a decreasing number of output channels to achieve a 4x upsampling, thus mapping low-resolution, high-channel-count feature maps to high-resolution, low-channel-count feature maps. Each deconvolutional block includes the following operations: input deconvolution layer, batch normalization, discarding 50% of neurons, and ReLU activation. The three deconvolutional layers have output channels of 32, 16, and 8 respectively, and use convolutional kernel sizes and strides symmetrical to the decoder's convolutional layers. The two convolutional blocks further reduce the number of channels to 1 using 3×3 convolutional kernels, ensuring that the output of the convolutional self-attention encoding structure has the same size as the input feature map.

[0046] There is a feature concatenation path between the first and second convolutional blocks of the encoder and the second and third deconvolutional blocks of the decoder. A residual dense block is used in this path to further process the output features of the encoder's convolutional blocks. The residual dense block contains two cascaded convolutional blocks. A feedback-free dense connection is used between the input of the residual dense block and the two convolutional blocks. The second convolutional block is followed by a 1×1 convolutional layer to maintain the same kernel size as the input features of the residual dense block. Finally, a residual connection is used to perform feature concatenation with the input features of the residual dense block to obtain the output features of the residual dense block.

[0047] Between the encoder and decoder is a self-attention encoding module, used to establish long-range dependencies within the encoder output features. The module uses two self-attention encoders. Each self-attention encoder has eight multi-head self-attentions. The self-attention output is added to the input through a residual connection and then normalized to obtain the attention output. Then, the attention output is added to the output of the feedforward layer through a residual connection and normalized to obtain the output of the self-attention encoding module.

[0048] The loss function is the mean squared error of the ideally scaled mask:

[0049]

[0050] Where loss represents the loss function of the convolutional self-attention coding structure, and (f, k) is the f-th frequency value of the k-th frame of the speech signal after Fourier transform. IRM(f,k) is the masking value obtained by mapping the amplitude spectrum of the noisy speech signal through a convolutional self-attention coding structure. IRM(f,k) is the ideal proportional masking value. S(f,k) and N(f,k) are the spectra of the clean speech signal and the noisy signal, respectively.

[0051] Step 4: In the testing phase, for the noisy test speech, following the method in Step 2, the amplitude spectrum and phase spectrum of the noisy test speech are obtained through frame segmentation, windowing, and short-time Fourier transform.

[0052] Step 5: Using the amplitude spectrum of the test noisy speech from Step 4 as the input feature of the trained convolutional self-attention coding structure, the estimated value of the ideal proportional masking matrix is ​​obtained. This value is then multiplied element-wise with the input amplitude spectrum of the test noisy speech to obtain the amplitude spectrum of the enhanced speech. Using the phase spectrum of the test noisy speech from Step 4, the spectrum of the enhanced speech signal is obtained. After short-time inverse Fourier transform and overlapping addition, the time-domain waveform is obtained, thus achieving speech enhancement.

[0053] This invention evaluates the effectiveness of speech enhancement using various objective metrics for assessing speech quality and intelligibility. Two metrics are employed: Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI). The PESQ score, based on the ITU-T P.862 standard, is an objective speech quality assessment method that uses the original signal as a reference to measure the quality of the degraded signal, returning a score ranging from -0.5 to 4.5. The STOI score is an objective evaluation method that measures the intelligibility of speech by the human auditory perception system, with a score between 0 and 1. It uses the original signal as a reference to measure the intelligibility of the degraded signal.

[0054] The above methods were simulated and verified, and the final performance evaluation is as follows:

[0055] (1) Indicators under matched acoustic conditions

[0056] For five noise types—crowd conversation noise, car noise, restaurant noise, street noise, and train noise—noise and clean speech were mixed using signal-to-noise ratios of 0, 5, and 10 dB to generate noisy speech training sets, while test data was generated using the same acoustic environment. The convolutional self-attention coding structure used in this invention was compared with CNN (U-Net-like structure), CNN (U-Net-like structure without feature paths), DNN, and RNN, and the speech enhancement effect was evaluated using PESQ and STOI metrics.

[0057] Table 1. PESQ index under matched acoustic conditions

[0058] SNR = 0dB SNR = 5dB SNR = 10dB noisy 1.4095 1.5819 1.8791 This invention 1.7392 2.0065 2.2928 CNN 1.4960 1.7287 2.0240 CNN (Feature-less Pathway) 1.4976 1.7164 1.9968 DNN 1.4947 1.6950 1.9460 RNN 1.4747 1.6729 1.9195

[0059] Table 2. STOI Index under Matched Acoustic Conditions

[0060] SNR = 0dB SNR = 5dB SNR = 10dB noisy 0.6762 0.7850 0.8798 This invention 0.8144 0.8839 0.9344 CNN 0.7159 0.8251 0.9074 CNN (Feature-less Pathway) 0.7139 0.8213 0.9022 DNN 0.7199 0.8236 0.9048 RNN 0.7557 0.851 0.9180

[0061] (2) Indicators under unmatched acoustic conditions

[0062] For five noise types—crowd conversation noise, car noise, restaurant noise, street noise, and train noise—the training set was generated by mixing noise and clean speech with signal-to-noise ratios (SNRs) of 0, 5, and 10 dB. The test data used SNRs of -2.5, 2.5, and 7.5 dB, with noise types including white noise, factory noise, engine noise, and pink noise. The convolutional self-attention encoding structure of this invention was compared with CNN (U-Net-like structure), CNN (U-Net-like structure without feature paths), DNN, and RNN. The PESQ and STOI metrics are shown in the table below.

[0063] Table 3. PESQ index under unmatched acoustic conditions

[0064] SNR = -2.5dB SNR = 2.5dB SNR = 7.5dB noisy 1.3358 1.5102 1.7733 This invention 1.4975 1.7102 1.992 CNN 1.4159 1.6408 1.9140 CNN (Feature-less Pathway) 1.4124 1.629 1.8954 DNN 1.3982 1.6273 1.8887 RNN 1.3887 1.6024 1.8688

[0065] Table 4. STOI Index under Unmatched Acoustic Conditions

[0066] SNR = -2.5dB SNR = 2.5dB SNR = 7.5dB noisy 0.6094 0.7109 0.7983 This invention 0.6964 0.7890 0.8544 CNN 0.6355 0.7378 0.8225 CNN (Feature-less Pathway) 0.6324 0.7354 0.8175 DNN 0.6356 0.7349 0.8220 RNN 0.6351 0.7434 0.8251

[0067] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention; these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A speech enhancement method based on a convolutional self-attention coding structure, characterized in that, Includes the following steps: Step 1: Mix the single-channel speech signal to be trained with various additive noises at signal-to-noise ratios of 0, 5, and 10 dB to generate noisy speech, and combine it with clean speech to form a training dataset. Step 2: Take the noisy speech and clean speech in the training dataset obtained in Step 1, and obtain the amplitude spectra of the noisy speech and clean speech respectively through frame segmentation, windowing and short-time Fourier transform. Then, calculate the ideal proportional masking matrix based on the amplitude spectra of the noisy speech and the corresponding clean speech. Step 3: Use the amplitude spectrogram of the noisy speech and the ideal scale masking matrix obtained in Step 2 as the input features and labels of the convolutional self-attention coding structure, respectively, and train the convolutional self-attention coding structure based on the forward propagation and backward propagation algorithms. Step 4: Mix the single-channel clean speech signal to be tested with various additive noises at arbitrary signal-to-noise ratios to obtain the test dataset. Then, through frame segmentation, windowing, and short-time Fourier transform, obtain the amplitude spectrum and phase spectrum of the noisy test speech. Step 5: The amplitude spectrum of the test speech obtained in Step 4 is used as the input feature of the convolutional self-attention coding structure to estimate the ideal proportional masking matrix output. The amplitude spectrum of the enhanced speech is then multiplied element-wise with the amplitude spectrum of the noisy test speech obtained in Step 4. The phase spectrum of the enhanced speech signal is obtained using the phase spectrum of the noisy test speech obtained in Step 4. The time-domain waveform is obtained by inverse short-time Fourier transform and overlapping addition to achieve speech enhancement. The convolutional self-attention coding structure is an encoder-decoder structure based on a convolutional neural network. It adds a self-attention coding module between the convolutional layer and the deconvolutional layer, and uses residual dense blocks in the feature concatenation path between the convolutional layer and the deconvolutional layer. The convolutional self-attention coding structure uses a self-attention coding module between the convolutional and deconvolutional layers to capture the dependencies within the features. It uses two self-attention encoders, each containing eight multi-head self-attention modules. The convolutional self-attention encoding structure uses residual dense blocks to further process the output features of the convolutional layers in the feature concatenation path. Each residual dense block contains two cascaded convolutional blocks. The input of the residual dense block and the two convolutional blocks are connected by a feedback-free dense connection. The second convolutional block is followed by a convolutional layer with a kernel size of 1×1 to maintain the same size as the input feature of the residual dense block. Finally, residual connections are used to perform feature concatenation with the input features of the residual dense block to obtain the output features of the residual dense block.

2. The speech enhancement method based on a convolutional self-attention coding structure according to claim 1, characterized in that: The convolutional self-attention encoding structure uses a mean squared error loss function with ideally proportioned masking, and its formula is: Where loss represents the loss function of the convolutional self-attention coding structure, and (f,k) is the f-th frequency value of the k-th frame of the speech signal after Fourier transform. IRM(f,k) is the ideal proportional masking matrix estimate obtained by mapping the amplitude spectrum of the noisy speech signal through a convolutional self-attention coding structure. IRM(f,k) is the ideal proportional masking value, and S(f,k) and N(f,k) are the spectra of the clean speech signal and the noisy signal, respectively.

Images