Speech Enhancement Method and Device Based on Attention-Enhanced Dual-Path Convolutional Recurrent Network

Abstract

This invention discloses a speech enhancement method and device based on an enhanced attention dual-path convolutional recurrent network. The method includes: framing and windowing noisy training speech signals, and calculating the improved discrete cosine transform (DCT) spectrogram; constructing an enhanced attention dual-path convolutional recurrent network, consisting of an encoder module, a SimAM attention block, a dual-path long short-term memory (LSTM) module, a decoder module, and a hardtanh function mapping module; using the improved DCT spectrogram of the noisy training speech and clean speech as the input features and labels of the network, respectively, and training the enhanced attention dual-path convolutional recurrent network based on a combined loss function; using the improved DCT spectrogram of the noisy test speech signal as the input feature, the enhanced attention dual-path convolutional recurrent network estimates the improved DCT spectrogram of the corresponding clean speech, and after inverse transformation, obtains the enhanced speech in the time domain. This invention achieves better perceptual speech quality with a smaller number of parameters and computational load.

Description

Technical Field

[0001] This invention relates to speech enhancement technology, and more particularly to a speech enhancement method and device based on an enhanced attention dual-path convolutional recurrent network. Background Technology

[0002] Speech enhancement is a key technology in the field of signal processing, aiming to recover clean speech signals from noisy speech signals. This technology is of great significance for improving the quality of voice communication, enhancing the accuracy of automatic speech recognition systems, and improving the performance of hearing aids. Based on the input and output, it can be divided into: frequency domain-based speech enhancement technology and time domain-based speech enhancement technology. Frequency domain-based speech enhancement technology can be further divided into: speech enhancement technology based on short-time Fourier transform and speech enhancement technology based on short-time discrete cosine transform.

[0003] Neural networks are an effective technique. In recent years, convolutional neural networks, recurrent neural networks, long short-term memory networks, and attention mechanisms have been widely applied in the field of speech enhancement. However, convolutional neural networks mainly capture local information and lack the ability to understand contextual information; traditional long short-term memory networks can learn time dependencies well, but cannot effectively extract frequency information; attention mechanisms can improve the information extraction ability of the network, but introduce additional parameters and increase computation. These problems limit the further application of speech enhancement technology. Therefore, the field of speech enhancement still needs extensive and in-depth research to meet the growing demand for high-quality speech. Summary of the Invention

[0004] To address the problems existing in the prior art, the purpose of this invention is to provide a speech enhancement method and device based on an enhanced attention dual-path convolutional recurrent network, which can achieve better perceptual speech quality with a smaller number of parameters and computational load.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0006] A speech enhancement method based on an attention-enhanced dual-path convolutional recurrent network includes the following steps:

[0007] Step 1: Mix the single-channel clean training speech signal with various additive noises at different signal-to-noise ratios to obtain several noisy training speech signals;

[0008] Step 2: Preprocess the noisy training speech signal and calculate the improved discrete cosine transform spectrum of the preprocessed noisy training speech signal.

[0009] Step 3: Construct an enhanced attention dual-path convolutional recurrent network. The enhanced attention dual-path convolutional recurrent network consists of a SimAM attention module, a dual-path long short-term memory module, a hardtanh function mapping module, several encoder modules, and several decoder modules. The encoder modules, dual-path long short-term memory module, several decoder modules, and hardtanh function mapping module are connected sequentially. Each encoder module and its corresponding decoder module are also connected via the SimAM attention module. The encoder module is used for downsampling and feature extraction, the decoder module is used for feature processing and upsampling, the dual-path long short-term memory module is used to capture contextual dependencies, and the hardtanh function mapping module is used to scale the improved discrete cosine transform spectrum.

[0010] Step 4: Use the improved discrete cosine transform spectrogram of the noisy training speech signal as the input feature of the enhanced attention dual-path convolutional recurrent network, and the clean training speech signal as the label to train the network.

[0011] Step 5: After preprocessing the noisy test language signal to be enhanced, calculate the improved discrete cosine transform spectrum.

[0012] Step 6: Use the improved discrete cosine transform spectrum of the noisy test speech signal as the input feature of the enhanced attention dual-path convolutional recurrent network to obtain the improved discrete cosine transform spectrum of the estimated clean speech. Then, after inverse improved discrete cosine transform and post-processing, the time-domain enhanced speech signal is obtained.

[0013] Furthermore, step 2 specifically includes:

[0014] Step 2.1: Perform frame segmentation and windowing preprocessing on the noisy training speech signal, and perform improved discrete cosine transform on the preprocessed noisy training speech signal to obtain the improved discrete cosine transform spectrum of each frame of speech signal.

[0015] Step 2.2: Arrange the improved discrete cosine transform spectra of several frames of the noisy training speech signal according to the frame order and frequency point order to form an improved discrete cosine transform spectrum matrix, which is the improved discrete cosine transform spectrum diagram.

[0016] Furthermore, the encoder module comprises a feature reshaping submodule and a feature processing submodule. The feature reshaping submodule and the feature processing submodule have the same structure, both including a convolutional layer, a batch normalization layer and a PReLU activation function connected in sequence, but the parameters of the convolutional layer are different.

[0017] Furthermore, the SimAM attention module is specifically an energy function.

[0018] Furthermore, the dual-path long short-term memory module includes an intra-frame bidirectional recurrent network and an inter-frame unidirectional recurrent network. The intra-frame bidirectional recurrent network includes a bidirectional recurrent neural network, a fully connected layer, and a layer normalization layer connected in sequence. The inter-frame unidirectional recurrent network includes a unidirectional recurrent neural network, a fully connected layer, and a layer normalization layer connected in sequence. Residual connections are made between the intra-frame bidirectional recurrent network and the inter-frame unidirectional recurrent network.

[0019] Furthermore, the decoder module includes a feature processing submodule and a feature reshaping submodule. The feature processing submodule includes a convolutional layer, a batch normalization layer, and a PReLU activation function connected in sequence. The feature reshaping submodule includes a convolutional layer, a pixel reconstruction layer, a batch normalization layer, and a PReLU activation function connected in sequence.

[0020] Furthermore, the output of the feature reshaping submodule of the encoder module is connected to the input of the feature reshaping submodule of the corresponding decoder module via a SimAM attention block, and the output of the feature processing submodule of the encoder module is connected to the input of the feature processing submodule of the corresponding decoder module via a SimAM attention block.

[0021] Furthermore, the loss function used during training of the enhanced attention dual-path convolutional recurrent network is:

[0022]

[0023] Where Loss represents the total loss, parameter α represents the weighting coefficient, with a value ranging from 0 to 1, SISNR represents the scale-invariant signal-to-noise ratio, and PMSQE represents the perceptual metric for speech quality assessment. represents the enhanced time-domain speech signal, and s represents the clean time-domain speech signal.

[0024] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described above.

[0025] A computer program product includes a computer program / instructions that, when executed by a processor, implement the above-described method.

[0026] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention introduces an improved discrete cosine transform (DCT), which, based on the real-valued characteristics of the improved DCT, avoids complex complex networks and reduces the number of feature points by half, thus reducing computational load and model computation time. This invention introduces the SimAM attention mechanism, which can enhance information capture without increasing the number of parameters, further reducing computational load and shortening the required model computation time. This invention introduces a dual-channel long short-term memory (LSTM) module, establishing models for both intra-frame and inter-frame information, enabling the capture of intra-frame information features while maintaining the causality of inter-frame information. Attached Figure Description

[0027] Figure 1 This is a schematic flowchart of the speech enhancement method based on an enhanced attention dual-path convolutional recurrent network provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the structure of the enhanced attention dual-path convolutional recurrent network provided by the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of the dual-path cyclic network provided by the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0032] Example 1

[0033] This invention provides a speech enhancement method based on an enhanced attention dual-path convolutional recurrent network, such as... Figure 1 As shown, it includes the following steps:

[0034] Step 1: Mix the single-channel clean training speech signal with various additive noises at different signal-to-noise ratios to obtain several noisy training speech signals.

[0035] The speech signals can be used to train the network using an open-source speech database. In this embodiment, the dataset used is the VoiceBank+DEMAND dataset. The clean speech comes from the VoiceBank corpus, consisting of English passages read by 30 speakers of different genders using different accents. The DEMAND noise dataset contains 13 different types of noise. The training dataset contains speech from 28 speakers, which is mixed with 10 types of noise at signal-to-noise ratios (SNRs) of 0dB, 5dB, 10dB, and 15dB. The test dataset contains speech from 2 speakers, which is mixed with 2 types of noise at SNRs of 2.5dB, 7.5dB, 12.5dB, and 17.5dB. In summary, the training dataset contains 11,572 speech samples from 40 different acoustic environments, and the test dataset contains 824 speech samples from 20 different acoustic environments.

[0036] Step 2: Preprocess the noisy training speech signal and calculate the improved discrete cosine transform spectrum of the preprocessed noisy training speech signal.

[0037] This step specifically includes:

[0038] Step 2.1: Perform frame segmentation and windowing preprocessing on the noisy training speech signal, and perform improved discrete cosine transform on the preprocessed noisy training speech signal to obtain the improved discrete cosine transform spectrum (MDCT) of each frame of speech signal.

[0039] In practice, the sampling frequency of both the clean training speech signal and the noisy training speech signal is 16kHz, the frame length is 16ms, the frame shift is 8ms, and the Vorbis window function is selected.

[0040] The noisy speech signal x(i,n) in the i-th frame after framing and windowing is:

[0041] x(i,n)=x(n-iL)w V (n)

[0042] Where w V (n) is a 256-point Vorbis window, i is the frame number, n is the sample number of each frame, and L is the frame length.

[0043]

[0044] The improved discrete cosine transform is as follows:

[0045]

[0046] Where X(i,k) is the improved discrete cosine transform spectrum of the i-th frame of the noisy training speech, k represents the frequency point, and k is 128 points.

[0047] Step 2.2: Arrange the improved discrete cosine transform spectra of several frames of the noisy training speech signal according to the frame order and frequency point order to form an improved discrete cosine transform spectrum matrix, which is the improved discrete cosine transform spectrum diagram.

[0048] In this embodiment, the improved discrete cosine transform spectrum of the noisy training speech signal is divided into a 377×128 matrix according to the frame order (center padding, zero padding on both sides, resulting in 377 time frames). Its dimensions represent the time frame and frequency point, respectively, and the resulting improved discrete cosine transform spectrum is obtained.

[0049] Step 3: Construct an enhanced attention dual-path convolutional recurrent network.

[0050] like Figure 2 As shown, the enhanced attention dual-path convolutional recurrent network comprises a SimAM attention module, a dual-path long short-term memory module, a hardtanh function mapping module, several encoder modules, and several decoder modules. These encoder modules, modules, and decoder modules are sequentially connected. Each encoder module is also connected to its corresponding decoder module via the SimAM attention module. The encoder modules are used for downsampling and feature extraction, the decoder modules are used for feature processing and upsampling, the dual-path long short-term memory module is used to capture contextual dependencies, and the hardtanh function mapping module is used to scale the improved discrete cosine transform spectrum. In fact, the structure of the enhanced attention dual-path convolutional recurrent network proposed in this invention is not limited to the network structure of this embodiment; minor modifications also fall within the scope of temporal attention convolutional neural networks.

[0051] like Figure 2As shown, the encoder module includes a feature reshaping submodule and a feature processing submodule. Both submodules consist of sequentially connected convolutional layers, batch normalization layers, and PReLU activation functions. The convolutional layer parameters differ between the two submodules. The feature reshaping submodule in the encoder reshapes the feature dimension using a convolutional layer with a kernel size of (3,5) and a stride of (1,2), while maintaining the time dimension. This reduces the feature map size and increases the number of channels, achieving downsampling in the frequency dimension. The feature processing submodule extracts features using a convolutional layer with a kernel size of (3,5) and a stride of (1,1). The feature processing submodule in the decoder has the same structure as the encoder's submodule. The decoder's feature reshaping submodule reshapes the dimension using convolutional layers and pixel reconstruction. The convolutional layer has a kernel size of (3,5) and a stride of (1,1). The output channel number of the convolutional layer is 2 in the second decoder reshaping submodule's convolutional layer. Pixel reconstruction is then used to increase the feature map size and restore the number of channels. The number of feature channels for the encoder and decoder are [1,32,32,64,64] and [64,64,32,32,1], respectively.

[0052] The SimAM attention module is an energy function. This energy function, based on the linear separability of neurons, assigns higher priority to active neurons exhibiting significant spatial inhibition effects, and is calculated by minimizing the difference between the target neuron and other neurons. SimAM requires no additional parameters and is lightweight.

[0053] like Figure 3 As shown, the dual-path long short-term memory module consists of an intra-frame bidirectional recurrent neural network and an inter-frame unidirectional recurrent neural network. The intra-frame bidirectional recurrent neural network includes a bidirectional recurrent neural network, a fully connected layer, and a layer normalization layer connected in sequence. The inter-frame unidirectional recurrent neural network includes a unidirectional recurrent neural network, a fully connected layer, and a layer normalization layer connected in sequence. Both types of recurrent neural networks have 32 hidden layer neurons and 64 input points. The intra-frame bidirectional recurrent neural network circulates information along the frequency direction, reshaping the input feature tensor into [batch size * number of time frames, frequency point, number of channels]; the inter-frame unidirectional recurrent neural network circulates information along the time direction, reshaping the input feature tensor into [batch size * frequency point, number of time frames, number of channels].

[0054] The hardtanh function mapping module is used to scale the improved cosine spectrum. The hardtanh function defines a maximum and a minimum value. If the input value of the function is within the range of the maximum and minimum values, the output remains unchanged. If it exceeds the maximum value, the output is the maximum value. Similarly, if the input value is less than the minimum value, the output is the minimum value.

[0055] Step 4: Use the improved discrete cosine transform spectrogram of the noisy training speech signal as the input feature of the enhanced attention dual-path convolutional recurrent network, and the clean training speech signal as the label to train the network.

[0056] The training process uses forward propagation and backward propagation algorithms, with the loss function being:

[0057]

[0058] Where Loss represents the total loss, parameter α represents the weighting coefficient, with a value ranging from 0 to 1, SISNR represents the scale-invariant signal-to-noise ratio, and PMSQE represents the perceptual metric for speech quality assessment. represents the enhanced time-domain speech signal, and s represents the clean time-domain speech signal.

[0059] Step 5: After preprocessing the noisy test language signal to be enhanced, calculate the improved discrete cosine transform spectrum.

[0060] The preprocessing and calculation processes are the same as in step 2.

[0061] Step 6: Use the improved discrete cosine transform spectrum of the noisy test speech signal as the input feature of the enhanced attention dual-path convolutional recurrent network to obtain the improved discrete cosine transform spectrum of the enhanced speech. Then, after inverse improved discrete cosine transform and post-processing, the time-domain enhanced speech signal is obtained.

[0062] The formulas for the inverse improved discrete cosine transform and windowing are as follows:

[0063]

[0064] in The improved discrete cosine transform spectrum of the i-th frame of enhanced speech obtained by passing the noisy test speech through an enhanced attention dual-path convolutional recurrent network, where i is the frame number, y(i,n) is the temporal enhanced speech signal of the i-th frame after inverse transform, and w v (n) is consistent with the above.

[0065] The time-domain waveform of the enhanced speech signal is obtained by overlapping and adding the signals using the principle of time-domain anti-aliasing.

[0066] To verify the effectiveness of this invention, simulations were performed on the above methods. The Perceptual Evaluation of Speech Quality (PESQ) and the Monograph for the Overall Speech Quality (COVL) were used to evaluate speech quality and intelligibility. PESQ is a widely used objective speech quality assessment method. Standardized by the International Telecommunication Union (ITU) as ITU-T Recommendation P.862 (02 / 01), it aims to simulate subjective human evaluation of speech quality, with scores ranging from -0.5 to 4.5; higher scores indicate better speech quality. COVL is an objective speech quality evaluation index used to predict the overall quality of speech signals after speech enhancement or other speech processing techniques. Its score ranges from 1 to 5, with higher scores indicating better speech quality.

[0067] The baseline models compared to this invention are DCTCRN and VSANet. The DCTCRN model is a speech enhancement model based on a U-Net network and discrete cosine transform; the VSANet model is a speech enhancement model based on a multi-task learning U-Net network and discrete cosine transform, where the U-Net network is an encoder-decoder structure. Compared to these two baseline models, this invention shows a significant improvement in objective evaluation metrics, as detailed below:

[0068] Table 1 Comparison of PESQ and COVL indices with different models

[0069]

[0070] Example 2

[0071] Figure 4 This is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. The embodiment of the present invention provides services for the implementation of the method of the first embodiment of the present invention. Figure 4 As shown, the device may include: a memory 301 storing a computer-executable program; a processor 302 coupled to the memory 301; the processor 302 calls the computer-executable program stored in the memory 301 to perform the steps in the method described in Embodiment 1.

[0072] Memory 301 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, memory 301 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard disk drive"). A program / utility having a set (at least one) of program modules may be stored, for example, in memory 301. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The computer-executable program of the program modules typically performs the functions and / or methods described in the embodiments of the present invention.

[0073] The processor 302 executes various functional applications and data processing by running programs stored in the memory 301, such as implementing the method provided in Embodiment 1 of the present invention.

[0074] The code of a computer executable program can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages.

[0075] Example 3

[0076] This invention also provides a computer product, such as an app on a mobile phone or tablet, or an installer on a computer. This product includes a computer program / instructions that, when executed by a processor, implement the method described in Embodiment 1. The code for a computer-executable program that performs the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0077] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0078] It should be understood that the embodiments and descriptions above are only the principles, main features and advantages of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the invention, and all such changes and modifications fall within the protection scope of the present invention.

Claims

1. A speech enhancement method based on an enhanced attention dual-path convolutional recurrent network, characterized in that, Includes the following steps: Step 1: Mix the single-channel clean training speech signal with various additive noises at different signal-to-noise ratios to obtain several noisy training speech signals; Step 2: Preprocess the noisy training speech signal and calculate the improved discrete cosine transform spectrum of the preprocessed noisy training speech signal; the improved discrete cosine transform spectrum is obtained by the following formula: ， in, For noisy training speech The improved discrete cosine transform spectrum of the frame Indicates frequency point, Indicates the first preprocessed step Noisy speech signal frame Here, N is the sample number for each frame, and N is the total number of samples. Step 3: Construct an enhanced attention dual-path convolutional recurrent network. This network comprises a SimAM attention module, a dual-path long short-term memory module, a hardtanh function mapping module, several encoder modules, and several decoder modules. These encoder modules, modules, decoder modules, and hardtanh function mapping modules are sequentially connected. Each encoder module is also connected to its corresponding decoder module via the SimAM attention module. The encoder modules are used for downsampling and feature extraction, the decoder modules are used for feature processing and upsampling, the dual-path long short-term memory module is used to capture contextual dependencies, and the hardtanh function mapping module is used to scale the improved discrete cosine transform spectrum. The memory module includes an intra-frame bidirectional recurrent neural network and an inter-frame unidirectional recurrent neural network. The intra-frame bidirectional recurrent neural network includes a bidirectional recurrent neural network, a fully connected layer, and a layer normalization layer connected in sequence. The inter-frame unidirectional recurrent neural network includes a unidirectional recurrent neural network, a fully connected layer, and a layer normalization layer connected in sequence. The intra-frame bidirectional recurrent neural network and the inter-frame unidirectional recurrent neural network are connected via residual connections. The encoder module includes a feature reshaping submodule and a feature processing submodule. The decoder module includes a feature processing submodule and a feature reshaping submodule. The output of the feature reshaping submodule of the encoder module is connected to the input of the feature reshaping submodule of the corresponding decoder module via a SimAM attention block. The output of the feature processing submodule of the encoder module is connected to the input of the feature processing submodule of the corresponding decoder module via a SimAM attention block. Step 4: Use the improved discrete cosine transform spectrogram of the noisy training speech signal as the input feature of the enhanced attention dual-path convolutional recurrent network, and the clean training speech signal as the label to train the network. Step 5: After preprocessing the noisy test language signal to be enhanced, calculate the improved discrete cosine transform spectrum. Step 6: Use the improved discrete cosine transform spectrum of the noisy test speech signal as the input feature of the enhanced attention dual-path convolutional recurrent network to obtain the improved discrete cosine transform spectrum of the estimated clean speech. Then, after inverse improved discrete cosine transform and post-processing, the time-domain enhanced speech signal is obtained.

2. The speech enhancement method based on an enhanced attention dual-path convolutional recurrent network according to claim 1, characterized in that: Step 2 specifically includes: Step 2.1: Perform frame segmentation and windowing preprocessing on the noisy training speech signal, and perform improved discrete cosine transform on the preprocessed noisy training speech signal to obtain the improved discrete cosine transform spectrum of each frame of speech signal. Step 2.2: Arrange the improved discrete cosine transform spectra of several frames of the noisy training speech signal according to the frame order and frequency point order to form an improved discrete cosine transform spectrum matrix, which is the improved discrete cosine transform spectrum diagram.

3. The speech enhancement method based on an enhanced attention dual-path convolutional recurrent network according to claim 1, characterized in that: The encoder module consists of a feature reshaping submodule and a feature processing submodule. The feature reshaping submodule and the feature processing submodule have the same structure, both including a convolutional layer, a batch normalization layer and a PReLU activation function connected in sequence, but the parameters of the convolutional layer are different.

4. The speech enhancement method based on an enhanced attention dual-path convolutional recurrent network according to claim 1, characterized in that: The SimAM attention module is specifically an energy function.

5. The speech enhancement method based on an enhanced attention dual-path convolutional recurrent network according to claim 3, characterized in that: The decoder module includes a feature processing submodule and a feature reshaping submodule. The feature processing submodule includes a convolutional layer, a batch normalization layer, and a PReLU activation function connected in sequence. The feature reshaping submodule includes a convolutional layer, a pixel reconstruction layer, a batch normalization layer, and a PReLU activation function connected in sequence.

6. The speech enhancement method based on an enhanced attention dual-path convolutional recurrent network according to claim 1, characterized in that: The loss function used during training of the enhanced attention dual-path convolutional recurrent network is: ， in, Represents the total loss, parameter This represents the weighting coefficient, with a value ranging from 0 to 1. This represents the scale-invariant signal-to-noise ratio. This represents a perceptual metric for speech quality assessment. This indicates an enhanced time-domain speech signal. This represents a clean time-domain speech signal.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: The processor executes the computer program to implement the method as described in any one of claims 1-6.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of any one of claims 1-6.

Images