Single-channel speech enhancement method based on improved convolutional recurrent network

Abstract

The invention provides a single-channel speech enhancement method based on an improved convolutional recurrent network, and the method comprises the steps: 1, downloading a data set for model training and testing, carrying out the preprocessing of the speech of the data set through employing a VoiceBank + DEMAND data set, and extracting the amplitude spectrum of a speech signal through short-time Fourier transform (STFT); 2, inputting the amplitude spectrum into an encoder, and extracting high-dimensional features from the amplitude spectrum; step 3, using the Channel-S4D block as a recursion module of the model, inputting a result output by the Channel-S4D block into a decoder, and recovering the result to an original dimension; 4, constructing a joint loss function; and 5, reconstructing and evaluating the enhanced voice signal. According to the invention, the Channel-S4D module constructed based on the diagonalization state space model efficiently processes a voice sequence, captures channel-level voice key information, better models the time dependence relationship of voice signals, and improves the adaptability of the model to a complex noise environment while reducing the parameter quantity and the calculation quantity of the model.

Description

Technical Field

[0001] The present invention relates to a single-channel speech enhancement method based on an improved convolutional recurrent network, and in particular to a novel, efficient and lightweight convolutional recurrent network (CRNv2) for single-channel speech enhancement, belonging to the technical field of speech enhancement. Background Art

[0002] In modern society, clear speech is the basis of effective communication. However, adverse factors such as background noise and reverberation often impair the intelligibility and quality of speech signals, affecting the quality of our daily voice communication. Therefore, it is crucial to study speech enhancement (SE) technology to solve noise interference. Speech enhancement technology aims to improve the quality and clarity of speech signals, making them easier to understand and recognize. In recent years, with the development of deep learning technology, speech enhancement methods based on deep learning have gradually become a hot topic of research. Deep learning can automatically learn the features and patterns in speech signals through training with large-scale data, so that the model can more accurately analyze and reconstruct damaged speech signals. Some common models of deep learning, such as Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN), have shown remarkable results in the field of speech enhancement. Among them, Convolutional Recurrent Network (CRN), which integrates the advantages of CNN and RNN, has been proven to be very effective in speech enhancement, and some models derived from CRN, such as DCCRN and FRCRN, have also shown strong competitiveness in the field of speech enhancement.

[0003] However, these models usually have large parameters and take a long time to train, which limits their practical application scenarios. For example, in areas such as real-time speech recognition that require instant response, long training times and high computational complexity of the models may result in the system being unable to meet the immediate needs of users. In addition, in resource-constrained environments, such as mobile devices or embedded systems, the storage and computing requirements of these models may exceed the carrying capacity of the device. Summary of the invention

[0004] The purpose of the present invention is to address the defects and shortcomings of the above-mentioned prior art and propose a single-channel speech enhancement method based on an improved convolutional recurrent network. By improving the classic convolutional recurrent network (CRN), while reducing the number of parameters and the amount of calculation of the model, the present invention ensures that its noise reduction performance is better than the traditional CRN model. A new lightweight convolutional recurrent network CRNv2 is proposed in the present invention. By optimizing the model structure and the loss function, a more efficient speech enhancement effect is achieved.

[0005] The technical solution adopted by the present invention to solve the technical problem is: a single-channel speech enhancement method based on an improved convolutional recurrent network, the method comprising the following steps:

[0006] Step 1: Download the dataset used for model training and testing, preprocess the speech of the dataset by using the VoiceBank+DEMAND dataset, and extract the amplitude spectrum of the speech signal by short-time Fourier transform (STFT);

[0007] Step 2: Input the amplitude spectrum into the encoder to extract high-dimensional features from the amplitude spectrum;

[0008] Step 3: Use the Channel-S4D block as the recursive module of the model to effectively model the temporal dependency of the speech signal. The output of the Channel-S4D block is input into the decoder and restored to the original dimension.

[0009] Step 4: Construct a joint loss function. In order to jointly constrain the difference between enhanced speech and clean speech in the time domain and frequency domain, a joint loss function of the weighted signal distortion ratio (Weighted-SDR) loss in the time domain and the mean square error (MSE) loss in the frequency domain is adopted. First, the weighted signal distortion ratio (Weighted-SDR) loss of the enhanced speech and clean speech in the time domain is calculated. Secondly, the mean square error (MSE) loss of the enhanced speech and clean speech in the frequency domain is calculated. The above two losses are weighted and summed. The constraints of the two are dynamically adjusted through the weight hyperparameters to form the final joint loss function. Then, the constraints of the joint loss function are used to update the model parameters through the back propagation algorithm to optimize the model.

[0010] Step 5: Reconstruct and evaluate the enhanced speech signal. The amplitude spectrum of the enhanced speech signal is obtained based on the trained model. Combined with the phase of the original signal, it is reconstructed into an enhanced speech signal in the time domain through the inverse short-time Fourier transform (ISTFT). The reconstructed enhanced speech signal is evaluated, mainly using the wideband speech perception quality (WB-PESQ) and short-time objective intelligibility (STOI) indicators to measure the quality and intelligibility of the enhanced speech. In the test phase, the optimal model is selected based on the results of these indicators to ensure that the final model can provide the best speech enhancement effect in practical applications.

[0011] Furthermore, the step 1 specifically includes:

[0012] Step 1-1: In the VoiceBank-DEMAND dataset used, the noisy part comes from the DEMAND dataset, while the clear speech part is taken from VoiceBank. The sampling frequency of its audio file is 48kHz. In order to construct the mixed speech data for training, 10 different types of noise are selected, including two artificial synthetic noises and 8 directly from the DEMAND dataset. The mixed speech is synthesized according to four signal-to-noise ratio levels (15dB, 10dB, 5dB, and 0dB respectively). When constructing the test set, the remaining 5 noises in DEMAND are selected, and combined with the recordings of two speakers in VoiceBank, the input data is created according to the signal-to-noise ratio levels different from the training set (17.5dB, 12.5dB, 7.5dB, and 2.5dB respectively);

[0013] Step 1-2: resample the speech signals in the data set to 16KHz, and unify the speech length to 3s. Specifically, truncate the speech longer than 3s, and fill the speech shorter than 3s with zeros at the end.

[0014] Step 1-3: Extract the amplitude spectrum features of the speech signal through short-time Fourier transform. Specifically, we set the window length to 25ms, the frame shift to 6.25ms, and the FFT point number to 400 to ensure that the spectral features of the signal can be captured efficiently.

[0015] Furthermore, the step 2 specifically includes:

[0016] Step 2-1: The encoder consists of 6 convolutional layers, and the output dimensions of each layer are 16, 32, 64, 128, 256, 256, respectively. Two-dimensional convolution is used, where the convolution kernel size is set to 3×2, the step size is (2,1), and the padding is (0,1). Each layer of the encoder uses the ELU activation function with BatchNorm2d regularization to optimize the training process and improve the generalization ability of the model;

[0017] Step 2-2: Each 2D convolutional layer is followed by BatchNorm2d regularization for normalization;

[0018] Step 2-3: After normalization, use the ELU activation function for nonlinear activation.

[0019] Furthermore, the step 3 specifically includes:

[0020] Step 3-1: Introduce the specially designed Channel-S4D block, which integrates residual connections, efficient channel attention layer (ECA layer), diagonal state space model layer, gated linear unit, one-dimensional convolution layer, dropout and channel normalization layer to enhance the stability of the model when processing long sequences. The ECA layer is mainly weighted according to the importance of the features, thereby enhancing the model's ability to capture key information. Channel normalization is used instead of the traditional layer normalization method, so that the model can be more effectively normalized when processing features of different channels, thereby improving training stability and model performance, helping the model to better improve the sensitivity of each channel feature, thereby improving the speech enhancement effect and the adaptability of the model. By introducing the Channel-S4D module, the model shows higher stability and performance when processing long sequence signals, improving the overall effect of speech enhancement and the performance of the model in parallel processing and modeling long-term time dependencies;

[0021] Step 3-2: The decoder module consists of 6 deconvolution layers. Each layer is connected to the corresponding encoder convolution layer through a skip connection to enhance the information flow, thereby obtaining multi-level and multi-scale features. The input dimensions of each convolution layer are 512, 512, 256, 128, 64, and 32, respectively. The convolution kernel size and step size are the same as those of the encoder. Each layer of the decoder is also equipped with BatchNorm2d regularization. In terms of activation function, the first 5 layers use the ELU activation function, and the activation function of the last layer is Softplus. The purpose is to ensure that the amplitude spectrum of the final output remains non-negative and promote high-quality speech reconstruction. This mixed activation function strategy is to balance the representation ability of the model and the output quality throughout the decoding process.

[0022] Further, the step 3-1 specifically includes:

[0023] Step 3-1-1: Define the hidden layer size of the diagonalized state space layer to be 256, the state space size to be 64, and adjust the tensor dimension of the encoder output to a shape that matches Channel-S4D;

[0024] Step 3-1-2: Before inputting the features into the diagonalized state space layer, the features are weighted according to their importance through the efficient channel attention layer, thereby enhancing the model's ability to capture key information;

[0025] Step 3-1-3: Use residual connections to combine input features with output features processed by the Channel-S4D block to improve training stability and model expressiveness;

[0026] Step 3-1-4: To further optimize feature processing, add a gated linear unit (GLU) to enhance nonlinear transformation capabilities and make the model better adaptable to complex speech signals;

[0027] Step 3-1-5: During feature processing, add a one-dimensional convolution layer to capture local features at different time steps, and use a dropout layer to prevent overfitting;

[0028] Step 3-1-6: In order to prevent the gradient from exploding, channel normalization is used instead of traditional layer normalization, so that the model can be more effectively normalized when processing features of different channels, thereby improving training stability and model performance.

[0029] Furthermore, the step 4 specifically includes:

[0030] Step 4-1: Construct a joint loss function. The loss function used in the model training process is a weighted combination of the mean square error (MSE) loss and the weighted signal-to-distortion ratio (wSDR) loss. The weight factor is set to 10 to ensure that the two losses have a balanced impact on model optimization during the training process.

[0031] Step 4-2: Calculate the mean square error (MSE) loss. The MSE loss is used to measure the difference between the enhanced speech and the clean speech in the frequency domain. By calculating the mean square error between the enhanced speech and the target speech signal, the frequency domain enhancement effect of the model is reflected. The frequency domain loss of the model is selected using the mean square error (MSE) loss function L mse Calculation, which is defined as follows:

[0032]

[0033] in, and Respectively represent the spectrum of the model-estimated speech signal and the spectrum of the clean speech signal;

[0034] Step 4-3: Calculate the weighted signal distortion ratio (wSDR) loss. The wSDR loss is used to measure the difference between enhanced speech and clean speech in the time domain. By calculating the weighted signal distortion ratio between the enhanced speech and the target speech signal, the time domain enhancement effect of the model is reflected. The time domain loss of the model is calculated by the weighted signal distortion ratio loss function. The weighted calculation is performed by calculating the proportion of noise and clean speech in order to calculate a more accurate loss. The definition is as follows:

[0035]

[0036] in, y and Denote the clean speech signal and the predicted speech signal, z and denote the noise in noisy speech and the noise in predicted speech respectively, α=||y|| 2 / (||x|| 2 +||y|| 2 ), represents the energy ratio of noise z and clean speech y;

[0037] Step 4-4: Weighted sum of MSE loss and wSDR loss to form the final joint loss function. The joint loss function can constrain the prediction results of the model in both the frequency domain and the time domain to ensure that the model can effectively improve the speech enhancement effect in both aspects. The model training adopts the joint loss function L joint , which is obtained by weighting the mean square error loss and the weighted signal distortion ratio loss, setting β as the weight factor, which is defined as:

[0038] L joint =L mse +βL wSDR

[0039] Among them, the hyperparameter weight factor β adopts the optimal value through multiple experiments to balance the constraints of the two losses.

[0040] Furthermore, the step 5 specifically includes:

[0041] Step 5-1: During the training process, the amplitude signal output by the model in each training round is combined with the original signal phase and reconstructed into an enhanced speech signal in the time domain through short-time Fourier inverse transform. Then, the loss of the enhanced speech signal in the training round is calculated to ensure that the loss decreases normally during the model training process.

[0042] Step 5-2: During the test, two indicators, wideband perceptual speech quality (WB-PESQ) and short-time objective intelligibility (STOI), are mainly used to evaluate the quality and intelligibility of the enhanced speech. The value range of WB-PESQ is -0.5 to 4.5, and the value range of STOI is 0 to 1. The larger the value of the two indicators, the higher the speech quality and intelligibility.

[0043] Step 5-3: After each training round, calculate the WB-PESQ and STOI values ​​of the enhanced speech signal, record the evaluation results, compare the training results of multiple rounds, monitor the changing trends of the indicators, and ensure the continuous improvement of model performance.

[0044] Step 5-4: Considering the randomness of deep learning, the model will be trained in multiple groups to find the optimal model parameters. According to the evaluation results of WB-PESQ and STOI, the model with the highest index value is selected as the optimal model. In this process, it is necessary to monitor the trend of the index change to ensure that the final saved model can provide the best speech enhancement effect in practical applications. The selection of the optimal model is based on its performance on the validation set to ensure that it has the best effect in practical applications.

[0045] Step 5-5: Save the optimal model for subsequent use and deployment. Through this systematic training and evaluation process, the final CRNv2 model is guaranteed to have superior performance in speech enhancement tasks.

[0046] Beneficial effects:

[0047] 1. The present invention develops a new CRNv2 model by improving the existing CRN (Convolutional Recurrent Network) model structure and loss function. The CRNv2 model adopts an encoder-decoder structure, which can make full use of the information in the input amplitude spectrum. By introducing the Channel-S4D block, it not only solves the problem of insufficient adaptability of the CRN model to complex noise environments in speech enhancement tasks, but also makes significant progress in modeling the temporal dependency of speech signals, so that when dealing with noise and speaker-independent speech enhancement, speech features can be more effectively extracted and retained, thereby significantly improving the speech enhancement effect.

[0048] 2. Compared with the CRN model, the CRNv2 model of the present invention significantly reduces the number of model parameters while improving performance. The number of parameters in the CRNv2 model is only 2.23M, which is only one-eighth of the number of parameters in the original CRN model. This significant reduction in parameters not only reduces the computing and storage costs, but also improves the practical application value of the model. Due to the substantial reduction in the number of parameters in the CRNv2 model, the required computing resources are significantly reduced, making the model more efficient during training and reasoning, and able to complete the same task in a shorter time, thereby improving overall computing efficiency. At the same time, the smaller number of parameters means that the size of the model file is also reduced. For devices with limited storage resources (such as embedded systems, mobile devices, etc.), the storage requirements of the CRNv2 model are greatly reduced, making it more suitable for application in these environments. The lower computing and storage costs make the CRNv2 model more advantageous in practical applications. Whether it is processed in the cloud or deployed on edge devices, the CRNv2 model can provide high-quality speech enhancement effects with lower resource consumption.

[0049] 3. The test results of the present invention on the VoiceBank-DEMAND dataset show that the CRNv2 model performs well in objective evaluation indicators. Specifically, the wideband objective speech quality evaluation (WB-PSEQ) score of the CRNv2 model is 2.76, which is 0.37 higher than the traditional CRN model. At the same time, the short-term objective intelligibility index (STOI) score of the CRNv2 model is 0.94, which is 0.02 higher than the CRN model. The improvement of these indicators proves that the CRNv2 model has better performance and higher robustness in speech enhancement tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a system block diagram of the overall model of the present invention.

[0051] Figure 2 Detailed diagram of the Channel-S4D module of the present invention.

[0052] Figure 3 Detailed diagram of the encoder module of the present invention.

[0053] Figure 4 Detailed diagram of the decoder module of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings of the specification. It should be noted that, in order to avoid blurring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solution of the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0055] In addition, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0056] like Figures 1 to 4 As shown, the present invention proposes a single-channel speech enhancement method based on an improved convolutional recurrent network, comprising the following steps:

[0057] Step 1: Download the dataset used for model training and testing, preprocess the speech of the dataset by using the VoiceBank+DEMAND dataset, and extract the amplitude spectrum of the speech signal by short-time Fourier transform (STFT);

[0058] Step 2: Input the amplitude spectrum into the encoder to extract high-dimensional features from the amplitude spectrum;

[0059] Step 3: Use the Channel-S4D block as the recursive module of the model to effectively model the temporal dependency of the speech signal. The output of the Channel-S4D block is input into the decoder and restored to the original dimension.

[0060] Step 4: Construct a joint loss function. In order to jointly constrain the difference between enhanced speech and clean speech in the time domain and frequency domain, a joint loss function of the weighted signal distortion ratio (Weighted-SDR) loss in the time domain and the mean square error (MSE) loss in the frequency domain is adopted. First, the weighted signal distortion ratio (Weighted-SDR) loss of the enhanced speech and clean speech in the time domain is calculated. Secondly, the mean square error (MSE) loss of the enhanced speech and clean speech in the frequency domain is calculated. The above two losses are weighted and summed. The constraints of the two are dynamically adjusted through the weight hyperparameters to form the final joint loss function. Then, the constraints of the joint loss function are used to update the model parameters through the back propagation algorithm to optimize the model.

[0061] Step 5: Reconstruct and evaluate the enhanced speech signal. The amplitude spectrum of the enhanced speech signal is obtained based on the trained model. Combined with the phase of the original signal, it is reconstructed into an enhanced speech signal in the time domain through the inverse short-time Fourier transform (ISTFT). The reconstructed enhanced speech signal is evaluated, mainly using the wideband speech perception quality (WB-PESQ) and short-time objective intelligibility (STOI) indicators to measure the quality and intelligibility of the enhanced speech. In the test phase, the optimal model is selected based on the results of these indicators to ensure that the final model can provide the best speech enhancement effect in practical applications.

[0062] The above step 1 of the present invention specifically comprises:

[0063] Step 1-1: In the VoiceBank-DEMAND dataset used, the noisy part comes from the DEMAND dataset, while the clear speech part is taken from VoiceBank. The sampling frequency of its audio file is 48kHz. In order to construct the mixed speech data for training, 10 different types of noise are selected, including two artificial synthetic noises and 8 directly from the DEMAND dataset. The mixed speech is synthesized according to four signal-to-noise ratio levels (15dB, 10dB, 5dB, and 0dB respectively). When constructing the test set, the remaining 5 noises in DEMAND are selected, and combined with the recordings of two speakers in VoiceBank, the input data is created according to the signal-to-noise ratio levels different from the training set (17.5dB, 12.5dB, 7.5dB, and 2.5dB respectively);

[0064] Step 1-2: resample the speech signals in the data set to 16KHz, and unify the speech length to 3s. Specifically, truncate the speech longer than 3s, and fill the speech shorter than 3s with zeros at the end.

[0065] Step 1-3: Extract the amplitude spectrum features of the speech signal through short-time Fourier transform. Specifically, we set the window length to 25ms, the frame shift to 6.25ms, and the FFT point number to 400 to ensure that the spectral features of the signal can be captured efficiently.

[0066] The above step 2 of the present invention specifically includes:

[0067] Step 2-1: The encoder consists of 6 convolutional layers, and the output dimensions of each layer are 16, 32, 64, 128, 256, 256, respectively. Two-dimensional convolution is used, where the convolution kernel size is set to 3×2, the step size is (2,1), and the padding is (0,1). Each layer of the encoder uses the ELU activation function with BatchNorm2d regularization to optimize the training process and improve the generalization ability of the model;

[0068] Step 2-2: Each 2D convolutional layer is followed by BatchNorm2d regularization for normalization;

[0069] Step 2-3: After normalization, use the ELU activation function for nonlinear activation.

[0070] The above step 3 of the present invention specifically includes:

[0071] Step 3-1: Introduce the specially designed Channel-S4D block, which integrates residual connections, efficient channel attention layer (ECA layer), diagonal state space model layer, gated linear unit, one-dimensional convolution layer, dropout and channel normalization layer to enhance the stability of the model when processing long sequences. The ECA layer is mainly weighted according to the importance of the features, thereby enhancing the model's ability to capture key information. Channel normalization is used instead of the traditional layer normalization method, so that the model can be more effectively normalized when processing features of different channels, thereby improving training stability and model performance, helping the model to better improve the sensitivity of each channel feature, thereby improving the speech enhancement effect and the adaptability of the model. By introducing the Channel-S4D module, the model shows higher stability and performance when processing long sequence signals, improving the overall effect of speech enhancement and the performance of the model in parallel processing and modeling long-term time dependencies;

[0072] Step 3-2: The decoder module consists of 6 deconvolution layers. Each layer is connected to the corresponding encoder convolution layer through a skip connection to enhance the information flow, thereby obtaining multi-level and multi-scale features. The input dimensions of each convolution layer are 512, 512, 256, 128, 64, and 32, respectively. The convolution kernel size and step size are the same as those of the encoder. Each layer of the decoder is also equipped with BatchNorm2d regularization. In terms of activation function, the first 5 layers use the ELU activation function, and the activation function of the last layer is Softplus. The purpose is to ensure that the amplitude spectrum of the final output remains non-negative and promote high-quality speech reconstruction. This mixed activation function strategy is to balance the representation ability of the model and the output quality throughout the decoding process.

[0073] The above step 3-1 of the present invention specifically includes:

[0074] Step 3-1-1: Define the hidden layer size of the diagonalized state space layer to be 256, the state space size to be 64, and adjust the tensor dimension of the encoder output to a shape that matches Channel-S4D;

[0075] Step 3-1-2: Before inputting the features into the diagonalized state space layer, the features are weighted according to their importance through the efficient channel attention layer, thereby enhancing the model's ability to capture key information;

[0076] Step 3-1-3: Use residual connections to combine input features with output features processed by the Channel-S4D block to improve training stability and model expressiveness;

[0077] Step 3-1-4: To further optimize feature processing, add a gated linear unit (GLU) to enhance nonlinear transformation capabilities and make the model better adaptable to complex speech signals;

[0078] Step 3-1-5: During feature processing, add a one-dimensional convolution layer to capture local features at different time steps, and use a dropout layer to prevent overfitting;

[0079] Step 3-1-6: In order to prevent the gradient from exploding, channel normalization is used instead of traditional layer normalization, so that the model can be more effectively normalized when processing features of different channels, thereby improving training stability and model performance.

[0080] The above step 4 of the present invention specifically includes:

[0081] Step 4-1: Construct a joint loss function. The loss function used in the model training process is a weighted combination of the mean square error (MSE) loss and the weighted signal-to-distortion ratio (wSDR) loss. The weight factor is set to 10 to ensure that the two losses have a balanced impact on model optimization during the training process.

[0082] Step 4-2: Calculate the mean square error (MSE) loss. The MSE loss is used to measure the difference between the enhanced speech and the clean speech in the frequency domain. By calculating the mean square error between the enhanced speech and the target speech signal, the frequency domain enhancement effect of the model is reflected. The frequency domain loss of the model is selected using the mean square error (MSE) loss function L mse Calculation, which is defined as follows:

[0083]

[0084] in, and Respectively represent the spectrum of the model-estimated speech signal and the spectrum of the clean speech signal;

[0085] Step 4-3: Calculate the weighted signal distortion ratio (wSDR) loss. The wSDR loss is used to measure the difference between enhanced speech and clean speech in the time domain. By calculating the weighted signal distortion ratio between the enhanced speech and the target speech signal, the time domain enhancement effect of the model is reflected. The time domain loss of the model is calculated by the weighted signal distortion ratio loss function. The weighted calculation is performed by calculating the proportion of noise and clean speech in order to calculate a more accurate loss. The definition is as follows:

[0086]

[0087] in, y and Denote the clean speech signal and the predicted speech signal, z and denote the noise in noisy speech and the noise in predicted speech respectively, α=||y|| 2 / (||x|| 2 +||y|| 2 ), represents the energy ratio of noise z and clean speech y;

[0088] Step 4-4: Weighted sum of MSE loss and wSDR loss to form the final joint loss function. The joint loss function can constrain the prediction results of the model in both the frequency domain and the time domain to ensure that the model can effectively improve the speech enhancement effect in both aspects. The model training adopts the joint loss function L joint , which is obtained by weighting the mean square error loss and the weighted signal distortion ratio loss, setting β as the weight factor, which is defined as:

[0089] L joint =Lmse +βL wSDR

[0090] Among them, the hyperparameter weight factor β adopts the optimal value through multiple experiments to balance the constraints of the two losses.

[0091] The above step 5 of the present invention specifically includes:

[0092] Step 5-1: During the training process, the amplitude signal output by the model in each training round is combined with the original signal phase and reconstructed into an enhanced speech signal in the time domain through short-time Fourier inverse transform. Then, the loss of the enhanced speech signal in the training round is calculated to ensure that the loss decreases normally during the model training process.

[0093] Step 5-2: During the test, two indicators, wideband perceptual speech quality (WB-PESQ) and short-time objective intelligibility (STOI), are mainly used to evaluate the quality and intelligibility of the enhanced speech. The value range of WB-PESQ is -0.5 to 4.5, and the value range of STOI is 0 to 1. The larger the value of the two indicators, the higher the speech quality and intelligibility.

[0094] Step 5-3: After each training round, calculate the WB-PESQ and STOI values ​​of the enhanced speech signal, record the evaluation results, compare the training results of multiple rounds, monitor the changing trends of the indicators, and ensure the continuous improvement of model performance.

[0095] Step 5-4: Considering the randomness of deep learning, the model will be trained in multiple groups to find the optimal model parameters. According to the evaluation results of WB-PESQ and STOI, the model with the highest index value is selected as the optimal model. In this process, it is necessary to monitor the trend of the index change to ensure that the final saved model can provide the best speech enhancement effect in practical applications. The selection of the optimal model is based on its performance on the validation set to ensure that it has the best effect in practical applications.

[0096] Step 5-5: Save the optimal model for subsequent use and deployment. Through this systematic training and evaluation process, the final CRNv2 model is guaranteed to have superior performance in speech enhancement tasks.

[0097] The simulation experiment and data of the present invention are as follows:

[0098] The complete process is implemented under the pytorch framework, and the required Python libraries include but are not limited to: Numpy, Librosa, einops, Soundfile, math, etc.

[0099] Use Librosa to load the speech file and perform STFT transform. The FFT window size n_fft is 400, the window length win_len is 400, and the frame shift hop_len is 100.

[0100] The encoder consists of 6 convolutional layers, and the input and output dimensions of each layer are (1,16), (16,32), (32,64), (64,128), (128,256), and (256,256), respectively. It uses two-dimensional convolution, the convolution kernel size is 3×2, the step size is (2,1), the padding is (0,1), the regularization uses BatchNorm2d, and the activation function is ELU.

[0101] The sequence modeling uses the Channel-S4D block, which integrates residual connections, efficient channel attention layers (ECA layers), diagonalized state space model layers, gated linear units, one-dimensional convolutional layers, dropout, and channel normalization layers to enhance the stability of the model when processing long sequences. The input and output dimensions of the Channel-S4D block are 256 dimensions.

[0102] The decoder module consists of 6 deconvolution layers, and the input and output dimensions of each layer are (512,512), (512,256), (256,128), (128,64), (64,32), (32,1). Each layer is connected to the corresponding encoder convolution layer output through a skip connection to enhance the information flow. The convolution kernel size is 3×2, the step size is (2,1), the padding is (0,1), the activation function of the first 5 convolution layers is ELU, and the activation function of the last convolution layer is Softplus.

[0103] During model training, the batch size is 2. The SGD optimizer is used, the learning rate lr is set to 1e-3, and the loss function Loss adopts a joint loss function, which is obtained by weighting the mean square error (MSE) loss and the weighted signal-to-distortion ratio (wSDR) loss, with a weight factor of 10.

[0104] The number of training rounds is 100 epochs. During the training process, the evaluation indicators are wideband speech perception quality (WB-PESQ) and short-term objective intelligibility (STOI). The value range of WB-PESQ is -0.5 to 4.5, and the value range of STOI is 0 to 1. The best model is selected by comparing the indicators. The WB-PESQ score of the CRNv2 model on the VoiceBank-DEMAND dataset is 2.80, and the STOI score is 0.94, which is significantly better than the traditional CRN model.

[0105] The comparison results of the CRNv2 model of the present invention and other speech enhancement models on the VoiceBank-DEMAND dataset are shown in Table 1:

[0106] Table 1 Performance comparison of CRNv2 and other speech enhancement methods

[0107]

[0108]

[0109] Among them, CRNv2 is a new model proposed by the present invention. The results in the table show that CRNv2 is higher than the existing model in both wideband speech perception quality (WB-PESQ) and short-time objective intelligibility (STOI) with a lower number of model parameters. This proves that the improved model can obtain enhanced speech with high subjective and objective quality from noisy speech, and has a good learning and reference significance in practical applications.

[0110] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A single-channel speech enhancement method based on an improved convolutional recurrent network, characterized in that: The method comprises the following steps: Step 1: Download and preprocess the dataset used for model training and testing, and perform short-time Fourier transform (STFT) on the speech in the dataset to extract the amplitude spectrum of the speech signal; Step 2: Input the amplitude spectrum feature into the encoder to extract high-dimensional features from the amplitude spectrum; Step 3: Use the constructed Channel-S4D block as a recursive module to model the temporal dependency of the speech signal. The output of the Channel-S4D block is input into the decoder and restored to the same dimension as the original input. Step 4: Construct a joint loss function. In order to jointly constrain the difference between enhanced speech and clean speech in the time domain and frequency domain, a loss function that jointly constrains the time domain weighted signal distortion ratio loss and the frequency domain mean square error loss is adopted. First, the weighted signal distortion ratio loss of enhanced speech and clean speech in the time domain is calculated. Secondly, the mean square error loss of enhanced speech and clean speech in the frequency domain is calculated. The two losses are weighted and summed. The constraints of the two are dynamically adjusted through the weight hyperparameters to form the final joint loss function. Using the constraints of the joint loss function, the model parameters are updated through the back propagation algorithm to optimize the model. Step 5: Reconstruct and evaluate the enhanced speech signal. The amplitude spectrum of the enhanced speech signal is obtained based on the trained model. Combined with the phase of the original signal, it is reconstructed into an enhanced speech signal in the time domain through short-time Fourier inverse transform. The reconstructed enhanced speech signal is evaluated. The quality and intelligibility of the enhanced speech are measured by using broadband speech perception quality and short-time objective intelligibility indicators. The optimal model is selected according to the results of these indicators during the testing phase to ensure that the final model can provide the best speech enhancement effect in practical applications.

2. A single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that: The step 1 specifically includes: Step 1-1: In the VoiceBank+DEMAND dataset used, the noisy part comes from the DEMAND dataset, and the clear speech part is taken from VoiceBank. The sampling frequency of the audio file is 48kHz. The training set contains 10 different types of noise, which are mixed according to four signal-to-noise ratio levels (15dB, 10dB, 5dB, and 0dB). The test set selects the remaining 5 types of noise in DEMAND, combined with the recordings of two speakers in VoiceBank, and synthesized according to different signal-to-noise ratio levels from the training set (17.5dB, 12.5dB, 7.5dB, and 2.5dB); Step 1-2: resample the speech signals in the data set to 16KHz, unify the speech length to 3 seconds, truncate the speech longer than 3 seconds, and fill the end of the speech shorter than 3 seconds with zeros; Step 1-3: Extract the amplitude spectrum features of the speech signal through short-time Fourier transform, with a window length of 25 ms, a frame shift of 6.25 ms, and a FFT point number of 400.

3. The single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that: The step 2 specifically includes: Step 2-1: The encoder consists of 6 layers of 2D convolutional layers. The output dimensions of each layer are 16, 32, 64, 128, 256, and 256, respectively. The size of the convolution kernel is set to 3×2, the stride is (2,1), and the padding is (0,1). Step 2-2: Each 2D convolutional layer is followed by BatchNorm2d regularization for normalization; Step 2-3: After normalization, use the ELU activation function for nonlinear activation.

4. The single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that: The step 3 specifically includes: Step 3-1: Construct the Channel-S4D module by integrating residual connection, efficient channel attention layer, diagonal state space model layer, gated linear unit, one-dimensional convolution layer, dropout layer and channel normalization layer to obtain key speech information at the channel level, so that the model can more accurately capture the temporal dependency of speech signals; Step 3-2: The decoder module consists of 6 deconvolution layers. Each layer is connected to the output of the corresponding encoder convolution layer through skip connections to enhance the information flow. The input dimensions of the convolution layer are 512, 512, 256, 128, 64, and 32 respectively. The convolution kernel size and step size are set the same as those of the encoder. The first 5 layers of the decoder use the ELU activation function, and the last layer uses the Softplus activation function.

5. A single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 4, characterized in that: The step 3-1 specifically includes: Step 3-1-1: Define the hidden layer size of the diagonalized state space layer to be 256, the state space size to be 64, and adjust the tensor dimension of the encoder output to a shape that matches Channel-S4D; Step 3-1-2: Before inputting the features into the diagonalized state space layer, the features are weighted according to their importance through an efficient channel attention layer, thereby enhancing the model's ability to capture key information; Step 3-1-3: Use residual connections to combine input features with output features processed by the Channel-S4D block to improve training stability and model expressiveness; Step 3-1-4: To optimize feature processing, add gated linear units to enhance nonlinear transformation capabilities and make the model better adaptable to complex speech signals; Step 3-1-5: During feature processing, add a one-dimensional convolution layer to capture local features at different time steps, and use a dropout layer to prevent overfitting; Step 3-1-6: In order to prevent the gradient from exploding, channel normalization is used instead of traditional layer normalization, so that the model can be more effectively normalized when processing features of different channels, thereby improving training stability and model performance.

6. The single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that: The step 4 specifically includes: Step 4-1: The frequency domain loss of the model is selected using the mean square error (MSE) loss function L mse Calculation, which is defined as follows: in, and Respectively represent the spectrum of the model-estimated speech signal and the spectrum of the clean speech signal; Step 4-2: The time domain loss of the model is calculated by selecting the weighted signal distortion ratio loss function, which is weighted by calculating the proportion of noise and clean speech to calculate a more accurate loss. The definition is as follows: in, y and Denote the clean speech signal and the predicted speech signal, z and denote the noise in noisy speech and the noise in predicted speech respectively, α=||y|| 2 / (||x|| 2 +||y|| 2 ), represents the energy ratio of noise z and clean speech y; Step 4-3: Model training uses the joint loss function L joint , which is obtained by weighting the mean square error loss and the weighted signal distortion ratio loss, setting β as the weight factor, which is defined as: L joint =L mse +βL wSDR Among them, the hyperparameter weight factor β adopts the optimal value through multiple experiments to balance the constraints of the two losses.

7. The single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that: The step 5 specifically includes: Step 5-1: During the training process, the amplitude signal output by the model in each training round is combined with the original signal phase and reconstructed into an enhanced speech signal in the time domain through short-time Fourier inverse transform. Then, the loss of the enhanced speech signal in the training round is calculated to ensure that the loss decreases normally during the model training process. Step 5-2: During the test, the quality and intelligibility of the enhanced speech are measured using two indicators: broadband speech perception quality and short-term objective intelligibility. Step 5-3: After each training round, calculate the WB-PESQ and STOI index values ​​of the model trained in this round on the test set, record and compare the evaluation results of each round to ensure that the model performance is continuously improved; Step 5-4: Considering the randomness of deep learning, the model will be trained in multiple groups to find the optimal model parameters. According to the evaluation results of WB-PESQ and STOI, the model with the highest index value is selected as the optimal model. During this process, it is necessary to monitor the trend of the index to ensure that the final saved model can provide the best speech enhancement effect in practical applications; Step 5-5: Save the optimal model for subsequent use and deployment. Through this systematic training and evaluation process, the final CRNv2 model is guaranteed to have superior performance in speech enhancement tasks.

Images