A Single-Channel Speech Enhancement Method Based on Improved Convolutional Recurrent Networks

Abstract

This invention provides a single-channel speech enhancement method based on an improved convolutional recurrent network, comprising: Step 1: Downloading the dataset used for model training and testing, preprocessing the speech in the dataset using the VoiceBank+DEMAND dataset, and extracting the amplitude spectrum of the speech signal using Short Time Fourier Transform (STFT); Step 2: Inputting the amplitude spectrum into the encoder to extract high-dimensional features from the amplitude spectrum; Step 3: Using the Channel-S4D block as the recursive module of the model, the output of the Channel-S4D block is input into the decoder and restored to the original dimension; Step 4: Constructing a joint loss function; Step 5: Reconstructing and evaluating the enhanced speech signal. This invention utilizes the Channel-S4D module constructed based on a diagonalized state-space model to efficiently process speech sequences, capture key speech information at the channel level, better model the temporal dependencies of speech signals, and improve the model's adaptability to complex noisy environments while reducing the number of model parameters and computational cost.

Description

Technical Field

[0001] This invention relates to a single-channel speech enhancement method based on an improved convolutional recurrent network, specifically to a novel, efficient, and lightweight convolutional recurrent network (CRNv2) for single-channel speech enhancement, belonging to the field of speech enhancement technology. Background Technology

[0002] In modern society, clear speech is the foundation of effective communication. However, adverse factors such as background noise and reverberation often impair the intelligibility and quality of speech signals, affecting the quality of our daily voice communication. Therefore, researching speech enhancement (SE) techniques to address noise interference is crucial. Speech enhancement techniques aim to improve the quality and clarity of speech signals, making them easier to understand and recognize. In recent years, with the development of deep learning technology, deep learning-based speech enhancement methods have gradually become a research hotspot. Deep learning, through training on large-scale data, can automatically learn the features and patterns in speech signals, enabling models to more accurately analyze and reconstruct damaged speech signals. Some common deep learning models, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), have shown significant effects in the field of speech enhancement. Among them, Convolutional Recurrent Networks (CRNs), which integrate the advantages of CNNs and RNNs, have proven to be very effective in speech enhancement, and some models derived from CRNs, such as DCCRN and FRCRN, have also shown strong competitiveness in the field of speech enhancement.

[0003] However, these models typically have a large number of parameters and require a long training time, which limits their application scenarios in practice. For example, in fields such as real-time speech recognition that require instant response, the long training time and high computational complexity of the models may prevent the system from meeting the user's immediate needs. In addition, in resource-constrained environments, such as mobile devices or embedded systems, the storage and computational requirements of these models may exceed the device's capacity. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings and deficiencies of the existing technology by proposing a single-channel speech enhancement method based on an improved convolutional recurrent network. By improving the classic convolutional recurrent network (CRN), the invention reduces the number of model parameters and computational cost while ensuring that its noise reduction performance is superior to that of the traditional CRN model. This invention proposes a novel lightweight convolutional recurrent network CRNv2, which achieves more efficient speech enhancement by optimizing the model structure and loss function.

[0005] The technical solution adopted by this invention to solve its technical problem is: a single-channel speech enhancement method based on an improved convolutional recurrent network, the method comprising the following steps:

[0006] Step 1: Download the dataset used for model training and testing. Preprocess the speech in the dataset using the VoiceBank+DEMAND dataset and extract the amplitude spectrum of the speech signal using Short Time Fourier Transform (STFT).

[0007] Step 2: Input the amplitude spectrum into the encoder and extract high-dimensional features from the amplitude spectrum;

[0008] Step 3: Use the Channel-S4D block as the recursive module of the model to effectively model the temporal dependencies of the speech signal. The output of the Channel-S4D block will be input into the decoder and restored to the original dimension.

[0009] Step 4: Construct a joint loss function. To jointly constrain the difference between enhanced and clean speech in the time and frequency domains, a joint loss function is adopted, consisting of time-domain weighted signal distortion ratio (Weighted-SDR) loss and frequency-domain mean square error (MSE) loss. First, the weighted signal distortion ratio (Weighted-SDR) loss for enhanced and clean speech in the time domain is calculated. Second, the mean square error (MSE) loss for enhanced and clean speech in the frequency domain is calculated. The two losses are then weighted and summed. The constraint force of the two losses is dynamically adjusted through the weight hyperparameters to form the final joint loss function. Then, the model parameters are updated using the constraints of this joint loss function through the backpropagation algorithm, thereby optimizing the model.

[0010] Step 5: Reconstruct and evaluate the enhanced speech signal. Based on the trained model, the amplitude spectrum of the enhanced speech signal is obtained. Combined with the phase of the original signal, the enhanced speech signal in the time domain is reconstructed through Inverse Short-Time Fourier Transform (ISTFT). The reconstructed enhanced speech signal is evaluated, mainly using Wideband Perceptual Speech Quality (WB-PESQ) and Short-Time Objective Intelligibility (STOI) metrics to measure the quality and intelligibility of the enhanced speech. During the testing phase, the optimal model is selected based on the results of these metrics to ensure that the final model can provide the best speech enhancement effect in practical applications.

[0011] Furthermore, step 1 specifically includes:

[0012] Step 1-1: In the VoiceBank-DEMAND dataset used, the noisy parts are from the DEMAND dataset, while the clear speech parts are from VoiceBank. The audio files are sampled at a frequency of 48kHz. To construct the mixed speech data for training, 10 different types of noise were selected—including two artificial synthetic noises and eight directly from the DEMAND dataset. The mixed speech was synthesized according to four signal-to-noise ratio levels (15dB, 10dB, 5dB, and 0dB). When constructing the test set, the remaining five noises from DEMAND were selected and combined with recordings of two speakers from VoiceBank. Input data was created based on different signal-to-noise ratio levels than the training set (17.5dB, 12.5dB, 7.5dB, and 2.5dB).

[0013] Steps 1-2: Resample the speech signals in the dataset to 16kHz, and unify the speech length to 3s. Specifically, truncate speech longer than 3s and pad speech shorter than 3s with zeros at the end.

[0014] Steps 1-3: Extract the amplitude spectrum features of the speech signal through short-time Fourier transform. Specifically, we set the window length to 25ms, the frame shift to 6.25ms, and the FFT points to 400 to ensure that the spectral features of the signal can be captured efficiently.

[0015] Furthermore, step 2 specifically includes:

[0016] Step 2-1: The encoder consists of 6 convolutional layers, with output dimensions of 16, 32, 64, 128, 256, and 256 for each layer, respectively. All layers use two-dimensional convolution, with the kernel size set to 3×2, a stride of (2,1), and padding of (0,1). Each layer of the encoder uses an ELU activation function with BatchNorm2d regularization to optimize the training process and improve the model's generalization ability.

[0017] Step 2-2: Each 2D convolutional layer is followed by BatchNorm2d regularization for normalization;

[0018] Steps 2-3: After normalization, use the ELU activation function for nonlinear activation.

[0019] Furthermore, step 3 specifically includes:

[0020] Step 3-1: Introduce the specially designed Channel-S4D block, which enhances the stability of the model when processing long sequences by integrating residual connections, efficient channel attention layer (ECA layer), diagonalized state space model layer, gated linear unit, one-dimensional convolutional layer, dropout, and channel normalization layer. The ECA layer mainly weights features based on their importance, thereby enhancing the model's ability to capture key information. Channel normalization is used instead of the traditional layer normalization method, enabling the model to normalize features of different channels more effectively, thereby improving training stability and model performance. This helps the model to better improve the sensitivity of features of each channel, thus improving the speech enhancement effect and the model's adaptability. By introducing the Channel-S4D module, the model exhibits higher stability and performance when processing long sequence signals, improving the overall effect of speech enhancement as well as the model's performance in parallel processing and modeling long-term time dependencies.

[0021] Step 3-2: The decoder module consists of 6 deconvolutional layers. Each layer and the output of the corresponding encoder convolutional layer are enhanced through skip connections to obtain multi-level and multi-scale features. The input dimensions of each convolutional layer are 512, 512, 256, 128, 64, and 32, respectively. The kernel size and stride are the same as those of the encoder. Each layer of the decoder is also equipped with BatchNorm2d regularization. In terms of activation functions, the first 5 layers use the ELU activation function, and the last layer uses the Softplus activation function. The purpose is to ensure that the amplitude spectrum of the final output remains non-negative and promote high-quality speech reconstruction. This strategy of using hybrid activation functions is to balance the representational power of the model and the quality of the output throughout the decoding process.

[0022] Furthermore, step 3-1 specifically includes:

[0023] Step 3-1-1: Define the size of the hidden layer of the diagonalized state space layer as 256, the size of the state space as 64, and adjust the tensor dimension of the encoder output to match the shape of Channel-S4D;

[0024] Step 3-1-2: Before inputting the features into the diagonalized state space layer, the model is weighted according to the importance of the features through an efficient channel attention layer, thereby enhancing the model's ability to capture key information.

[0025] Step 3-1-3: Use residual connections to combine the input features with the output features processed by the Channel-S4D block to improve the stability of training and the expressive power of the model;

[0026] Step 3-1-4: To further optimize feature processing, gated linear units (GLUs) are added to enhance nonlinear transformation capabilities and enable the model to better adapt to complex speech signals;

[0027] Step 3-1-5: During feature processing, a one-dimensional convolutional layer is added to capture local features at different time steps, and a dropout layer is used to prevent overfitting.

[0028] Step 3-1-6: To prevent gradient explosion, channel normalization is used instead of traditional layer normalization, so that the model can normalize more effectively when processing features of different channels, thereby improving training stability and model performance.

[0029] Furthermore, step 4 specifically includes:

[0030] Step 4-1: Construct a joint loss function. The loss function used during model training is obtained by weighting the mean squared error (MSE) loss and the weighted signal distortion ratio (wSDR) loss. The weight factor is set to 10 to ensure that the two losses have a balanced impact on model optimization during training.

[0031] Step 4-2: Calculate the mean squared error (MSE) loss. The MSE loss measures the difference between enhanced and clean speech in the frequency domain. By calculating the mean squared error between the enhanced and target speech signals, it reflects the frequency domain enhancement effect of the model. The frequency domain loss of the model is selected using the mean squared error (MSE) loss function L. mse Calculation, which is defined as follows:

[0032]

[0033] in, and These represent the spectrum of the estimated speech signal and the spectrum of the clean speech signal, respectively.

[0034] Step 4-3: Calculate the weighted signal distortion ratio (wSDR) loss. The wSDR loss measures the temporal difference between enhanced and clean speech. By calculating the weighted signal distortion ratio between the enhanced and target speech signals, it reflects the temporal enhancement effect of the model. The temporal loss of the model is calculated using the weighted signal distortion ratio loss function, which is weighted by calculating the proportion of noise and clean speech to obtain a more accurate loss. Its definition is as follows:

[0035]

[0036] in, y and Let z and z represent the clean speech signal and the predicted speech signal, respectively. Let α = ||y|| represent the noise in the noisy speech and the noise in the predicted speech, respectively. 2 / (||x|| 2 +||y|| 2 ), representing the energy ratio of noise z to clean speech y;

[0037] Step 4-4: The MSE loss and wSDR loss are weighted and summed to form the final joint loss function. The joint loss function constrains the model's prediction results in both the frequency and time domains, ensuring the model effectively improves speech enhancement in both aspects. The model training uses the joint loss function L... joint It is obtained by weighting the mean square error loss and the weighted signal distortion ratio loss, and β is set as the weighting factor, which is defined as:

[0038] L joint =L mse +βL wSDR

[0039] Among them, the hyperparameter weighting factor β is optimized through multiple experiments to balance the constraints of these two types of losses.

[0040] Furthermore, step 5 specifically includes:

[0041] Step 5-1: During the training process, the amplitude signal output by the model in each training round is combined with the phase of the original signal and reconstructed into an enhanced speech signal in the time domain through short-time Fourier transform. Then, the loss of the enhanced speech signal in the training round is calculated to ensure that the loss decreases normally during the model training process.

[0042] Step 5-2: During the testing process, two indicators are mainly used to evaluate the quality and intelligibility of enhanced speech: Wideband Perceptual Speech Quality (WB-PESQ) and Short-Time Objective Intelligibility (STOI). The value of WB-PESQ ranges from -0.5 to 4.5, and the value of STOI ranges from 0 to 1. The larger the value of the two indicators, the higher the speech quality and intelligibility.

[0043] Step 5-3: After each training round, calculate the WB-PESQ and STOI values ​​of the enhanced speech signal, record the evaluation results, and monitor the trend of the indicators by comparing the training results of multiple rounds to ensure the continuous improvement of model performance.

[0044] Step 5-4: Considering the randomness of deep learning, the model will be trained multiple times to find the optimal model parameters. Based on the evaluation results of WB-PESQ and STOI, the model with the highest index value will be selected as the optimal model. During this process, it is necessary to monitor the trend of index changes to ensure that the final saved model can provide the best speech enhancement effect in practical applications. The selection of the optimal model is based on its performance on the validation set to ensure that it performs best in practical applications.

[0045] Step 5-5: Save the optimal model for subsequent use and deployment. This systematic training and evaluation process ensures that the final CRNv2 model has superior performance in speech enhancement tasks.

[0046] Beneficial effects:

[0047] 1. This invention develops a novel CRNv2 model by improving the existing CRN (Convolutional Recurrent Network) model structure and loss function. The CRNv2 model employs an encoder-decoder structure, which can fully utilize information from the input amplitude spectrum. By introducing the Channel-S4D block, it not only solves the problem of insufficient adaptability of the CRN model to complex noise environments in speech enhancement tasks, but also achieves significant progress in modeling the temporal dependencies of speech signals. This enables more effective extraction and preservation of speech features when processing speech enhancement that is noise- and speaker-independent, thereby significantly improving the speech enhancement effect.

[0048] 2. Compared to the CRN model, the CRNv2 model of this invention significantly reduces the number of parameters while improving performance. The CRNv2 model has only 2.23M parameters, which is only one-eighth of the original CRN model's parameters. This significant reduction in parameters not only lowers computational and storage costs but also enhances the model's practical application value. Due to the substantial reduction in the number of parameters in the CRNv2 model, the required computational resources are significantly reduced, making the model more efficient in training and inference, enabling it to complete the same task in a shorter time and improving overall computational efficiency. Simultaneously, the smaller number of parameters means a smaller model file size. For devices with limited storage resources (such as embedded systems and mobile devices), the storage requirements of the CRNv2 model are significantly reduced, making it more suitable for application in these environments. The lower computational and storage costs give the CRNv2 model a greater advantage in practical applications. Whether processed in the cloud or deployed on edge devices, the CRNv2 model can provide high-quality speech enhancement effects with lower resource consumption.

[0049] 3. Test results on the VoiceBank-DEMAND dataset demonstrate that the CRNv2 model performs excellently on objective evaluation metrics. Specifically, the CRNv2 model achieves a Wideband Objective Speech Quality Assessment (WB-PSEQ) score of 2.76, an improvement of 0.37 compared to the traditional CRN model. Simultaneously, the CRNv2 model achieves a Short-Time Objective Intelligibility Index (STOI) score of 0.94, an improvement of 0.02 compared to the traditional CRN model. These improvements demonstrate that the CRNv2 model possesses better performance and greater robustness in speech enhancement tasks. Attached Figure Description

[0050] Figure 1 This is a block diagram of the overall model system of the present invention.

[0051] Figure 2 This is a detailed diagram of the Channel-S4D module of the present invention.

[0052] Figure 3 This is a detailed diagram of the encoder module of the present invention.

[0053] Figure 4 This is a detailed diagram of the decoder module of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be noted that, to avoid obscuring the invention with unnecessary details, only structures and / or processing steps closely related to the invention are shown in the drawings, while other details not directly related to the invention are omitted.

[0055] Additionally, it should be noted that the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0056] like Figures 1 to 4 As shown, this invention proposes a single-channel speech enhancement method based on an improved convolutional recurrent network, comprising the following steps:

[0057] Step 1: Download the dataset used for model training and testing. Preprocess the speech in the dataset using the VoiceBank+DEMAND dataset and extract the amplitude spectrum of the speech signal using Short Time Fourier Transform (STFT).

[0058] Step 2: Input the amplitude spectrum into the encoder and extract high-dimensional features from the amplitude spectrum;

[0059] Step 3: Use the Channel-S4D block as the recursive module of the model to effectively model the temporal dependencies of the speech signal. The output of the Channel-S4D block will be input into the decoder and restored to the original dimension.

[0060] Step 4: Construct a joint loss function. To jointly constrain the difference between enhanced and clean speech in the time and frequency domains, a joint loss function is adopted, consisting of time-domain weighted signal distortion ratio (Weighted-SDR) loss and frequency-domain mean square error (MSE) loss. First, the weighted signal distortion ratio (Weighted-SDR) loss for enhanced and clean speech in the time domain is calculated. Second, the mean square error (MSE) loss for enhanced and clean speech in the frequency domain is calculated. The two losses are then weighted and summed. The constraint force of the two losses is dynamically adjusted through the weight hyperparameters to form the final joint loss function. Then, the model parameters are updated using the constraints of this joint loss function through the backpropagation algorithm, thereby optimizing the model.

[0061] Step 5: Reconstruct and evaluate the enhanced speech signal. Based on the trained model, the amplitude spectrum of the enhanced speech signal is obtained. Combined with the phase of the original signal, the enhanced speech signal in the time domain is reconstructed through Inverse Short-Time Fourier Transform (ISTFT). The reconstructed enhanced speech signal is evaluated, mainly using Wideband Perceptual Speech Quality (WB-PESQ) and Short-Time Objective Intelligibility (STOI) metrics to measure the quality and intelligibility of the enhanced speech. During the testing phase, the optimal model is selected based on the results of these metrics to ensure that the final model can provide the best speech enhancement effect in practical applications.

[0062] Step 1 of the present invention specifically includes:

[0063] Step 1-1: In the VoiceBank-DEMAND dataset used, the noisy parts are from the DEMAND dataset, while the clear speech parts are from VoiceBank. The audio files are sampled at a frequency of 48kHz. To construct the mixed speech data for training, 10 different types of noise were selected—including two artificial synthetic noises and eight directly from the DEMAND dataset. The mixed speech was synthesized according to four signal-to-noise ratio levels (15dB, 10dB, 5dB, and 0dB). When constructing the test set, the remaining five noises from DEMAND were selected and combined with recordings of two speakers from VoiceBank. Input data was created based on different signal-to-noise ratio levels than the training set (17.5dB, 12.5dB, 7.5dB, and 2.5dB).

[0064] Steps 1-2: Resample the speech signals in the dataset to 16kHz, and unify the speech length to 3s. Specifically, truncate speech longer than 3s and pad speech shorter than 3s with zeros at the end.

[0065] Steps 1-3: Extract the amplitude spectrum features of the speech signal through short-time Fourier transform. Specifically, we set the window length to 25ms, the frame shift to 6.25ms, and the FFT points to 400 to ensure that the spectral features of the signal can be captured efficiently.

[0066] Step 2 of the present invention specifically includes:

[0067] Step 2-1: The encoder consists of 6 convolutional layers, with output dimensions of 16, 32, 64, 128, 256, and 256 for each layer, respectively. All layers use two-dimensional convolution, with the kernel size set to 3×2, a stride of (2,1), and padding of (0,1). Each layer of the encoder uses an ELU activation function with BatchNorm2d regularization to optimize the training process and improve the model's generalization ability.

[0068] Step 2-2: Each 2D convolutional layer is followed by BatchNorm2d regularization for normalization;

[0069] Steps 2-3: After normalization, use the ELU activation function for nonlinear activation.

[0070] Step 3 of the present invention specifically includes:

[0071] Step 3-1: Introduce the specially designed Channel-S4D block, which enhances the stability of the model when processing long sequences by integrating residual connections, efficient channel attention layer (ECA layer), diagonalized state space model layer, gated linear unit, one-dimensional convolutional layer, dropout, and channel normalization layer. The ECA layer mainly weights features based on their importance, thereby enhancing the model's ability to capture key information. Channel normalization is used instead of the traditional layer normalization method, enabling the model to normalize features of different channels more effectively, thereby improving training stability and model performance. This helps the model to better improve the sensitivity of features of each channel, thus improving the speech enhancement effect and the model's adaptability. By introducing the Channel-S4D module, the model exhibits higher stability and performance when processing long sequence signals, improving the overall effect of speech enhancement as well as the model's performance in parallel processing and modeling long-term time dependencies.

[0072] Step 3-2: The decoder module consists of 6 deconvolutional layers. Each layer and the output of the corresponding encoder convolutional layer are enhanced through skip connections to obtain multi-level and multi-scale features. The input dimensions of each convolutional layer are 512, 512, 256, 128, 64, and 32, respectively. The kernel size and stride are the same as those of the encoder. Each layer of the decoder is also equipped with BatchNorm2d regularization. In terms of activation functions, the first 5 layers use the ELU activation function, and the last layer uses the Softplus activation function. The purpose is to ensure that the amplitude spectrum of the final output remains non-negative and promote high-quality speech reconstruction. This strategy of using hybrid activation functions is to balance the representational power of the model and the quality of the output throughout the decoding process.

[0073] Step 3-1 of the present invention specifically includes:

[0074] Step 3-1-1: Define the size of the hidden layer of the diagonalized state space layer as 256, the size of the state space as 64, and adjust the tensor dimension of the encoder output to match the shape of Channel-S4D;

[0075] Step 3-1-2: Before inputting the features into the diagonalized state space layer, the model is weighted according to the importance of the features through an efficient channel attention layer, thereby enhancing the model's ability to capture key information.

[0076] Step 3-1-3: Use residual connections to combine the input features with the output features processed by the Channel-S4D block to improve the stability of training and the expressive power of the model;

[0077] Step 3-1-4: To further optimize feature processing, gated linear units (GLUs) are added to enhance nonlinear transformation capabilities and enable the model to better adapt to complex speech signals;

[0078] Step 3-1-5: During feature processing, a one-dimensional convolutional layer is added to capture local features at different time steps, and a dropout layer is used to prevent overfitting.

[0079] Step 3-1-6: To prevent gradient explosion, channel normalization is used instead of traditional layer normalization, so that the model can normalize more effectively when processing features of different channels, thereby improving training stability and model performance.

[0080] Step 4 of the present invention specifically includes:

[0081] Step 4-1: Construct a joint loss function. The loss function used during model training is obtained by weighting the mean squared error (MSE) loss and the weighted signal distortion ratio (wSDR) loss. The weight factor is set to 10 to ensure that the two losses have a balanced impact on model optimization during training.

[0082] Step 4-2: Calculate the mean squared error (MSE) loss. The MSE loss measures the difference between enhanced and clean speech in the frequency domain. By calculating the mean squared error between the enhanced and target speech signals, it reflects the frequency domain enhancement effect of the model. The frequency domain loss of the model is selected using the mean squared error (MSE) loss function L. mse Calculation, which is defined as follows:

[0083]

[0084] in, and These represent the spectrum of the estimated speech signal and the spectrum of the clean speech signal, respectively.

[0085] Step 4-3: Calculate the weighted signal distortion ratio (wSDR) loss. The wSDR loss measures the temporal difference between enhanced and clean speech. By calculating the weighted signal distortion ratio between the enhanced and target speech signals, it reflects the temporal enhancement effect of the model. The temporal loss of the model is calculated using the weighted signal distortion ratio loss function, which is weighted by calculating the proportion of noise and clean speech to obtain a more accurate loss. Its definition is as follows:

[0086]

[0087] in, y and Let z and z represent the clean speech signal and the predicted speech signal, respectively. Let α = ||y|| represent the noise in the noisy speech and the noise in the predicted speech, respectively. 2 / (||x|| 2 +||y|| 2 ), representing the energy ratio of noise z to clean speech y;

[0088] Step 4-4: The MSE loss and wSDR loss are weighted and summed to form the final joint loss function. The joint loss function constrains the model's prediction results in both the frequency and time domains, ensuring the model effectively improves speech enhancement in both aspects. The model training uses the joint loss function L... joint It is obtained by weighting the mean square error loss and the weighted signal distortion ratio loss, and β is set as the weighting factor, which is defined as:

[0089] L joint =Lmse +βL wSDR

[0090] Among them, the hyperparameter weighting factor β is optimized through multiple experiments to balance the constraints of these two types of losses.

[0091] Step 5 of the present invention specifically includes:

[0092] Step 5-1: During the training process, the amplitude signal output by the model in each training round is combined with the phase of the original signal and reconstructed into an enhanced speech signal in the time domain through short-time Fourier transform. Then, the loss of the enhanced speech signal in the training round is calculated to ensure that the loss decreases normally during the model training process.

[0093] Step 5-2: During the testing process, two indicators are mainly used to evaluate the quality and intelligibility of enhanced speech: Wideband Perceptual Speech Quality (WB-PESQ) and Short-Time Objective Intelligibility (STOI). The value of WB-PESQ ranges from -0.5 to 4.5, and the value of STOI ranges from 0 to 1. The larger the value of the two indicators, the higher the speech quality and intelligibility.

[0094] Step 5-3: After each training round, calculate the WB-PESQ and STOI values ​​of the enhanced speech signal, record the evaluation results, and monitor the trend of the indicators by comparing the training results of multiple rounds to ensure the continuous improvement of model performance.

[0095] Step 5-4: Considering the randomness of deep learning, the model will be trained multiple times to find the optimal model parameters. Based on the evaluation results of WB-PESQ and STOI, the model with the highest index value will be selected as the optimal model. During this process, it is necessary to monitor the trend of index changes to ensure that the final saved model can provide the best speech enhancement effect in practical applications. The selection of the optimal model is based on its performance on the validation set to ensure that it performs best in practical applications.

[0096] Step 5-5: Save the optimal model for subsequent use and deployment. This systematic training and evaluation process ensures that the final CRNv2 model has superior performance in speech enhancement tasks.

[0097] The simulation experiments and data of this invention are as follows:

[0098] The complete process is implemented in the PyTorch framework, and the Python libraries required include, but are not limited to: NumPy, Librosa, einops, Soundfile, math, etc.

[0099] The audio file was loaded using Librosa and subjected to STFT transformation. The FFT window size n_fft was 400, the window length win_len was 400, and the frame shift hop_len was 100.

[0100] The encoder consists of 6 convolutional layers, with input and output dimensions of (1,16), (16,32), (32,64), (64,128), (128,256), and (256,256) for each layer, respectively. It uses two-dimensional convolution with a kernel size of 3×2, a stride of (2,1), and padding of (0,1). BatchNorm2d is used for regularization, and ELU is used for activation.

[0101] Sequence modeling employs the Channel-S4D block, which enhances the model's stability when processing long sequences by integrating residual connections, efficient channel attention layers (ECAlayers), diagonalized state-space model layers, gated linear units, one-dimensional convolutional layers, dropout, and channel normalization layers. The Channel-S4D block has 256 input and output dimensions.

[0102] The decoder module consists of 6 deconvolutional layers, with input and output dimensions of (512, 512), (512, 256), (256, 128), (128, 64), (64, 32), and (32, 1), respectively. Each layer is connected to the corresponding encoder convolutional layer output via skip connections to enhance the information flow. The convolutional kernel size is 3×2, the stride is (2, 1), and the padding is (0, 1). The activation function for the first 5 convolutional layers is ELU, and the activation function for the last convolutional layer is Softplus.

[0103] During model training, the batch size is 2. The SGD optimizer is used, the learning rate (lr) is set to 1e-3, and the loss function (Loss) is a joint loss function, which is obtained by weighting the mean squared error (MSE) loss and the weighted signal distortion ratio (wSDR) loss, with a weight factor of 10.

[0104] The training run consisted of 100 epochs. During training, the evaluation metrics were Wideband Perceptual Quality of Speech (WB-PESQ) and Short-Time Objective Intelligibility (STOI). WB-PESQ values ​​ranged from -0.5 to 4.5, and STOI values ​​ranged from 0 to 1. The optimal model was selected by comparing these metrics. The CRNv2 model achieved a WB-PESQ score of 2.80 and an STOI score of 0.94 on the VoiceBank-DEMAND dataset, significantly outperforming traditional CRN models.

[0105] Table 1 shows the comparison results of the CRNv2 model of this invention with other speech enhancement models on the VoiceBank-DEMAND dataset:

[0106] Table 1. Performance comparison of CRNv2 with other speech enhancement methods

[0107]

[0108]

[0109] Among them, CRNv2 is a novel model proposed in this invention. The results in the table show that CRNv2 outperforms existing models in both Wideband Perceptual Quality of Speech (WB-PESQ) and Short-Time Objective Intelligibility (STOI) with a lower number of model parameters. This proves that the modified model can obtain enhanced speech with both high subjective and objective quality from noisy speech, and has great learning and reference value in practical applications.

[0110] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A single-channel speech enhancement method based on an improved convolutional recurrent network, characterized in that, The method includes the following steps: Step 1: Download and preprocess the dataset used for model training and testing, and perform short-time Fourier transform (STFT) on the speech in the dataset to extract the amplitude spectrum of the speech signal; Step 2: Input the amplitude spectrum features into the encoder to extract high-dimensional features from the amplitude spectrum; Step 3: Using the constructed Channel-S4D block as a recursive module, the temporal dependencies of the speech signal are modeled. The output of the Channel-S4D block is input into the decoder and restored to the same dimension as the original input. Step 3-1: Construct the Channel-S4D module by integrating residual connections, efficient channel attention layers, diagonalized state space model layers, gated linear units, one-dimensional convolutional layers, dropout layers, and channel normalization layers. This allows the model to acquire key speech information at the channel level, enabling it to more accurately capture the temporal dependencies of speech signals. Step 3-2: The decoder module consists of 6 deconvolutional layers. Each layer is connected to the output of the corresponding encoder convolutional layer through skip connections to enhance the information flow. The input dimensions of the convolutional layers are 512, 512, 256, 128, 64, and 32, respectively. The kernel size and stride are set the same as those of the encoder. The first 5 layers of the decoder use the ELU activation function, and the last layer uses the Softplus activation function. Step 4: Construct a joint loss function. To jointly constrain the difference between enhanced and clean speech in the time and frequency domains, a loss function is adopted that combines time-domain weighted signal distortion ratio loss and frequency-domain mean square error loss. First, the weighted signal distortion ratio loss of enhanced and clean speech in the time domain is calculated. Second, the mean square error loss of enhanced and clean speech in the frequency domain is calculated. The two losses are then weighted and summed. The constraint force of the two losses is dynamically adjusted through the weight hyperparameters to form the final joint loss function. Using the constraints of this joint loss function, the model parameters are updated through the backpropagation algorithm to optimize the model. Step 5: Reconstruct and evaluate the enhanced speech signal. Based on the trained model, the amplitude spectrum of the enhanced speech signal is obtained. Combined with the phase of the original signal, the enhanced speech signal in the time domain is reconstructed through short-time inverse Fourier transform. The reconstructed enhanced speech signal is evaluated. The quality and intelligibility of the enhanced speech are measured by using wideband speech perceptual quality and short-time objective intelligibility metrics. During the testing phase, the optimal model is selected based on the results of these metrics to ensure that the final model can provide the best speech enhancement effect in practical applications.

2. The single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that, Step 1 specifically includes: Step 1-1: In the VoiceBank+DEMAND dataset used, the noisy parts are from the DEMAND dataset, while the clear speech parts are from VoiceBank. The audio files are sampled at 48kHz. The training set contains 10 different types of noise, which are mixed according to four signal-to-noise ratio levels: 15dB, 10dB, 5dB, and 0dB. The test set selects the remaining 5 types of noise from DEMAND and combines them with recordings of two speakers from VoiceBank, which are synthesized according to different signal-to-noise ratio levels than the training set: 17.5dB, 12.5dB, 7.5dB, and 2.5dB. Steps 1-2: Resample the speech signals in the dataset to 16kHz, unify the speech length to 3 seconds, truncate speech longer than 3 seconds, and pad speech shorter than 3 seconds with zeros at the end. Steps 1-3: Extract the amplitude spectrum features of the speech signal by short-time Fourier transform, with a window length of 25ms, a frame shift of 6.25ms, and 400 FFT points.

3. The single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that, Step 2 specifically includes: Step 2-1: The encoder consists of 6 two-dimensional convolutional layers. The output dimensions of each layer are 16, 32, 64, 128, 256 and 256 respectively. The size of the convolutional kernel is set to 3×2, the stride is configured as (2,1), and the padding is (0,1). Step 2-2: Each 2D convolutional layer is followed by BatchNorm2d regularization for normalization; Steps 2-3: After normalization, use the ELU activation function for nonlinear activation.

4. The single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that, Step 3-1 specifically includes: Step 3-1-1: Define the size of the hidden layer of the diagonalized state space layer as 256, the size of the state space as 64, and adjust the tensor dimension of the encoder output to match the shape of Channel-S4D; Step 3-1-2: Before inputting the features into the diagonalized state space layer, the model is weighted according to the importance of the features through an efficient channel attention layer, thereby enhancing the model's ability to capture key information. Step 3-1-3: Use residual connections to combine the input features with the output features processed by the Channel-S4D block to improve the stability of training and the expressive power of the model; Step 3-1-4: To optimize feature processing, gated linear units are added to enhance nonlinear transformation capabilities and enable the model to better adapt to complex speech signals; Step 3-1-5: During feature processing, a one-dimensional convolutional layer is added to capture local features at different time steps, and a dropout layer is used to prevent overfitting. Step 3-1-6: To prevent gradient explosion, channel normalization is used instead of traditional layer normalization, so that the model can normalize more effectively when processing features of different channels, thereby improving training stability and model performance.

5. A single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that, Step 4 specifically includes: Step 4-1: Select the mean squared error (MSE) loss function L for the frequency domain loss of the model. mse Calculation, which is defined as follows: in, and These represent the spectrum of the estimated speech signal and the spectrum of the clean speech signal, respectively. Step 4-2: The temporal loss of the model is calculated using a weighted signal-to-distortion ratio loss function. This function is weighted by calculating the proportions of noise and clean speech to obtain a more accurate loss. The definition is as follows: in, y and Let z and z represent the clean speech signal and the predicted speech signal, respectively. Let α = ||y|| represent the noise in the noisy speech and the noise in the predicted speech, respectively. 2 / (||x|| 2 +||y|| 2 ), representing the energy ratio of noise z to clean speech y; Step 4-3: Model training uses the joint loss function L joint It is obtained by weighting the mean square error loss and the weighted signal distortion ratio loss, and β is set as the weighting factor, which is defined as: L joint =L mse +βL wSDR Among them, the hyperparameter weighting factor β is optimized through multiple experiments to balance the constraints of these two types of losses.

6. The single-channel speech enhancement method based on an improved convolutional recurrent network according to claim 1, characterized in that, Step 5 specifically includes: Step 5-1: During the training process, the amplitude signal output by the model in each training round is combined with the phase of the original signal and reconstructed into an enhanced speech signal in the time domain through short-time Fourier transform. Then, the loss of the enhanced speech signal in the training round is calculated to ensure that the loss decreases normally during the model training process. Step 5-2: During the testing process, two metrics, broadband speech perceptual quality and short-term objective intelligibility, are used to measure the quality and intelligibility of the enhanced speech. Step 5-3: After each training round, calculate the WB-PESQ and STOI values ​​of the model trained in this round on the test set, record and compare the evaluation results of each round to ensure that the model performance is continuously improved. Step 5-4: Considering the randomness of deep learning, the model will be trained multiple times to find the optimal model parameters. Based on the evaluation results of WB-PESQ and STOI, the model with the highest index value will be selected as the optimal model. During this process, it is necessary to monitor the trend of index changes to ensure that the final saved model can provide the best speech enhancement effect in practical applications. Step 5-5: Save the optimal model for subsequent use and deployment. Through this systematic training and evaluation process, the final CRNv2 model is guaranteed to have superior performance in speech enhancement tasks.

Images