A lightweight single-channel speech enhancement method based on improved convolutional recurrent networks

Abstract

This invention relates to the field of speech enhancement technology, specifically to a lightweight single-channel speech enhancement method based on an improved convolutional recurrent network. It fully utilizes the superior feature extraction capabilities of the improved convolutional recurrent network, employing an aggregated grouped dual-path recurrent network and a convolutional hybrid grouped dual-path recurrent network to enhance the deep time-frequency features of multiple channels and fuse features between channels. This enriches the speech information contained in the deep features. The deep features are then used to train a separation model, further enhancing speech performance. Furthermore, the invention proposes an aggregated grouped dual-path recurrent network and a convolutional hybrid grouped dual-path recurrent network, improving the grouped dual-path recurrent network architecture. This not only enhances the speech enhancement performance of the convolutional recurrent network but also maintains the model's lightweight nature, thereby improving the effectiveness of the enhancement model and increasing the clarity and intelligibility of the enhanced speech.

Description

Technical Field

[0001] This invention relates to the field of speech enhancement technology, and in particular to a lightweight single-channel speech enhancement method based on an improved convolutional recurrent network. Background Technology

[0002] In daily life, speech plays an indispensable role as a medium for human communication. Speech enhancement is a technique that aims to recover clean speech from noisy speech to improve speech intelligibility and perceived quality. This technique is currently widely used in voice communication, digital hearing aids, and other fields, and is used as the front end of many speech systems, such as automatic speech recognition, speech coding, and human-computer interaction.

[0003] With the rapid development of AI and deep neural networks, artificial neural networks have become a favorite in the fields of computer and communication technologies due to their excellent modeling capabilities, highly abstract predictive abilities, and superior relational mapping capabilities. Single-channel speech enhancement algorithms based on deep learning are widely used and researched in the field of speech enhancement.

[0004] Compared to traditional speech enhancement algorithms, deep neural network (DNN)-based methods offer overwhelming performance, but often come with significant model complexity. DNN-based speech enhancement can be broadly categorized into time-frequency and time-domain methods. Time-frequency methods aim to extract noise features from acoustic characteristics (e.g., complex spectrum or logarithmic power spectrum). Common training objectives include ideal ratio mask (IRI) and target amplitude spectrum (TMS). Phase spectrum is also considered beneficial for speech quality. Time-domain methods directly estimate clean speech waveforms through end-to-end training, avoiding the hassle of estimating phase information in the time-frequency domain. However, time-domain methods struggle to model extremely long sequences, and traditional recurrent neural networks (RNNs) are also inefficient at handling such lengths. Therefore, a dual-path recurrent neural network (DPRNN) was proposed to address this issue. In this DPRNN, long sequence features are segmented into smaller blocks and iteratively processed by intra-block and inter-block RNNs, thus reducing the sequence length that each RNN must process. Intra-block operations in DPRNN aim to model signal features within a single frame. This method is also applicable to the frequency domain and has the potential to fully utilize the harmonic spectrum structure of speech. In lightweight scenarios, to reduce computational overhead, Grouped Dual-path RNN (G-DPRNN) has been proposed. It replaces the RNN in DPRNN with a grouped RNN, effectively reducing the number of model parameters and computational complexity using group recurrent layers. Recently, a network structure called Convolution Recurrent Network (CRN) has been proposed. CRN leverages the advantages of CNN and RNN, not only capturing local patterns in the spectrum but also modeling dependencies between consecutive time frames. Therefore, combining the time-frequency domain characteristics of G-DPRNN and CRN achieves performance comparable to or better than traditional RNN models with a significant reduction in the number of parameters and computational cost.

[0005] Traditional models combining G-DPRNN and CRN suffer from limitations in lightweight networks, exhibiting limited capabilities in time-frequency dynamic modeling, feature space integration, and detail capture. Furthermore, the recurrent neural network (RNN) structure of G-DPRNN results in a lack of direct interactions between channels, restricting the network's versatility. Summary of the Invention

[0006] The purpose of this invention is to provide a lightweight single-channel speech enhancement method based on an improved convolutional recurrent network, which solves the limitations of traditional models combining G-DPRNN and CRN in terms of lightweight networks. These models exhibit limited capabilities in time-frequency dynamic modeling, feature space integration, and detail capture. Furthermore, the recurrent neural network (RNN) structure of G-DPRNN leads to a lack of direct interaction between channels, which limits the diversity of the network.

[0007] To achieve the above objectives, this invention provides a lightweight single-channel speech enhancement method based on an improved convolutional recurrent network. The lightweight single-channel speech enhancement method based on the improved convolutional recurrent network includes the following steps:

[0008] Step 1: Preprocess the input single-channel speech signal by extracting the real spectrum, imaginary spectrum and amplitude spectrum features of the speech signal to form a three-channel real-valued tensor input. At the same time, use the equivalent rectangular bandwidth frequency band processing module to compress the high-frequency band.

[0009] Step 2: Expand and reconstruct the compressed frequency band features using the spectrum feature enhancement module;

[0010] Step 3: Input the expanded and reconstructed frequency band features into the improved convolutional recurrent network to extract the depth features of the speech and enhance the complex ratio mask of the signal as the output. By training the network weights and biases, the trained convolutional recurrent network for enhancing the speech signal is obtained.

[0011] Step 4: Input the audio features of the noisy speech signal used for testing into the trained convolutional recurrent network for enhancing the speech signal, so as to enhance the noisy speech signal.

[0012] As a further improvement of the present invention, the preprocessing in step 1 includes framing and windowing; the equivalent rectangular bandwidth frequency band processing in step 1 is to merge high-frequency bands and use linear transformation to map the amplitude characteristics of multiple high-frequency bands into one feature dimension.

[0013] As a further improvement of the present invention, the spectral feature enhancement module in step 2 remaps the sub-band information of the frequency dimension to the channel dimension through spectral expansion, spectral dimension conversion to channel dimension and convolution operation.

[0014] As a further improvement of the present invention, the improved convolutional recurrent network in step 3 consists of four parts: encoder, decoder, aggregated grouping dual-path recurrent network, and convolutional hybrid grouping dual-path recurrent neural network.

[0015] The encoder is used to generate a high-dimensional embedding representation, the decoder is used to progressively restore the original dimension of the features, the aggregated grouped dual-path recurrent network further enhances the high-dimensional time-frequency feature input, the convolutional hybrid grouped dual-path recurrent neural network is used for local feature extraction and feature fusion between channels, and the output of the decoder deconvolution block is a complex ratio mask of the enhanced signal. After multiplying the mask with the input complex spectrum matrix, the enhanced speech signal is reconstructed by inverse short-time Fourier transform, and the parameters are optimized by constrained loss function to train the convolutional recurrent network.

[0016] As a further improvement of the present invention, the encoder in step 3 consists of two convolutional layers and three grouped convolutional layers. Specifically, the encoder input feature size is 9×T×129, the kernel size of the first two convolutional layers is 1×5, the number of input and output channels is 16, the stride in the frequency dimension is 2, and the stride in the time dimension is 1. Therefore, the number of time frames T remains unchanged, resulting in an output feature size of 16×T×33 for the first two layers. The kernel size of the last three grouped convolutional layers is 3×3, and the dilation coefficients gradually increase to 1, 2, and 5 respectively to expand the receptive field. The number of input and output channels is 16, resulting in a depth feature size of 16×T×33 for the last three layers.

[0017] The decoder is connected to the encoder in a skip connection through a series of deconvolutional layers to gradually restore the time-frequency resolution of the features and reconstruct the enhanced speech signal;

[0018] The aggregated grouped dual-path recurrent network combines a grouped dual-path recurrent network with a time-frequency attention mechanism. The time-frequency attention mechanism includes depthwise separable convolution, adaptive mean pooling, intra-band convolution, and inter-band convolution to generate attention weights, and assigns weights to each time-frequency position.

[0019] As a further improvement of the present invention, the adaptive average pooling is divided into average pooling along the time dimension and average pooling along the frequency dimension, generating two different axis vectors T for the time axis and the frequency axis, respectively. P and F P Then, broadcast addition processing is performed; the intra-band convolution and inter-band convolution use convolution kernels of size 1×k and k×1 to perform strip convolution modeling in two directions, intra-band and inter-band, respectively, to generate attention weights in the time-frequency domain. Then, a depthwise convolution of size 3×3 is used to further extract local details of the input features. The attention features are assigned to each time-frequency position using the Hadamard product. Finally, a multilayer perceptron module is used to perform nonlinear mapping on the time-frequency features.

[0020] As a further improvement of the present invention, the expressions for intra-band convolution and inter-band convolution are as follows:

[0021]

[0022] Where γ represents large-kernel strip convolution, and k represents the kernel size of the strip convolution. Represents the time axis vector T P and frequency axis vector F P The result after broadcast addition, where φ represents the batch normalization after passing through the ReLU function, and δ represents the Sigmoid activation function;

[0023] The expression for attention feature assignment is:

[0024]

[0025] Where, γ 3*3 This indicates a 3×3 depthwise convolution kernel, where y is the attention feature obtained in the previous step. Represents the Adama product.

[0026] As a further improvement of the present invention, the convolutional hybrid grouped dual-path recurrent neural network in step 3 is used to combine an improved depthwise separable convolutional module with a grouped dual-path recurrent network. The improved depthwise separable convolutional module includes depthwise convolution, pointwise convolution and residual connections. The depthwise convolution uses a large convolutional kernel to extract global information for each channel. The pointwise convolution adopts an inverted bottleneck design. The hidden dimension between two pointwise convolutional layers is set to four times the width of the input dimension to fully fuse the global information between channels. Finally, the GELU activation function and BatchNorm regularization are used after each convolutional layer.

[0027] The expression for the improved depthwise separable convolutional module is:

[0028] X′ out =BN(σ l {DepthwiseConv(X in )})+X in

[0029] X″ out =BN(σ l {PointwiseConv(X′ out )})

[0030] X out =BN(σ l {PointwiseConv(X″ out )})

[0031] Among them, X in This indicates that the output of the grouped dual-path recurrent network is used as the input to a depthwise separable convolutional module, σ ldenoted as GELU activation function, and BN represents BatchNorm regularization.

[0032] As a further improvement of the present invention, step 4 specifically involves: preprocessing the noisy speech signal used for testing, extracting the real spectrum, imaginary spectrum, and amplitude spectrum features of the noisy speech signal, inputting them into the improved convolutional recurrent network after optimization training, obtaining a predicted complex ratio mask, multiplying the predicted complex ratio mask with the real spectrum and imaginary spectrum features to obtain the target complex spectrum, and reconstructing the speech signal through inverse short-time Fourier transform to complete the enhancement of the noisy speech signal.

[0033] As a further improvement of the present invention, the improved convolutional recurrent network has only 30.2K parameters and a computational complexity of only 48.2MMACs, making it suitable for lightweight speech enhancement applications on edge devices.

[0034] As a further improvement of the present invention, the lightweight single-channel speech enhancement method based on the improved convolutional recurrent network further includes combining perceptual contrast stretching technology to further optimize and enhance the perceptual quality of speech, significantly improving speech clarity and intelligibility; the perceptual contrast stretching technology utilizes the sensitivity of the human ear to specific frequency bands to redistribute the weights of specific frequency bands, and uses the PCS weight array to multiply the logarithmic amplitude spectrum bandwise to amplify the specific frequency bands.

[0035] This invention presents a lightweight single-channel speech enhancement method based on an improved convolutional recurrent network (CRN). It fully leverages the superior feature extraction capabilities of the improved CRN by employing an aggregated grouped dual-path recurrent network and a convolutional hybrid grouped dual-path recurrent network to enhance the deep time-frequency features of multiple channels and fuse features between channels. This enriches the speech information contained in the deep features. The deep features are then used to train a separation model, further improving the performance of the enhanced speech. Furthermore, this technical solution addresses the limitations of the CRN's time-frequency feature extraction capability, feature space integration capability, and inter-channel feature interaction capability under lightweight model requirements, resulting in limited improvement in enhanced speech quality. Based on the traditional CRN model, it proposes an aggregated grouped dual-path recurrent network and a convolutional hybrid grouped dual-path recurrent network, improving the grouped dual-path recurrent network architecture. This not only enhances the speech enhancement performance of the CRN but also maintains the model's lightweight nature, thereby improving the effectiveness of the enhancement model and increasing the clarity and intelligibility of the enhanced speech. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a structural diagram of a lightweight single-channel speech enhancement model based on an improved convolutional recurrent network provided by the present invention.

[0038] Figure 2 This invention provides Figure 1 The model structure diagram of the Aggregation Grouping Dual Path Recurrent Network (AG-DPRNN).

[0039] Figure 3 This invention provides Figure 1 The model structure diagram of the medium convolutional hybrid grouped dual-path recurrent network CG-DPRNN. Detailed Implementation

[0040] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0041] Please see Figures 1 to 3 This invention provides a lightweight single-channel speech enhancement method based on an improved convolutional recurrent network. The lightweight single-channel speech enhancement method based on the improved convolutional recurrent network includes the following steps:

[0042] Step 1: Preprocess the input single-channel speech signal by extracting the real spectrum, imaginary spectrum and amplitude spectrum features of the speech signal to form a three-channel real-valued tensor input. At the same time, use the equivalent rectangular bandwidth frequency band processing module to compress the high-frequency band.

[0043] Step 2: Expand and reconstruct the compressed frequency band features using the spectrum feature enhancement module;

[0044] Step 3: Input the expanded and reconstructed frequency band features into the improved convolutional recurrent network to extract the depth features of the speech and enhance the complex ratio mask of the signal as the output. By training the network weights and biases, the trained convolutional recurrent network for enhancing the speech signal is obtained.

[0045] Step 4: Input the audio features of the noisy speech signal used for testing into the trained convolutional recurrent network for enhancing the speech signal, so as to enhance the noisy speech signal.

[0046] Furthermore, the improved convolutional recurrent network in step 3 consists of four parts: an encoder, a decoder, an aggregated grouping dual-path recurrent network, and a convolutional hybrid grouping dual-path recurrent neural network.

[0047] The encoder is used to generate a high-dimensional embedding representation, the decoder is used to progressively restore the original dimension of the features, the aggregated grouped dual-path recurrent network further enhances the high-dimensional time-frequency feature input, the convolutional hybrid grouped dual-path recurrent neural network is used for local feature extraction and feature fusion between channels, and the output of the decoder deconvolution block is a complex ratio mask of the enhanced signal. After multiplying the mask with the input complex spectrum matrix, the enhanced speech signal is reconstructed by inverse short-time Fourier transform, and the parameters are optimized by constrained loss function to train the convolutional recurrent network.

[0048] In this embodiment, the superior feature extraction capabilities of improved convolutional recurrent networks are fully utilized. Aggregated grouped dual-path recurrent networks and convolutional hybrid grouped dual-path recurrent networks are used to enhance the deep time-frequency features of multiple channels and fuse features between channels, enriching the speech information contained in the deep features. These deep features are then used to train a separation model, further improving the performance of the enhanced speech. Furthermore, this technical solution addresses the limitations of convolutional recurrent networks in terms of time-frequency feature extraction, feature space integration, and inter-channel feature interaction under lightweight model requirements, resulting in limited improvement in enhanced speech quality. Based on traditional convolutional recurrent network models, aggregated grouped dual-path recurrent networks and convolutional hybrid grouped dual-path recurrent networks are proposed. These improvements to the grouped dual-path recurrent network architecture not only enhance the speech enhancement performance of convolutional recurrent networks but also maintain the model's lightweight nature, thereby increasing the effectiveness of the enhancement model and improving the clarity and intelligibility of the enhanced speech.

[0049] The specific steps of this invention are as follows:

[0050] Step 1: Preprocess multiple speech signals to extract the real spectrum, imaginary spectrum and amplitude spectrum features of the noisy speech signal, and compress the high-frequency band using the Equivalent Rectangular Bandwidth (ERB) processing module.

[0051] Step 1.1: Preprocess multiple speech signals.

[0052] Preprocessing techniques are of considerable importance in the field of digital speech processing. Because speech signals are short-term stationary, preprocessing is necessary before performing a Fast Fourier Transform (FFT). Common preprocessing methods include framing, windowing, and pre-emphasis. In speech signal enhancement techniques, preprocessing can improve the effectiveness of speech enhancement to a certain extent.

[0053] In this embodiment, the noisy speech signal is preprocessed. The sampling rate of each speech signal is 48kHz and the downsampling rate is 16kHz. Of course, in other embodiments, the number of speech signals can be set to other values, and the sampling rate of each speech signal can be set to other values, as long as the preprocessing of the speech signal can be achieved. No restrictions are imposed here.

[0054] Step 1.2: Extract multi-channel features of the preprocessed speech signal.

[0055] A Short-Time Fourier Transform (STFT) is performed on the preprocessed noisy speech signal to extract its real, imaginary, and amplitude spectrum features. For each time frame, the STFT contains 257 frequency bands in both the real and imaginary parts, as well as 257 frequency bands in the amplitude spectrum. Therefore, in the time dimension, the spectrum of each frame contains 257 frequency band features in the real, imaginary, and amplitude parts. These features are then concatenated in the channel dimension to obtain three channels.

[0056] Step 1.3: Compress the high-frequency band using the Equivalent Rectangular Bandwidth (ERB) band processing module.

[0057] The spectrum of each channel is divided into multiple ERB sub-bands. The high-frequency portion is compressed using the ERB's fc function to reduce high-frequency details while preserving the low-frequency characteristics of the speech signal. The high-frequency and low-frequency features are then combined to return the compressed time-frequency features.

[0058] Step 2: Expand and reconstruct the compressed frequency band features using the Subband Feature Extraction (SFE) module.

[0059] Specifically, multiple local sub-band feature blocks are separated on the frequency axis using a sliding window approach. The features extracted from each window are then rearranged according to a new shape, giving the output feature map a new channel dimension. That is, the frequency dimension of each feature map increases, and the number of channels increases. By extracting local frequency band features, the model's ability to model local conditions in each frequency band is enhanced, allowing for better modeling of complex time-frequency characteristics.

[0060] Step 3: Input the above features into the improved convolutional recurrent network, which consists of four parts: encoder, decoder, aggregated grouped dual-path recurrent network, and convolutional hybrid grouped dual-path recurrent neural network.

[0061] Step 3.1: The encoder extracts features, and the decoder restores features.

[0062] The time-frequency features extracted by SFE are processed by the encoder to obtain deep features. The encoder input feature size is 9×T×129. The kernel size of the first two convolutional layers is 1×5, the number of input and output channels is 16, the stride in the frequency dimension is 2, and the stride in the time dimension is 1. Therefore, the number of time frames T remains unchanged, resulting in a kernel size of 16×T×33 for the first two layers. The kernel size of the last three grouped convolutional layers is 3×3, and the dilation coefficients gradually increase to 1, 2, and 5 to expand the receptive field. The number of input and output channels is 16, resulting in a deep feature size of 16×T×33 for the last three layers.

[0063] Please see Figure 1 As shown, the decoder gradually recovers the time-frequency resolution of the features and reconstructs the enhanced speech signal through a series of deconvolutional layers and skip connections with the encoder. This will not be described in detail here.

[0064] Step 3.2: Extract deep time-frequency features using Aggregated Grouped Dual-path RNN (AG-DPRNN).

[0065] Faced with limitations in lightweight networks, such as the number of channels, the time-frequency dynamic modeling capability of Grouped Dual-path Recurrent Neural Networks (G-DPRNN) is also limited. To address this issue, this paper combines G-DPRNN with a time-frequency attention mechanism to enhance its time-frequency dynamic modeling capability. First, G-DPRNN divides the input features and hidden states into two disjoint groups. Each group is fed into a recurrent layer with half the number of parameters compared to the original group. Then, a representation rearrangement layer is applied to obtain the grouped outputs, which are then fed into the Dual-path Recurrent Neural Network (DPRNN). The intra-frame RNN models the spectral patterns within a single frame, while the inter-frame RNN models the temporal dependencies of specific frequency points. Intra-frame modeling uses a grouped bidirectional GRU, and inter-frame modeling uses a grouped unidirectional GRU, ensuring the causality of the model. Finally, a time-frequency attention mechanism is used to process the output features of size 16×T×33.

[0066] Please see Figure 2 As shown, the time-frequency attention mechanism includes depthwise separable convolution, adaptive mean pooling, intra-band convolution, and inter-band convolution to generate attention weights, assigning weights to each time-frequency position. Specifically, depthwise separable convolution captures the local time-frequency relationships of features; adaptive mean pooling is divided into average pooling along the time dimension and average pooling along the frequency dimension to capture axial global context information in both directions, generating two different axial vectors T for the time axis and frequency axis, respectively. P and F P Then, broadcast addition processing is performed; intra-band convolution and inter-band convolution are modeled using convolution kernels of size 1×k and k×1 in both directions within and between the frequency bands, respectively, to generate attention weights in the time-frequency domain. The expressions for intra-band convolution and inter-band convolution are:

[0067]

[0068] Where γ represents large-kernel strip convolution, and k represents the kernel size of the strip convolution. Represents the time axis vector T P and frequency axis vector F P The result after broadcast addition. φ represents the batch normalization after passing through the ReLU function, and δ represents the Sigmoid activation function.

[0069] Next, a 3×3 depthwise convolution is used to further extract local details of the input features. The Hadamard product is then used to assign attention features to each time-frequency location. The expression for attention feature assignment is:

[0070]

[0071] Where, γ 3*3 This indicates a 3×3 depthwise convolution kernel, where y is the attention feature obtained in the previous step. Represents the Adama product.

[0072] Finally, a multilayer perceptron (MLP) module is used to perform nonlinear mapping on the time-frequency features. It contains two 1×1 convolutional layers instead of traditional fully connected layers, which performs nonlinear mapping and feature enhancement with lower computational overhead, while the output feature dimension remains 16×T×33.

[0073] Step 3.3: ConvMixed Grouped Dual-path RNN (CG-DPRNN) is used to further extract deep time-frequency features.

[0074] Due to the limitations of lightweight networks, Grouped Dual-path Recurrent Neural Networks (G-DPRNNs) are insufficient in their ability to integrate feature spaces and capture details. Furthermore, the recurrent neural network (RNN) structure of G-DPRNNs leads to indirect interactions between channels, which in turn limits feature diversity. Therefore, CGDPRNN combines an improved deep separable convolutional module with G-DPRNNs, providing an additional feature enhancement and fusion module that can effectively fuse and correlate features between channels simultaneously.

[0075] Please see Figure 2 As shown, specifically, the output features of AG-DPRNN are first processed by an improved depthwise separable convolution module. This module consists of depthwise convolution (i.e., the number of groups equals the number of channels) and pointwise convolution (i.e., the kernel size is 1×1). The depthwise convolution is used to extract fine-grained features within the local frequency domain of each time frame, and then residually concatenates them with the time-frequency features from the previous step. The concatenated features are then mixed using pointwise convolution operations to blend the time-frequency information between the channels.

[0076] To fully blend the time-frequency dimension and information between channels, we apply two pointwise convolutions after the depthwise convolution and employ an inverted bottleneck design. Specifically, this design involves setting the hidden dimension between the two pointwise convolutional layers to four times the input dimension. The expanded hidden dimension allows for a comprehensive and complete blending of the fine-grained features extracted by the depthwise convolution. Furthermore, we use GELU activation and a post-activation BatchNorm layer after each convolution. Simultaneously, from a model lightweighting perspective, depthwise separable convolution effectively reduces network parameters and computational cost compared to ordinary convolution. The expression for the improved depthwise separable convolution module is:

[0077] X′ out =BN(σ l {DepthwiseConv(X in )})+X in

[0078] X″ out =BN(σ l {PointwiseConv(X′ out )})

[0079] X out =BN(σ l {PointwiseConv(X″ out )})

[0080] Among them, X in This indicates that the output of the grouped dual-path recurrent network is used as the input to a depthwise separable convolutional module, σ l denoted by GELU activation function, and BN by BatchNorm regularization. The output of the improved depthwise separable convolutional module is used as the input to G-DPRNN for further extraction of depthwise time-frequency features.

[0081] Step 3.4: Train and improve the weights and biases of the convolutional recurrent network.

[0082] The parameter optimization process of improved convolutional recurrent networks is divided into forward propagation and back propagation stages. The forward propagation (FP) stage randomly initializes the weights and biases of each neuron in each layer. The back propagation (BP) stage integrates optimization to minimize the joint constraint loss function, iteratively updating and adjusting the weights and biases of each neuron in each layer. The core of the BP and FP stages is the loss function, which is optimized using the gradient descent algorithm to get as close to its minimum as possible. The expression for the loss function is:

[0083]

[0084] in, 's' and 's' represent enhanced speech and clean speech, respectively. Let S and γ represent the enhanced speech spectrogram and the clean speech spectrogram, respectively. The values ​​of parameters γ and δ are set to 0.01 and 0.3, respectively. The remaining expressions in the above formula are:

[0085]

[0086]

[0087] The amplitude loss is calculated with a scale of 0.3 to reduce the sensitivity to amplitude differences, while the real part loss and imaginary part loss are calculated with a scale of 0.7 to balance the errors under different amplitudes.

[0088] Finally, an improved convolutional recurrent network for enhancing speech signals is obtained after training.

[0089] Step 4: Input the time-frequency features of the noisy speech signal used for testing into the improved convolutional recurrent network after optimization training to enhance the noisy speech signal.

[0090] 4.1 Testing phase.

[0091] First, the noisy speech signal was preprocessed, and then STFT was performed to extract the real spectrum, imaginary spectrum, and amplitude spectrum features of the noisy speech signal.

[0092] 4.2 Reconstructing the speech time-domain signal.

[0093] In the stage of reconstructing the speech time-domain signal, the signal is input into the improved convolutional recurrent network after optimization training to obtain the predicted complex ratio mask. The predicted complex ratio mask is multiplied with the real spectrum and imaginary spectrum features to obtain the target complex spectrum. The speech signal is then reconstructed through inverse short-time Fourier transform to complete the enhancement of the noisy speech signal.

[0094] 4.3 Performance Evaluation.

[0095] To evaluate the performance of our proposed lightweight model, we chose to use the VoiceBank+DEMAND dataset (a recognized benchmark in the field of speech enhancement). This dataset combines noisy samples from the VoiceBank corpus (consisting of clean speech recordings) and the DEMAND dataset (creating a set of realistic noisy speech samples). The VoiceBank+DEMAND dataset contains 11,572 pairs of noisy and clean speech samples for training and 824 pairs for testing. To ensure the consistency of all data, each audio sample was resampled to a consistent 16kHz frequency. The experimental results are the average of the 824 speech results.

[0096] In the speech preprocessing, the Short Time Fourier Transform (STFT) is calculated using a Hanning window with square root weighting, set to a length of 32 milliseconds and a jump length of 16 milliseconds, while the Fourier Transform length is 512. The input features are the real spectrum, imaginary spectrum, and amplitude spectrum features of the speech signal, combined according to the channel.

[0097] This invention employs multiple speech metrics to measure the accuracy and effectiveness of the proposed algorithm, including Perceptual Evaluation of Speech Quality (PESQ), Scale-Invariant Signal-to-Noise Ratio (SISNR), and Short-Time Objective Intelligence (STOI). The values ​​of these metrics are positively correlated with speech enhancement performance.

[0098] Additionally, during the evaluation phase, we employed Perceptual Contrast Stretching (PCS), a spectral enhancement technique that leverages the human ear's sensitivity to specific frequency bands to improve the auditory quality of speech. This method adjusts the amplitude spectrum of speech based on the perceived significance of each frequency band, thereby improving the intelligibility of the most important regions. In our study, PCS served as an adjunct step in the evaluation phase following the initial enhancement stage, further enhancing speech quality with a focus on optimizing perceptual auditory characteristics.

[0099] To verify the effectiveness of the two improvement strategies for G-DPRNN, we conducted ablation experiments on the improved models. The experimental results are shown in Table 1. Both AG-DPRNN, which uses a time-frequency attention mechanism to further enhance the high-dimensional time-frequency feature input, and CG-DPRNN, which employs an improved depthwise separable convolutional module for local feature extraction and inter-channel feature fusion, outperform G-DPRNN under very limited computational resources. The best performance metrics can be achieved by integrating AG-DPRNN and CG-DPRNN.

[0100] To verify the lightweight nature and performance advantages of the improved convolutional recurrent network, the method of this invention was compared with other algorithms. The experimental results are shown in Table 2. The improved overall model ACCRN achieved a PESQ of 2.95 with only 30k model parameters and 48M MACs computation.

[0101] Although DeepFilterNet2, CCFNet+, and Dense-TSNet all outperform ACCRN slightly, DeepFilterNet2 has a model size of 2.3M, while ACCRN's size is only 3.4% of that. Furthermore, CCFNet+ has a computational cost of 1.47G, while ACCRN's computational cost is only 3% of that.

[0102] Meanwhile, compared to the recently proposed FSPEN (2024) model, ACCRN significantly reduces parameters and computational cost while maintaining comparable performance. As for Dense-TSNet (2024), although its model is more lightweight than ours, its enormous computational cost inevitably limits its real-time performance, making it less versatile for deployment on edge devices compared to our model. The results demonstrate that ACCRN achieves highly competitive performance with minimal resource consumption, making it more suitable for wearable and IoT devices.

[0103] Table 1 Ablation experiments of the improved convolutional recurrent network ACCRN

[0104]

[0105] Table 2 Comparison of the improved convolutional recurrent network ACCRN with other classic lightweight models

[0106]

[0107] In summary, this invention provides a lightweight single-channel speech enhancement method based on an improved convolutional recurrent network. The backbone network is a convolutional recurrent network based on an encoder and decoder, and two improvement strategies are used in the grouped dual-path recurrent network. One strategy combines a time-frequency attention mechanism with the grouped dual-path recurrent network, and the other uses an improved depthwise separable convolution for local feature extraction and feature fusion between channels, improving speech enhancement performance and increasing the clarity and intelligibility of the enhanced speech. These two strategies maintain the lightweight nature of the model while improving performance, making it suitable for a wider range of real-world applications. In the performance evaluation stage, perceptual contrast stretching technology is combined to further optimize the perceptual quality of the enhanced speech, significantly improving speech clarity and intelligibility.

[0108] The above description discloses only one preferred embodiment of the present invention, and should not be construed as limiting the scope of the present invention. Those skilled in the art will understand that all or part of the processes of the above embodiments can be implemented, and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A lightweight single-channel speech enhancement method based on an improved convolutional recurrent network, characterized in that, Includes the following steps: Step 1: Preprocess the input single-channel speech signal by extracting the real spectrum, imaginary spectrum and amplitude spectrum features of the speech signal to form a three-channel real-valued tensor input. At the same time, use the equivalent rectangular bandwidth frequency band processing module to compress the high-frequency band. Step 2: Expand and reconstruct the compressed frequency band features using the spectrum feature enhancement module; Step 3: Input the expanded and reconstructed frequency band features into the improved convolutional recurrent network to extract the depth features of the speech and enhance the complex ratio mask of the signal as the output. By training the network weights and biases, the trained convolutional recurrent network for enhancing the speech signal is obtained. Step 4: Input the audio features of the noisy speech signal used for testing into the trained convolutional recurrent network for enhancing the speech signal, so as to enhance the noisy speech signal. The improved convolutional recurrent network in step 3 consists of four parts: encoder, decoder, aggregation grouping dual-path recurrent network, and convolutional hybrid grouping dual-path recurrent neural network. The encoder is used to generate a high-dimensional embedding representation, the decoder is used to progressively restore the original dimension of the features, the aggregated grouped dual-path recurrent network further enhances the high-dimensional time-frequency feature input, the convolutional hybrid grouped dual-path recurrent neural network is used for local feature extraction and feature fusion between channels, and the output of the decoder deconvolution block is a complex ratio mask of the enhanced signal. After multiplying the mask with the input complex spectrum matrix, the enhanced speech signal is reconstructed by inverse short-time Fourier transform, and the parameters are optimized by constrained loss function to train the convolutional recurrent network. The convolutional hybrid grouped dual-path recurrent neural network in step 3 combines an improved depthwise separable convolutional module with a grouped dual-path recurrent network. The improved depthwise separable convolutional module includes depthwise convolution, pointwise convolution, and residual connections. The depthwise convolution uses a large convolutional kernel to extract global information for each channel. The pointwise convolution employs an inverted bottleneck design, with the hidden dimension between two pointwise convolutional layers set to four times the width of the input dimension to fully fuse global information between channels. Finally, the GELU activation function and BatchNorm regularization are used after each convolutional layer. The expression for the improved depthwise separable convolutional module is: ； ； ； in, This indicates that the output of the grouped dual-path recurrent network is used as the input to the depthwise separable convolutional module. This represents the GELU activation function. This indicates BatchNorm regularization.

2. The lightweight single-channel speech enhancement method based on an improved convolutional recurrent network as described in claim 1, characterized in that, The preprocessing in step 1 includes framing and windowing; the equivalent rectangular bandwidth processing in step 1 involves merging high-frequency bands and using linear transformation to map the amplitude characteristics of multiple high-frequency bands into a single feature dimension.

3. The lightweight single-channel speech enhancement method based on an improved convolutional recurrent network as described in claim 1, characterized in that, The spectral feature enhancement module described in step 2 remaps the sub-band information of the frequency dimension to the channel dimension through spectral unrolling, spectral dimension to channel dimension conversion, and convolution operations.

4. The lightweight single-channel speech enhancement method based on an improved convolutional recurrent network as described in claim 3, characterized in that, The encoder in step 3 consists of two convolutional layers and three grouped convolutional layers. Specifically, the encoder input feature size is 9 × T × 129, the kernel size of the first two convolutional layers is 1 × 5, the number of input and output channels is 16, the stride in the frequency dimension is 2, and the stride in the time dimension is 1. Therefore, the number of time frames T remains unchanged, resulting in an output feature size of 16 × T × 33 for the first two layers. The kernel size of the last three grouped convolutional layers is 3 × 3, and the dilation coefficients gradually increase to 1, 2, and 5 respectively to expand the receptive field. The number of input and output channels is 16, resulting in a depth feature with an output size of 16 × T × 33 for the last three layers. The decoder is connected to the encoder in a skip connection through a series of deconvolutional layers to gradually restore the time-frequency resolution of the features and reconstruct the enhanced speech signal; The aggregated grouped dual-path recurrent network combines a grouped dual-path recurrent network with a time-frequency attention mechanism. The time-frequency attention mechanism includes depthwise separable convolution, adaptive mean pooling, intra-band convolution, and inter-band convolution to generate attention weights, and assigns weights to each time-frequency position.

5. The lightweight single-channel speech enhancement method based on an improved convolutional recurrent network as described in claim 4, characterized in that, The adaptive average pooling is divided into average pooling along the time dimension and average pooling along the frequency dimension, generating two different axis vectors, the time axis and the frequency axis, respectively. and Then, broadcast addition processing is performed; the intra-band convolution and inter-band convolution respectively use 1 × and A × 1 convolutional kernel performs strip convolution modeling in both intra-band and inter-band directions to generate attention weights in the time-frequency domain. Then, a 3 × 3 depthwise convolution is used to further extract local details of the input features. The attention features are assigned to each time-frequency position using the Hadamard product. Finally, a multilayer perceptron module is used to perform nonlinear mapping on the time-frequency features.

6. The lightweight single-channel speech enhancement method based on an improved convolutional recurrent network as described in claim 5, characterized in that, The expressions for intra-band convolution and inter-band convolution are as follows: ； in, This represents large-kernel strip convolution. This indicates the kernel size of the band convolution. Represents the time axis vector and frequency axis vector The result after broadcast addition. This represents batch normalization after passing through the ReLU function. This represents the Sigmoid activation function; The expression for attention feature assignment is: ； in, This indicates a depthwise convolution with a kernel of 3 × 3. These are attention features obtained in the previous step. Represents the Adama product.

7. The lightweight single-channel speech enhancement method based on an improved convolutional recurrent network as described in claim 1, characterized in that, Step 4 specifically involves: preprocessing the noisy speech signal used for testing, extracting the real spectrum, imaginary spectrum, and amplitude spectrum features of the noisy speech signal, inputting them into the improved convolutional recurrent network after optimization training, obtaining the predicted complex ratio mask, multiplying the predicted complex ratio mask with the real spectrum and imaginary spectrum features to obtain the target complex spectrum, and reconstructing the speech signal through inverse short-time Fourier transform to complete the enhancement of the noisy speech signal.

8. The lightweight single-channel speech enhancement method based on an improved convolutional recurrent network as described in claim 1, characterized in that, The lightweight single-channel speech enhancement method based on the improved convolutional recurrent network further optimizes and enhances the perceptual quality of speech by combining perceptual contrast stretching technology, which significantly improves speech clarity and intelligibility. The perceptual contrast stretching technology utilizes the sensitivity of the human ear to specific frequency bands to redistribute the weights of specific frequency bands, and uses the PCS weight array to multiply the logarithmic amplitude spectrum bandwise to amplify the specific frequency bands.

Images