A lightweight speech enhancement method based on frame resampling and subband pruning

Abstract

The application discloses a lightweight speech enhancement method based on frame resampling and sub-band pruning, and relates to the field of automatic speech recognition.The layer-wise frame resampling (LWS) and sub-band pruning (SBP) method proposed by the application can improve the efficiency of a speech enhancement model while ensuring the speech recognition performance in a complex noise environment.The application has high practicability, can be flexibly adjusted according to the requirements of specific application scenarios, and is especially suitable for deployment in the industrial field of real-time inference delay, limited computing resource end-side devices and speech recognition scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic speech recognition, and in particular to a lightweight speech enhancement method based on frame resampling and subband pruning. BACKGROUND

[0002] In recent years, automatic speech recognition (ASR) technology has made significant progress, but achieving robustness in noisy environments remains a long-standing challenge. Speech enhancement models aim to improve the intelligibility and quality of speech in noisy environments, and are often used as a preprocessing front-end for automatic speech recognition systems to enhance their noise-robust performance. Although the speech enhancement (SE) front-end is widely used as a preprocessing step for ASR to reduce noise, it often uses a complex deep learning architecture and requires the neural network to process information from all frequency bands. While this improves the noise reduction effect, it significantly increases the computational overhead, limiting its application in real-time speech recognition.

[0003] Reducing computational complexity to meet the needs of real-time applications is an important direction in speech enhancement research. Existing methods include applying techniques such as quantization and pruning in existing architectures to reduce model size and computational complexity. Other research focuses on designing more efficient models specifically for low-computational-overhead inference, and some recent research attempts to optimize the processing effect of high-performance speech enhancement models as ASR system front-ends by modifying their internal modules, but these methods also have certain defects and limitations. Preliminary experimental results show that these lightweight speech enhancement models often fail to significantly improve the performance of ASR when deployed as a preprocessing front-end. This challenge is often attributed to the speed-distortion effect, i.e., in order to achieve real-time processing, the model simplifies the signal, introducing distortion and affecting the subsequent ASR effect. Although these methods can maintain the enhancement quality to some extent while reducing computational overhead, their impact on ASR performance has not been fully verified and evaluated. In addition, these lightweight models often face performance issues when dealing with complex noise scenarios, making it difficult to achieve satisfactory speech enhancement effects and recognition accuracy while ensuring computational efficiency.

[0004] Therefore, those skilled in the art are committed to developing a lightweight speech enhancement method based on frame resampling and subband pruning. The proposed hierarchical frame resampling and subband pruning method can improve the efficiency of the speech enhancement model while ensuring the performance of speech recognition in complex noise environments. SUMMARY

[0005] In view of the above defects of the prior art, the technical problem to be solved by the present application is to reduce the computational complexity of the speech enhancement model while maintaining the performance of speech recognition in noisy environments.

[0006] To achieve the above object, the application provides a light-weight speech enhancement method based on frame resampling and sub-band pruning, which comprises sub-band pruning and step-by-step elimination of low-information-frequency bands in each layer.

[0007] Further, the calculation of high-frequency sub-bands is skipped.

[0008] Further, the method comprises aggressive strategy and gradual strategy.

[0009] Further, the aggressive strategy directly skips a number of highest sub-bands from the first time RNN layer.

[0010] Further, the gradual strategy step-by-step skips an additional high-frequency sub-band when entering a new time RNN layer.

[0011] Further, the standard RNN layer is replaced by a grouped RNN layer.

[0012] Further, hierarchical frame resampling is performed.

[0013] Further, the input is down-sampled in each layer, and then up-sampled through residual connection.

[0014] Further, the hierarchical frame resampling comprises:

[0015] LWS-ALL: simultaneously performing resampling operation on time and frequency band RNN in all layers;

[0016] LWS-SYNC: resampling time and frequency band RNN in odd layers, and keeping original features unchanged in even layers;

[0017] LWS-ASYNC: asynchronous resampling strategy, resampling time dimension in odd layers, and resampling frequency band dimension in even layers.

[0018] Further, the model training comprises the following steps:

[0019] Step 1, frequency band division: dividing the input audio into predefined sub-bands;

[0020] Step 2, hierarchical frame resampling: when the sub-band features pass through each layer, the frame resampling module is used to down-sample the features to reduce the redundant information in the time dimension; after the processing of each layer is completed, the features are recovered through up-sampling to ensure that the key information is not lost; residual connection is used to compensate for the information loss caused in the up-sampling process;

[0021] Step 3, step-by-step sub-band pruning: in the frequency domain, the model layer by layer evaluates the frequency bands of the input features, and eliminates the frequency bands with less information.

[0022] Step 4, replace the traditional RNN layer with a group RNN;

[0023] Step 5, the enhanced speech signal is input into the ASR system for recognition.

[0024] The speech enhancement model introduces a large additional overhead, which is not conducive to real-time speech recognition applications. The present application prunes sub-bands. Low information bands are gradually excluded at each layer.

[0025] Lightweight speech enhancement models often have a loss of speech recognition performance. The present application performs layer-wise frame resampling. The input is down-sampled within each layer, and then up-sampled through a residual connection to reduce information loss.

[0026] The layer-wise frame resampling (LWS) and sub-band pruning (SBP) methods proposed in this technical solution can improve the efficiency of the speech enhancement model while ensuring the speech recognition performance in complex noise environments. These methods are highly practical and can be flexibly adjusted according to the needs of specific application scenarios, especially suitable for deployment in industrial real-time inference delay, limited computing resource end-side devices and speech recognition scenarios.

[0027] 1. Flexible deployment for end-side devices and real-time speech applications

[0028] LWS reduces the computational complexity of each layer by applying frame resampling strategies while preserving important speech features. Compared to traditional global resampling methods, LWS can reduce unnecessary computational overhead without significantly affecting the speech recognition accuracy of the model. Through residual connection technology, other layers can still access complete speech features, ensuring a balance between computational efficiency and information preservation. This solution can adjust the resampling factor to reduce inference delay when running on end-side devices, according to the computational requirements of real-time applications.

[0029] SBP further reduces the computational load in the frequency domain dimension by selectively skipping the calculation of high-frequency sub-bands, making it suitable for deployment on computing resource-limited end-side devices. It can be flexibly adjusted in different computing environments, maintaining high performance while significantly reducing computational cost.

[0030] 2. Flexible adjustment to meet industrial needs

[0031] The LWS and the SBP provide various optimization strategies (such as LWS-ALL, LWS-ASYNC, SBP-A, and SBP-P), which can be flexibly adjusted according to specific requirements of the industry for inference delay, calculation resource limitation, and hardware performance. For example, the LWS can accurately control the inference time by setting different frame resampling ratios and the number of effective layers, thereby meeting the low-delay requirement in real-time applications; and the SBP can maximally reduce the calculation cost while maintaining the speech recognition performance by layer-by-layer pruning of high-frequency bands. The two methods are also easy to be combined with each other and other lightweight measures to jointly act on the speech enhancement model.

[0032] In summary, the scheme can significantly improve the calculation efficiency of the speech enhancement model, meet the actual requirements of the industry for low-delay, low-power-consumption, and high-performance speech processing, and has a wide commercial application prospect.

[0033] Compared with the prior art, the present application has the following obvious and substantial characteristics and advantages:

[0034] 1. The present application greatly reduces the calculation cost of the speech enhancement front end while maintaining relatively robust speech recognition performance.

[0035] 2. The present application greatly reduces the calculation cost of the speech enhancement front end while maintaining the ASR performance.

[0036] 3. The present application significantly improves the calculation efficiency of the speech enhancement model, meets the actual requirements of the industry for low-delay, low-power-consumption, and high-performance speech processing, and has a wide commercial application prospect.

[0037] The concept, specific structure, and technical effects of the present application will be further described below with reference to the accompanying drawings, so as to fully understand the purpose, features, and effects of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a schematic diagram of interlayer frame resampling of a preferred embodiment of the present application;

[0039] Figure 2 is a schematic diagram of subband pruning of a preferred embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical content of the present application will be more clearly and easily understood by introducing a plurality of preferred embodiments of the present application with reference to the accompanying drawings of the specification. The present application can be embodied in many different forms, and the protection scope of the present application is not limited to the embodiments mentioned in the text.

[0041] In the drawings, the same components have the same reference numerals, and components similar in structure or function have similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the size and thickness of each component are not limited in the present application. In order to make the drawing clearer, the thickness of some components is appropriately exaggerated in some places in the drawing.

[0042] The present application proposes two optimization methods aimed at reducing the computational complexity of speech enhancement models while maintaining the performance of speech recognition in noisy environments. The two methods are Layer-Wise Frame Resampling (LWS) and Sub-Band Pruning (SBP).

[0043] Resampling (including downsampling and upsampling) is a widely used technique to reduce the computational complexity of speech enhancement (SE) models. One intuitive and simple idea is to apply downsampling before the input enters the network and upsampling after the network has processed the features, i.e., using the pre-downsampling and post-upsampling (PPS) method. Although this method reduces the number of frames and subbands that the network needs to process, thereby reducing the computational complexity, the cost is a large amount of information loss. In fact, a significant portion of the input features is never processed by the neural network, resulting in the loss of important details.

[0044] Therefore, the Layer-Wise Frame Resampling (LWS) is further designed, as shown in Figure 1 selectively applies resampling in specific layers, ensuring that the output feature shape of each selected layer remains consistent with the input. Unlike PPS, which causes information loss throughout the entire network, LWS limits the impact of this information loss to the selected layers. The present application further enables other layers to still access complete, unprocessed input data features through residual connections. This approach strikes a balance between computational efficiency and the preservation of important information.

[0045] Three LWS strategies are proposed, as follows:

[0046] 1. LWS-ALL: Resampling is performed simultaneously on both time and frequency band RNNs (Recurrent Neural Networks) in all layers. This strategy performs downsampling before each set of time and frequency band RNNs and upsampling afterward, ensuring consistency in feature shape while reducing computational burden in both time and frequency dimensions.

[0047] 2. LWS-SYNC: Resampling is performed on both time and frequency band RNNs in odd layers, while the original features are kept unchanged in even layers, thereby preserving more information.

[0048] 3. LWS-ASYNC: Asynchronous resampling strategy. Odd layers resample the time dimension, and even layers resample the frequency band dimension. This strategy makes the information loss in the whole network more evenly distributed in the time and frequency band dimensions.

[0049] The resampling method for time and frequency band RNNs is the same, only differing in the dimension applied. Take the resampling of time RNNs as an example, the input frame is divided into blocks of size S along the time dimension T. The mean of each block is computed, resulting in an output Z' e R N×K×T' where In the up-sampling process, each feature in Z' is repeated S times along the time dimension, recovering the original time resolution. For frequency band RNNs, a similar process is applied along the sub-band dimension N.

[0050] As shown in FIG. 2, sub-band pruning (SBP) further reduces the computational cost of the model by skipping the computation of high-frequency sub-bands. Since the main information of speech is concentrated in the low and mid-frequency bands, skipping the computation of high-frequency bands does not significantly affect the performance of speech recognition. SBP is also divided into two strategies: Figure 2

[0051] 1. Aggressive strategy (SBP-A): This strategy starts from the first time RNN layer and directly skips the highest L sub-bands, reducing the computational burden as early as possible. The parameter L is user-settable and can be adjusted according to the actual performance or computational overhead requirements.

[0052] 2. Progressive strategy (SBP-P): In this strategy, an additional high-frequency sub-band is skipped gradually as a new time RNN layer is entered.

[0053] In addition, the standard RNN layer is replaced by a grouped RNN (GR) layer to reduce the computational overhead and use a group size of 4 to improve computational efficiency. Each GR layer divides the input sequence and hidden state into two disjoint groups and performs recursive learning within each group to capture the dependencies within the group. To recover the correlation between groups, a rearrangement layer is introduced between consecutive recursive layers and time steps.

[0054] Experimental data: Experiments were conducted on English and Mandarin datasets. The English experiment used the tr_05_simu_1_ch subset of the CHIME-4 corpus combined with noise samples from the DNS dataset to generate training data at SNR levels from -5 dB to 20 dB. The official validation and test sets (dt_05_real and et_05_real1chtrack) of CHIME-4 were used for testing. The DNS dataset contains recordings of over 10,000 speakers at a sampling rate of 16 kHz. According to the ESPnet recipe, 100 hours of noisy speech were generated, including reverberation and non-reverberation synthesized segments.​

[0055] The Mandarin experiments use clean speech from the train and dev sets of AISHELL-1 corpus combined with noise from the DNS dataset to generate new training and development sets at SNR levels from -5dB to 20dB. The evaluation datasets come from two real acoustic environments: kitchen and shopping mall, recorded by a dual-microphone phone at the top and bottom respectively. The reverberation of the shopping mall environment is significantly larger than the kitchen. There are three test sets for each environment, each containing 1.5 hours of data, evenly distributed in the two environments.

[0056] Training configuration: All experiments are conducted using ESPnet tools. The V4 band-division configuration in

[25] is adopted for the speech enhancement (SE) front-end, which generates 23 subbands in total. The SE front-end is trained for 54K steps using the Adam optimizer with a loss function of multi-resolution STFT magnitude loss and SI-SNR loss. In the experiments on DNS and CHIME-4 datasets, the un-finetuned Whisper model is used as the ASR backend. In the Mandarin experiments, the officially released Paraformer model is adopted, which performs the best.

[0057] Decoding: The single-channel SE front-end can lead to degraded ASR performance in noisy environments, therefore the observation addition (OA) technique is used. This method adds the observed signal to the enhanced signal in proportion, with the formula S OA =ω OA .S noisy +(1-ω OA ).S enh , where ω OA ∈[0,1]. The ω OA value is tuned on the validation set respectively. No language model is used during decoding, but text normalization is applied to the decoding results of the Whisper model.

[0058] The training process of the model can be summarized as follows:

[0059] (1) Band division: The input audio is divided into predefined subbands

[0060] (2) Hierarchical frame resampling: When the subband features pass through each layer, a frame resampling module is used to downsample the features to reduce redundant information in the time dimension. After the processing of each layer is completed, it is recovered through upsampling to ensure that key information is not lost. Residual connection is used to compensate for the information loss that may be caused during upsampling.

[0061] (3) Step-by-step subband pruning: In the frequency domain, the model will evaluate the frequency bands of the input features layer by layer and eliminate those with less information.

[0062] (4) Grouped RNN is used instead of traditional RNN layers.

[0063] (5) The speech features processed as described above are input to the ASR system as enhanced speech signals for recognition.

[0064] Table 1

[0065] Word error rates (%) of three sizes of Whisper models on the DNS challenge test set

[0066]

[0067]

[0068] Table 2 Word error rates (%) of three sizes of Whisper models on the CHiME4 single-channel real test set

[0069]

[0070] Table 3 Word error rates (%) of three sizes of Whisper models on the internal Mandarin test set

[0071]

[0072] Table 4 Method description table

[0073]

[0074]

[0075] where the number in ASYNC() / SYNC() indicates the resampling factor.

[0076] Analysis of experimental results:

[0077] 1. Comparison of resampling strategies

[0078] The results of Table I show that LWS-ALL and PPS are identical in terms of computational cost, with LWS-ALL(4) denoting the application of the LWS-ALL strategy with a resampling factor of 4. However, both methods exhibit a clear performance drop compared to the original SE model. This drop can be due to the overly aggressive resampling strategy, which weakens the model’s ability to capture key information in the full spectrum.

[0079] In contrast, LWS-SYNC and LWS-ASYNC outperform the aforementioned methods by resampling only selected target layers (as shown in Table I), while maintaining comparable performance to the original SE frontend. It is worth noting that LWS-ASYNC outperforms LWS-SYNC in all ASR backends on both reverberation-free and reverberation test sets. This suggests that alternating resampling in time and frequency dimensions more effectively preserves the key information, thus maintaining the performance of the model.

[0080] 2. Subband pruning

[0081] The proposed subband pruning strategy is validated on English and Chinese test sets using the Whisper model. To further evaluate the proposed method and its variants, LWS-ASYNC (with resampling size of 16) is combined with SBP-A and SBP-P strategies. As shown in Tables I, II, and III, SBP-P effectively limits the performance degradation to no more than 1% relatively on all test sets. These results show that SBP-P can maintain comparable performance to the original SE model while significantly reducing the computational overhead. Unlike SBP-A, which employs a more aggressive pruning strategy, SBP-P preserves performance by progressively reducing high-frequency subbands layer by layer. This strategy not only preserves performance but also improves ASR results over SBP-A with little difference in GMAC.

[0082] To further reduce the computational cost, LWS-ASYNC, SBP-P, and GR methods are combined. The GR layer replaces the standard RNN layer in the time and frequency RNNs, significantly improving computational efficiency while having minimal impact on ASR performance. Despite the significant reduction in computational overhead, the system performance remains robust, with only slight degradation on a few test sets.

[0083] Through the comparison of experimental results of different systems, the final system adopts frame resampling, subband pruning, and grouped RNN, significantly reducing the computational cost while still providing excellent ASR performance in noisy scenarios. Compared to the original model, the computational cost is greatly reduced, and the performance is maintained in a noisy environment.

[0084] The above describes in detail the preferred embodiments of the present application. It should be understood that those skilled in the art can make many modifications and changes to the present application without creative labor based on the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning, or limited experiments by those skilled in the art based on the concept of the present application and the prior art shall be within the protection scope determined by the claims.

Claims

1. A lightweight speech enhancement method based on frame resampling and subband pruning, characterized in that, Sub-band pruning, progressively excluding low-information bands; Aggressive and progressive strategies; The aggressive strategy directly skips the highest number of sub-bands starting from the first time RNN layer; The progressive strategy progressively skips an additional high-frequency sub-band every time a new time RNN layer is entered; Hierarchical frame resampling; Down-sampling the input within each layer, followed by up-sampling through residual connections; The hierarchical frame resampling includes: LWS-ALL: Resampling operations are performed simultaneously on both time and band RNNs across all layers; LWS-SYNC: Resampling is performed simultaneously on both time and band RNNs in odd layers, while the original features are kept unchanged in even layers; LWS-ASYNC: Asynchronous resampling strategy, odd layers resample the time dimension, even layers resample the band dimension; Model training includes the following steps: Step 1, Band division: Divide the input audio into predefined sub-bands; Step 2, Hierarchical frame resampling: When the sub-band features pass through each layer, use the frame resampling module to down-sample the features, reducing redundant information in the time dimension. After the processing of each layer is completed, up-sampling is performed to recover the key information that may be lost during the down-sampling process. Residual connections are used to compensate for the information loss caused by up-sampling; Step 3, Progressive sub-band pruning: In the frequency domain, the model evaluates the frequency bands of the input features layer by layer and eliminates bands with low information content; Step 4, Use group RNN instead of traditional RNN layers; Step 5, The enhanced speech signal is input into the ASR system for recognition.

2. The lightweight speech enhancement method based on frame resampling and subband pruning of claim 1, wherein, Skip the calculation of high-frequency sub-bands.

3. The lightweight speech enhancement method based on frame resampling and subband pruning of claim 1, wherein, Replace the standard RNN layer with a group RNN layer.

Images