Lightweight speech enhancement method based on frame resampling and sub-band pruning

Abstract

The invention discloses a lightweight speech enhancement method based on frame resampling and sub-band pruning, and relates to the field of automatic speech recognition. According to the method provided by the invention, a hierarchical frame resampling (LWS) and sub-band pruning (SBP) method is provided, so that the speech recognition performance in a complex noise environment can be ensured while the efficiency of a speech enhancement model is improved. The method has high practicability, can be flexibly adjusted according to the requirements of specific application scenes, and is particularly suitable for being deployed in end-side equipment and voice recognition scenes with limited real-time reasoning delay and computing resources in the industry.

Description

Technical Field

[0001] The present invention relates to the field of automatic speech recognition, and in particular to a lightweight speech enhancement method based on frame resampling and subband pruning. Background Art

[0002] In recent years, Automatic Speech Recognition (ASR) technology has made significant progress, but achieving robustness in noisy environments remains a long-standing challenge. Speech enhancement models are designed to improve the intelligibility and quality of speech in noisy environments, and are often used in the preprocessing front end of automatic speech recognition systems to enhance their noise resistance. Although the speech enhancement (SE) front end is widely used in noise reduction as a preprocessing step of ASR, it often adopts a more complex deep learning architecture and allows the neural network to process information in all frequency bands. Although this improves the noise reduction effect, it brings significant computational overhead, limiting its application in real-time speech recognition.

[0003] Reducing computational complexity to meet the needs of real-time applications is an important direction in speech enhancement research. Existing methods include applying techniques such as quantization and pruning to existing architectures to reduce model size and computational complexity. Other studies focus on designing more efficient models specifically for low-computational overhead inference. Some recent studies attempt to optimize the processing effect of high-performance speech enhancement models as the front end of ASR systems by modifying their internal modules, but these methods also have certain defects and limitations. Preliminary experimental results show that these lightweight speech enhancement models generally fail to significantly improve ASR performance when deployed as ASR preprocessing front ends. This challenge is often attributed to the speed-distortion effect, that is, the model simplifies the signal in order to achieve real-time processing, thereby introducing distortion and affecting the subsequent ASR effect. Although these methods can maintain the enhancement quality to a certain extent while reducing computational overhead, their impact on ASR performance has not been fully verified and evaluated. In addition, these lightweight models often face poor performance when dealing with complex noise scenes, and it is difficult to achieve satisfactory speech enhancement and recognition accuracy while ensuring computational efficiency.

[0004] Therefore, technicians in this field are committed to developing a lightweight speech enhancement method based on frame resampling and subband pruning. A hierarchical frame resampling and subband pruning method is proposed, which can improve the efficiency of the speech enhancement model while ensuring the speech recognition performance in a complex noise environment. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is to reduce the computational complexity of the speech enhancement model while maintaining the performance of speech recognition in a noisy environment.

[0006] To achieve the above object, the present invention provides a lightweight speech enhancement method based on frame resampling and subband pruning, including subband pruning, which gradually excludes frequency bands with low information content at each layer.

[0007] Furthermore, the calculation of the high frequency subband is skipped.

[0008] Further, it includes radical strategies and incremental strategies.

[0009] Furthermore, the aggressive strategy directly skips the highest several sub-bands starting from the first temporal RNN layer.

[0010] Furthermore, the progressive strategy gradually skips an additional high-frequency sub-band each time a new temporal RNN layer is entered.

[0011] Furthermore, the standard RNN layer is replaced with a grouped RNN layer.

[0012] Furthermore, hierarchical frame resampling is performed.

[0013] Furthermore, the input is downsampled within each layer and then upsampled via residual connections.

[0014] Further, the hierarchical frame resampling includes:

[0015] LWS-ALL: resampling operations are performed simultaneously on the time and frequency band RNNs of all layers;

[0016] LWS-SYNC: resamples both time and frequency band RNNs in odd layers, while keeping the original features unchanged in even layers;

[0017] LWS-ASYNC: Asynchronous resampling strategy, odd-numbered layers resample the time dimension, and even-numbered layers resample the frequency band dimension.

[0018] Furthermore, model training includes the following steps:

[0019] Step 1: Frequency band division: divide the input audio into predefined sub-bands;

[0020] Step 2, hierarchical frame resampling: When the sub-band features pass through each layer, the frame resampling module is used to downsample the features to reduce redundant information in the time dimension; after each layer is processed, it is restored by upsampling to ensure that key information is not lost; residual connections are used to compensate for possible information loss during upsampling;

[0021] Step 3: Stepwise sub-band pruning: In the frequency domain, the model evaluates the frequency bands of the input features layer by layer and removes the frequency bands with less information;

[0022] Step 4: Use grouped RNN instead of the traditional RNN layer;

[0023] Step 5: The enhanced speech signal is input into the ASR system for recognition.

[0024] The speech enhancement model introduces a large additional overhead, which is not conducive to real-time speech recognition applications. The present invention prunes subbands and gradually eliminates low-information frequency bands at each layer.

[0025] Lightweight speech enhancement models often suffer from loss of speech recognition performance. The present invention resamples the layers of the frames. The input is downsampled in each layer and then upsampled through residual connections to reduce information loss.

[0026] The layer-wise frame resampling (LWS) and sub-band pruning (SBP) methods proposed in this technical solution can improve the efficiency of the speech enhancement model while ensuring speech recognition performance in complex noise environments. These methods are highly practical and can be flexibly adjusted according to the needs of specific application scenarios. They are particularly suitable for deployment in industrial end-side devices and speech recognition scenarios with limited real-time inference delays and computing resources.

[0027] 1. Flexible deployment of client devices and real-time voice applications

[0028] LWS reduces the computational complexity of each layer while retaining important speech features by applying a frame resampling strategy. Compared with traditional global resampling methods, LWS can reduce unnecessary computational overhead without significantly affecting the model's speech recognition accuracy. Through residual connection technology, other layers can still access complete speech features, ensuring a balance between computational efficiency and information retention. This solution can be based on the computational requirements of real-time applications, such as when running on an end-side device, by adjusting the resampling factor to reduce inference latency.

[0029] SBP further reduces the amount of computation in the frequency domain dimension by selectively skipping the calculation of high-frequency sub-bands. It is also suitable for deployment on end-side devices with limited computing resources. It can be flexibly adjusted in different computing environments, maintaining high performance while significantly reducing computing costs.

[0030] 2.Flexible adjustment to suit industrial needs

[0031] LWS and SBP provide a variety of optimization strategies (such as LWS-ALL, LWS-ASYNC, SBP-A, SBP-P), which can be flexibly adjusted according to the industry's specific requirements for inference latency, computing resource constraints, and hardware performance. For example, LWS can accurately control the inference time by setting different frame resampling ratios and the number of layers to meet the low latency requirements in real-time applications; SBP minimizes computing overhead while maintaining speech recognition performance by pruning high-frequency bands layer by layer. The two methods are also easy to combine with each other and other lightweight measures to work together on the speech enhancement model.

[0032] In summary, this solution can significantly improve the computational efficiency of the speech enhancement model, meet the industry's actual needs for low-latency, low-power and high-performance speech processing, and has broad commercial application prospects.

[0033] Compared with the prior art, the present invention has the following obvious substantial features and significant advantages:

[0034] 1. The present invention significantly reduces the computational overhead of the speech enhancement front end and can also maintain relatively robust speech recognition performance.

[0035] 2. The present invention significantly reduces the computational overhead of the speech enhancement front end while maintaining the ASR performance.

[0036] 3. The present invention significantly improves the computational efficiency of the speech enhancement model, meets the industry's actual needs for low-latency, low-power consumption and high-performance speech processing, and has broad commercial application prospects.

[0037] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of inter-layer frame resampling of a preferred embodiment of the present invention;

[0039] Figure 2 It is a schematic diagram of sub-band pruning according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0041] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0042] This paper proposes two optimization methods to reduce the computational complexity of the speech enhancement model while maintaining the performance of speech recognition in noisy environments. The two methods are Layer-Wise Frame Resampling (LWS) and Sub-Band Pruning (SBP).

[0043] Resampling (including downsampling and upsampling) is a widely used technique to reduce the computational complexity of speech enhancement (SE) models. An intuitive and simple idea is to apply downsampling before the input enters the network, and upsample after the network processes the features, that is, the pre-downsampling and post-upsampling (PPS) method. Although this method reduces the number of frames and subbands that the network needs to process, thereby reducing computational complexity, it comes at the cost of a large loss of information. In fact, a considerable portion of the input features are never processed by the neural network, resulting in the loss of important details.

[0044] Therefore, the hierarchical frame resampling (LWS) is further designed, such as Figure 1 As shown, resampling is selectively applied in specific layers to ensure that the output feature shape of each selected layer is consistent with the input. Unlike PPS, which causes information loss to the entire network, LWS limits the impact of this information loss to the selected layers. The present invention further uses residual connections so that other layers can still access the complete, unprocessed input data features. This method strikes a balance between computational efficiency and retention of important information.

[0045] Three LWS strategies are proposed, as follows:

[0046] 1.LWS-ALL: Resampling operations are performed simultaneously on the time and frequency band RNNs (recurrent neural networks) of all layers. This strategy performs downsampling before each group of time and frequency band RNNs and upsampling afterward, ensuring the consistency of feature shapes while reducing the computational burden in both time and frequency band dimensions.

[0047] 2.LWS-SYNC: Resamples the time and frequency band RNNs simultaneously in odd layers, while keeping the original features unchanged in even layers, thereby retaining more information.

[0048] 3. LWS-ASYNC: Asynchronous resampling strategy. Odd-numbered layers resample the time dimension, and even-numbered layers resample the frequency band dimension. This strategy makes the information loss in the entire network more evenly distributed in the time and frequency band dimensions.

[0049] The resampling methods for time and frequency band RNNs are the same, differing only in the dimensions in which they are applied. Taking the resampling of time RNN as an example, the input frame is divided into blocks of size S along the time dimension T. The mean of each block is calculated to obtain the output Z'∈R N×K×T' ,in During upsampling, each feature in Z' is repeated S times along the time dimension, restoring the original temporal resolution. For band RNNs, a similar process is performed along the sub-band dimension N.

[0050] like Figure 2 As shown in the figure, subband pruning (SBP) further reduces the computational complexity of the model by skipping the computation of the high-frequency subband. Since the main information of speech is concentrated in the low-frequency and mid-frequency bands, skipping the computation of the high-frequency band will not significantly affect the performance of speech recognition. SBP is also divided into two strategies:

[0051] 1. Aggressive strategy (SBP-A): This strategy starts from the first time RNN layer and directly skips the highest L sub-bands to reduce the computational burden as early as possible. The parameter L can be set by the user and can be adjusted according to the actual performance or computational overhead requirements.

[0052] 2. Progressive strategy (SBP-P): In this strategy, each time a new temporal RNN layer is entered, an additional high-frequency sub-band is gradually skipped.

[0053] In addition, the standard RNN layer is replaced with a grouped RNN (GR) layer to reduce computational overhead, and a grouping of size 4 is used to improve computational efficiency. Each GR layer divides the input sequence and hidden state into two disjoint groups and performs recursive learning within each group to capture the dependencies within the group. In order to restore the correlation between groups, a reordering layer is introduced between consecutive recursive layers and time steps.

[0054] Experimental data: Experiments were conducted on English and Mandarin datasets. The English experiment used the tr_05_simu_1_ch subset of the CHIME-4 corpus combined with the noise samples of the DNS dataset to generate training data at SNR levels from -5dB to 20dB. The official validation set and test set of CHIME-4 (dt_05_real and et_05_real1chtrack) were used for testing. The DNS dataset contains recordings of more than 10,000 speakers with a sampling rate of 16kHz. 100 hours of noisy speech were generated according to the ESPnet recipe, including synthetic segments with and without reverberation.

[0055] The Mandarin experiment used clean speech from the train and dev sets of the AISHELL-1 corpus, combined with the noise from the DNS dataset, to generate new training and development sets at SNR levels from -5dB to 20dB. The evaluation datasets were from two real acoustic environments: a kitchen and a shopping mall, recorded using a dual-microphone phone at the top and bottom. The reverberation of the shopping mall environment is significantly greater than that of the kitchen. Each environment has three test sets, each containing 1.5 hours of data, evenly distributed in the two environments.

[0056] Training configuration: All experiments were conducted using the ESPnet tool. The speech enhancement (SE) frontend used the V4 band split configuration in

[25] to generate 23 subbands. The SE frontend was trained for 54K steps using the Adam optimizer with multi-resolution STFT amplitude loss and SI-SNR loss as the loss functions. In the experiments on the DNS and CHIME-4 datasets, the untuned Whisper model was used as the ASR backend. In the Mandarin experiments, the official released Paraformer model with excellent performance was used.

[0057] Decoding: A single-channel SE front end may cause ASR performance degradation in a noisy environment, so the observation addition (OA) technique is used. This method adds the observed signal to the enhanced signal in proportion to the formula S OA =ω OA .S noisy +(1-ω OA ).S enh , where ω OA ∈[0,1]. Tune ω on the validation set OA The language model is not used when decoding, but text normalization is applied to the decoding results of the Whisper model.

[0058] The model training process can be summarized as follows:

[0059] (1) Band division: Divide the input audio into predefined sub-bands

[0060] (2) Hierarchical frame resampling: When the sub-band features pass through each layer, the frame resampling module is used to downsample the features to reduce redundant information in the time dimension. After each layer is processed, it is restored through upsampling to ensure that key information is not lost. Residual connections are used to compensate for the information loss that may be caused during the upsampling process.

[0061] (3) Stepwise sub-band pruning: In the frequency domain, the model evaluates the frequency bands of the input features layer by layer and removes those with less information.

[0062] (4) Use grouped RNN to replace the traditional RNN layer.

[0063] (5) The speech features processed as above are input into the ASR system as enhanced speech signals for recognition.

[0064] Table 1

[0065] Word error rate (%) of three sizes of Whisper models on the DNS challenge test set

[0066]

[0067]

[0068] Table 2 Word error rates (%) of three sizes of Whisper models on the CHiME4 single-channel real test set

[0069]

[0070] Table 3 Word error rates (%) of three sizes of Whisper models on the internal Mandarin test set

[0071]

[0072] Table 4 Method description table

[0073]

[0074]

[0075] Among them: The number in ASYNC() / SYNC() represents the resampling factor.

[0076] Experimental results analysis:

[0077] 1. Comparison of resampling strategies

[0078] The results in Table I show that LWS-ALL and PPS have the same computational cost, where LWS-ALL(4) represents the application of the LWS-ALL strategy with a resampling factor of 4. However, both methods show a significant performance degradation compared to the original SE model. This degradation may be due to the overly aggressive resampling strategy, which weakens the model’s ability to capture key information in the full spectrum.

[0079] In contrast, LWS-SYNC and LWS-ASYNC outperform the aforementioned methods (as shown in Table I) by resampling only selected target layers, while maintaining comparable performance to the original SE front-end. Notably, LWS-ASYNC outperforms LWS-SYNC among all ASR back-ends on both the unaural and reverberation test sets. This suggests that alternating resampling in the time and frequency band dimensions more effectively preserves key information, thereby maintaining the performance of the model.

[0080] 2. Comparison of sub-band pruning

[0081] The proposed subband pruning strategy is validated using the Whisper model on the English and Chinese test sets. To further evaluate the proposed method and its variants, LWS-ASYNC (resampling size is 16) is combined with the SBP-A and SBP-P strategies. As shown in Tables I, II, and III, SBP-P effectively limits the performance degradation to no more than 1% relative on all test sets. These results show that SBP-P is able to maintain comparable performance to the original SE model while significantly reducing the computational overhead. Unlike SBP-A, which adopts a more aggressive pruning strategy, SBP-P achieves performance preservation by progressively reducing high-frequency subbands layer by layer. This strategy not only preserves the performance, but also improves on the ASR results of SBP-A, with a small difference in GMAC.

[0082] To further reduce computational cost, LWS-ASYNC, SBP-P and GR methods are combined. The GR layer replaces the standard RNN layer in the time and frequency band RNN, greatly improving computational efficiency while having minimal impact on ASR performance. Despite the significant reduction in computational overhead, the system performance remains robust, with only a slight drop on a few test sets.

[0083] By comparing the experimental results of different systems, the final system uses frame resampling, subband pruning and grouped RNN, which significantly reduces the computational cost while still providing excellent ASR performance in noisy scenes. Compared with the original model, the computational cost is greatly reduced and the performance is maintained in noisy environments.

[0084] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A lightweight speech enhancement method based on frame resampling and subband pruning, characterized in that: Including sub-band pruning, gradually eliminating low-information frequency bands at each layer.

2. The lightweight speech enhancement method based on frame resampling and subband pruning as claimed in claim 1, characterized in that: Skip calculation of high frequency subbands.

3. The lightweight speech enhancement method based on frame resampling and subband pruning as claimed in claim 1, characterized in that: Includes aggressive strategies and incremental strategies.

4. The lightweight speech enhancement method based on frame resampling and subband pruning as claimed in claim 3, characterized in that: The radical strategy starts from the first temporal RNN layer and directly skips the highest several sub-bands.

5. The lightweight speech enhancement method based on frame resampling and subband pruning as claimed in claim 3, characterized in that: The progressive strategy gradually skips an additional high-frequency sub-band each time a new temporal RNN layer is entered.

6. The lightweight speech enhancement method based on frame resampling and subband pruning as claimed in claim 1, characterized in that: Replace the standard RNN layer with a grouped RNN layer.

7. The lightweight speech enhancement method based on frame resampling and subband pruning as claimed in claim 1, characterized in that: Hierarchical frame resampling.

8. The lightweight speech enhancement method based on frame resampling and subband pruning as claimed in claim 7, characterized in that: The input is downsampled within each layer and then upsampled via residual connections.

9. The lightweight speech enhancement method based on frame resampling and subband pruning as claimed in claim 7, characterized in that: The hierarchical frame resampling comprises: LWS-ALL: resampling operations are performed simultaneously on all layers of the time and frequency band RNN; LWS-SYNC: resamples both the time and frequency band RNNs in odd layers, while keeping the original features unchanged in even layers; LWS-ASYNC: Asynchronous resampling strategy, odd-numbered layers resample the time dimension, and even-numbered layers resample the frequency band dimension.

10. The lightweight speech enhancement method based on frame resampling and subband pruning according to claim 1, characterized in that: Model training consists of the following steps: Step 1: Frequency band division: divide the input audio into predefined sub-bands; Step 2, hierarchical frame resampling: When the sub-band features pass through each layer, the frame resampling module is used to downsample the features to reduce redundant information in the time dimension; after each layer is processed, it is restored by upsampling to ensure that key information is not lost; residual connections are used to compensate for possible information loss during upsampling; Step 3: Stepwise sub-band pruning: In the frequency domain, the model evaluates the frequency bands of the input features layer by layer and removes the frequency bands with less information; Step 4: Use grouped RNN instead of the traditional RNN layer; Step 5: The enhanced speech signal is input into the ASR system for recognition.

Images