A speech model compression method, electronic device and storage medium

Abstract

This invention discloses a speech model compression method, electronic device, and storage medium. It is specifically designed for large-scale sequence-to-sequence speech recognition models with an encoder-decoder architecture. This method avoids cumbersome backpropagation computation by sequentially pruning the decoder and encoder. It can reduce the parameters of a Whisper-large model by approximately 60% without backpropagation or retraining, with almost no impact on the model's performance on various datasets. Furthermore, this method is applicable to multilingual datasets, and the pruned model maintains good robustness and generalization capabilities across multiple languages. This innovation significantly lowers the barrier to large-scale model deployment, making it easier to apply in resource-constrained environments.

Description

Technical Field

[0001] This invention mainly relates to the field of artificial intelligence, and in particular to a method, electronic device and storage medium for compressing large-scale sequence-to-sequence speech recognition models based on hybrid sparseness hierarchical pruning that only requires forward propagation. Background Technology

[0002] In recent years, large-scale sequence-to-sequence (Seq2Seq) speech recognition models such as Whisper have attracted much attention due to their superior speech recognition performance. These models possess complex encoder-decoder architectures and large-scale parameters, and through training on massive amounts of diverse speech data, they can handle various dialects, accents, and noisy environments. However, their large number of parameters leads to high computational overhead, making it difficult to deploy such models on devices with limited computing resources, thus limiting their widespread application. To address this issue, model compression techniques have become a research hotspot, among which pruning methods are widely used because they can effectively reduce model parameters and inference overhead.

[0003] However, current pruning methods are mainly applied to medium-sized self-supervised speech models containing only encoders, and are difficult to apply directly to ultra-large-scale sequence-to-sequence speech recognition models with encoder-decoder architectures. This is because these methods usually require a large amount of memory to compute the Hessian matrix or to retrain on the original dataset, which brings additional resource requirements and technical challenges. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention includes:

[0005] How to design a speech model compression method to overcome the above-mentioned technical problems?

[0006] To achieve the above objectives, this invention provides a speech recognition model compression method that reduces the parameters of the speech recognition model through hierarchical pruning; optimizes the effect of the hierarchical pruning through a forward propagation-based hybrid sparse allocation method; the reduction of the speech recognition model parameters through hierarchical pruning includes encoder pruning and decoder pruning; the optimization of the effect of the hierarchical pruning through the forward propagation-based hybrid sparse allocation method includes weight matrix grouping and reconstruction error control.

[0007] Furthermore, the decoder pruning involves pruning the decoder layer by layer using a calibration dataset and generating calibration features for pruning; the encoder pruning involves pruning the encoder part of the model layer by layer after pruning the decoder.

[0008] Furthermore, the decoder pruning specifically involves:

[0009] The transcribed text generated by the speech recognition model itself is used as a calibration dataset for the text modality in order to preserve the original semantic information and format of the model;

[0010] For the audio modality calibration dataset, dense features from the encoder output are used as input;

[0011] Prune the self-attention, cross-attention, and feedforward networks in the decoder layer by layer, and record the reconstruction error of each layer after each pruning.

[0012] Furthermore, the encoder pruning specifically involves:

[0013] Speech waveforms from an open-source dataset are used as the encoder's calibration dataset to obtain the feature responses of real data.

[0014] Record the reconstruction error and control the error range to ensure that the model's audio feature extraction capability is not affected;

[0015] Each layer in the encoder is processed layer by layer, and only the reconstruction error is recorded through forward propagation, without the need for gradient calculation.

[0016] Furthermore, the weight matrix is ​​grouped as follows: the weight matrices of each layer in the model are divided into multiple groups according to their functions, including the key, query and value projection matrices of the self-attention layer, the projection matrix of the cross-attention layer, and the feedforward network weights of each layer; the reconstruction error control is as follows: forward propagation calculation is performed on each group of weight matrices under different sparsity conditions to control the reconstruction error.

[0017] Furthermore, the weight matrix grouping specifically involves dividing each weight matrix in the encoder and decoder into eight groups; determining a suitable sparsity based on the importance of each weight matrix, so that the sparsity of different groups is flexibly adjustable.

[0018] Furthermore, the reconstruction error control specifically includes:

[0019] The reconstruction error under different sparsity conditions is calculated using forward propagation, and this is used as an indicator to judge the degree of parameter preservation under that sparsity.

[0020] For each weight matrix, a reconstruction error threshold is set, and the maximum sparsity configuration is selected from the candidate sparsity to ensure that the reconstruction error is within the threshold.

[0021] A uniform threshold is set for each group of weight matrices, thereby simplifying the complexity of sparsity selection.

[0022] An electronic device, comprising:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0026] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the aforementioned method.

[0027] A computer program product includes a computer program that, when executed by a processor, implements the aforementioned method.

[0028] Compared with existing technical solutions, the technical advantages of the present invention are as follows:

[0029] This invention proposes a hierarchical pruning method without backpropagation, specifically designed for large-scale sequence-to-sequence speech recognition models with encoder-decoder architectures. This method avoids the cumbersome backpropagation computation by pruning the decoder and encoder sequentially. It can reduce the parameters of Whisper-large models by approximately 60% without backpropagation or retraining, with almost no impact on the model's performance on various datasets. Furthermore, this method is applicable to multilingual datasets, and the pruned model maintains good robustness and generalization capabilities across multiple languages. This innovation significantly lowers the barrier to large-scale model deployment, making it easier to apply in resource-constrained environments.

[0030] This invention proposes a hybrid sparse allocation strategy that requires only forward propagation. By optimizing and preserving important parameters in the weight matrix, it makes sparse allocation more flexible and effective, further improving the model's compression efficiency. This method can complete the model's sparsity optimization through forward propagation alone, avoiding the additional retraining process. Experiments show that, under 60% sparsity conditions, this method outperforms traditional uniform sparse pruning on multiple test sets, especially effectively reducing the degradation of model performance when sparsity is high.

[0031] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the hierarchical pruning method of the present invention.

[0033] Figure 2 This is a schematic diagram of the hybrid sparse allocation based on forward propagation of the present invention. Detailed Implementation

[0034] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0035] This invention discloses a speech recognition model compression method, which reduces the parameters of the speech recognition model through hierarchical pruning; optimizes the effect of the hierarchical pruning through a hybrid sparse allocation method based on forward propagation; the reduction of the speech recognition model parameters through hierarchical pruning includes encoder pruning and decoder pruning; the optimization of the effect of the hierarchical pruning through the hybrid sparse allocation method based on forward propagation includes weight matrix grouping and reconstruction error control.

[0036] like Figure 1 As shown, the decoder pruning involves pruning the decoder layer by layer using a calibration dataset and generating calibration features for pruning. The decoder part is the part of the Seq2Seq model with a large number of parameters and a significant impact on output performance. The encoder pruning involves pruning the encoder part of the model layer by layer after the decoder pruning. The encoder is responsible for extracting features from the input speech and is crucial to the model's recognition performance.

[0037] The decoder pruning specifically involves:

[0038] Using the transcribed text generated by the speech recognition model itself as a calibration dataset for the text modality preserves the original semantic information and format of the model, thus ensuring the model's retention of important text information.

[0039] For the audio modality calibration dataset, dense features from the encoder output are used as input to ensure that the model's speech feature capture capability is maintained during pruning.

[0040] Prune the self-attention, cross-attention, and feedforward networks in the decoder layer by layer. Record the reconstruction error of each layer after each pruning to ensure that the error after pruning is within a reasonable range.

[0041] In traditional pruning methods, the decoder and encoder are typically not processed separately, and the generation of intrinsic calibration features is rare. This embodiment avoids performance degradation caused by inconsistent external data formats by adaptively generating a text modality calibration dataset consistent with the model format. This method effectively reduces the performance loss during pruning. After decoder pruning, the overall number of model parameters is significantly reduced, and inference speed is significantly improved. Experiments show that this separate pruning strategy ensures that the model maintains good recognition rates even with high sparsity.

[0042] The encoder pruning specifically involves:

[0043] Speech waveforms from an open-source dataset are used as the encoder's calibration dataset to obtain the feature responses of real data.

[0044] Record the reconstruction error and control the error range to ensure that the model's audio feature extraction capability is not affected;

[0045] Each layer in the encoder (including self-attention and feedforward networks) is processed layer by layer, and only the reconstruction error is recorded through forward propagation, without the need for gradient calculation.

[0046] This embodiment is the first to separate the encoder and decoder during the pruning process. In particular, it uses real audio waveforms for the encoder calibration data, ensuring that the feature extraction of the model remains stable during pruning. This makes it suitable for fine-grained pruning in large-scale Seq2Seq speech models. By pruning the encoder, the model parameters can be further reduced without significantly reducing the recognition performance, making the model more suitable for environments with limited computing resources.

[0047] like Figure 2 As shown, a hybrid sparse allocation strategy based on forward propagation is used to further optimize the pruning effect of the model. By allocating different sparsities, important weights in each layer can be retained, improving the performance of the pruned model. The weight matrix is ​​grouped as follows: the weight matrices of each layer in the model are divided into multiple groups according to their functions, including the key, query, and value projection matrices of the self-attention layer, the projection matrix of the cross-attention layer, and the feedforward network weights of each layer; the reconstruction error control is achieved by performing forward propagation calculations on each group of weight matrices under different sparsity conditions to control the reconstruction error.

[0048] The weight matrix grouping is specifically as follows: the weight matrices in the encoder and decoder are divided into eight groups:

[0049] (self_attn.k / q / v_proj), (self_attn.out_proj), (fc1), (fc2),

[0050] (self / encoder_attn.k / q / v_proj), (self / encoder_attn.out_proj), (fc1), (fc2); determine the appropriate sparsity based on the importance of each weight matrix, so that the sparsity of different groups can be flexibly adjusted.

[0051] Traditional uniform sparse pruning strategies neglect the differences in importance among weight matrices. This embodiment achieves differentiated processing by grouping weight matrices, configuring different sparsities for weight matrices with different functions, thereby improving overall performance. The sparse allocation after grouping enables the model to outperform traditional uniform sparse pruning methods at the same pruning rate, and it can maintain performance stability even with high sparsity.

[0052] The reconstruction error control specifically refers to:

[0053] The reconstruction error under different sparsity conditions is calculated using forward propagation, and this is used as an indicator to judge the degree of parameter preservation under that sparsity.

[0054] For each weight matrix, a reconstruction error threshold is set, and the maximum sparsity configuration is selected from the candidate sparsity to ensure that the reconstruction error is within the threshold.

[0055] A uniform threshold is set for each group of weight matrices, thereby simplifying the complexity of sparsity selection.

[0056] By calculating the reconstruction error under different sparsity conditions through forward propagation, the computation of the complex Hessian matrix is ​​avoided, and the pruning efficiency is significantly improved without sacrificing model accuracy.

[0057] Reconstruction error control ensures that the model can maintain output stability even under high sparsity conditions, making it particularly suitable for resource-constrained application scenarios and meeting the needs of actual production deployment.

[0058] The efficient pruning method for large-scale Seq2Seq speech models proposed in this invention has significant advantages in industrial applications, especially suitable for deployment needs in resource-constrained environments. The following is a detailed analysis of the practicality of this invention from the perspectives of technical advantages, performance indicators, and production implementation.

[0059] The efficient pruning method for large-scale Seq2Seq speech models proposed in this invention has significant advantages in industrial applications, especially suitable for deployment needs in resource-constrained environments. The following is a detailed analysis of the practicality of this invention from the perspectives of technical advantages, performance indicators, and production implementation.

[0060] Technological advantages

[0061] This invention effectively overcomes the limitations of traditional large-scale speech model pruning methods in terms of computational overhead and complexity by employing a retrain-free hierarchical pruning technique and a hybrid sparse allocation strategy based on forward propagation. Compared to existing pruning methods that rely on backpropagation and extensive retraining, this invention can directly reduce parameters by approximately 60% on large-scale Seq2Seq speech models while maintaining high recognition rates and robustness across domains and languages. The backpropagation-free nature of this scheme lowers the hardware barrier for devices, making it possible to deploy advanced speech recognition technology on devices with low computing resources, and providing technical support for speech recognition in IoT devices, mobile applications, and other scenarios.

[0062] Performance indicators

[0063] This invention achieves a 60% parameter pruning rate on the Whisper-large model while maintaining the original model's high recognition performance on multiple public datasets such as LibriSpeech, Common Voice, and Tedlium. The pruned model significantly reduces the number of parameters and memory usage without sacrificing accuracy, contributing to improved response speed and stability of speech recognition systems. Furthermore, in multilingual speech recognition tasks, the pruned model still maintains excellent multilingual recognition capabilities, further expanding its application potential in the global market. This balance between high compression ratio and excellent performance makes this invention highly promising for applications in smart terminals, in-vehicle systems, and edge computing.

[0064] Production Implementation

[0065] The pruning method employed in this invention eliminates the need for model retraining, effectively reducing computational resource consumption and production cycle time during implementation. By utilizing a calibration dataset to achieve efficient sparse configuration, this technology can complete the pruning operation within 6 hours on a standard GPU, significantly reducing the time cost of model compression. Furthermore, the pruned model is compatible with current deep learning frameworks and easily integrated into existing speech recognition products, shortening the transformation cycle from R&D to productization and improving enterprise productivity. Simultaneously, the pruning method of this invention supports combination with other model compression techniques such as knowledge distillation, further enhancing the model's lightweight nature. This provides application developers with greater flexibility, expanding the application boundaries of large-scale speech recognition models across a wide range of application scenarios.

[0066] In summary, the efficient pruning scheme of this invention demonstrates significant advantages in technological innovation, performance indicators, and production implementation, exhibiting great potential for industrialization. Its convenient deployment process, remarkable model compression effect, and strong robustness to multiple languages ​​and domains not only meet the current industrial demand for lightweight speech recognition technology but also lay a solid foundation for future speech recognition applications in smart devices and the Internet of Things.

[0067] This invention discloses an electronic device, comprising:

[0068] At least one processor; and

[0069] A memory communicatively connected to the at least one processor; wherein,

[0070] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0071] The present invention discloses a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the aforementioned method.

[0072] This invention discloses a computer program product, including a computer program that implements the aforementioned method when executed by a processor.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for compressing a speech recognition model, characterized in that, The parameters of the speech recognition model are reduced through hierarchical pruning; the effect of the hierarchical pruning is optimized through a hybrid sparse allocation method based on forward propagation; the reduction of parameters of the speech recognition model through hierarchical pruning includes encoder pruning and decoder pruning; the optimization of the effect of the hierarchical pruning through a hybrid sparse allocation method based on forward propagation includes weight matrix grouping and reconstruction error control. The decoder pruning involves pruning the decoder layer by layer using a calibration dataset and generating calibration features for pruning; the encoder pruning involves pruning the encoder part of the model layer by layer after pruning the decoder. The decoder pruning specifically involves: The transcribed text generated by the speech recognition model itself is used as a calibration dataset for the text modality in order to preserve the original semantic information and format of the model; For the audio modality calibration dataset, dense features from the encoder output are used as input; Prune the self-attention, cross-attention, and feedforward networks in the decoder layer by layer, and record the reconstruction error of each layer after each pruning. The weight matrix is ​​grouped as follows: the weight matrices of each layer in the model are divided into multiple groups according to their functions, including the key, query and value projection matrices of the self-attention layer, the projection matrix of the cross-attention layer, and the feedforward network weights of each layer; the reconstruction error control is performed by performing forward propagation calculations on each group of weight matrices under different sparsity conditions to control the reconstruction error. The reconstruction error control specifically refers to: The reconstruction error under different sparsity conditions is calculated using forward propagation, and this is used as an indicator to judge the degree of parameter preservation under that sparsity. For each weight matrix, a reconstruction error threshold is set, and the maximum sparsity configuration is selected from the candidate sparsity to ensure that the reconstruction error is within the threshold. A uniform threshold is set for each group of weight matrices, thereby simplifying the complexity of sparsity selection.

2. The speech recognition model compression method according to claim 1, characterized in that, The encoder pruning specifically involves: Speech waveforms from an open-source dataset are used as the encoder's calibration dataset to obtain the feature responses of real data. Record the reconstruction error and control the error range to ensure that the model's audio feature extraction capability is not affected; Each layer in the encoder is processed layer by layer, and only the reconstruction error is recorded through forward propagation, without the need for gradient calculation.

3. The speech recognition model compression method according to claim 1, characterized in that, The weight matrix grouping is specifically as follows: the weight matrices in the encoder and decoder are divided into eight groups; the appropriate sparsity is determined according to the importance of each weight matrix, so that the sparsity of different groups can be flexibly adjusted.

4. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3.

5. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-3.

6. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-3.

Images