Voice model compression method, electronic equipment and storage medium

Abstract

The invention discloses a voice model compression method, electronic equipment and a storage medium. The method is specially designed for a large-scale sequence-to-sequence speech recognition model with an encoder-decoder architecture. According to the method, the decoder and the encoder are pruned in sequence, so that heavy back propagation calculation is avoided. According to the method, under the condition of no back propagation or retraining, the parameters of the Whisper-large model can be reduced by about 60%, and the performance of the model on various data sets is hardly influenced. Meanwhile, the method is suitable for a multi-language data set, and the pruned model keeps good robustness and generalization in the aspect of multi-language ability. According to the innovation, the threshold of large-scale model deployment is greatly reduced, and the method is easier to apply in a resource-limited environment.

Description

Technical Field

[0001] The present invention mainly relates to the field of artificial intelligence, and in particular to a large-scale sequence-to-sequence speech recognition model compression method based on mixed sparsity hierarchical pruning that only requires forward propagation, an electronic device and a storage medium. Background Art

[0002] In recent years, large-scale sequence-to-sequence (Seq2Seq) speech recognition models such as Whisper have attracted much attention due to their excellent speech recognition performance. These models have complex encoder-decoder architectures and large-scale parameters. By training on massive amounts of diverse speech data, they can cope with a variety of dialects, accents, and noisy environments. However, their large number of parameters brings high computational overhead, making it difficult to deploy such models on devices with limited computing resources, limiting their widespread application. To solve this problem, model compression technology has become a research hotspot, among which pruning methods are widely used because they can effectively reduce model parameters and inference overhead.

[0003] However, current pruning methods are mainly applied to medium-sized, encoder-only self-supervised speech models, and are difficult to directly apply to ultra-large-scale sequence-to-sequence speech recognition models with encoder-decoder architectures because these methods usually require a lot of memory to calculate the Hessian matrix or retrain on the original dataset, which brings additional resource requirements and technical difficulties. Summary of the invention

[0004] In view of the above-mentioned defects of the prior art, the technical problems to be solved by the present invention include:

[0005] How to design a speech model compression method to overcome the above technical problems.

[0006] To achieve the above-mentioned objectives, the present invention provides a speech recognition model compression method, which reduces the parameters of the speech recognition model through hierarchical pruning; optimizes the effect of the hierarchical pruning through a hybrid sparse allocation method based on forward propagation; the method of reducing the parameters of the speech recognition model through hierarchical pruning includes encoder pruning and decoder pruning; the method of optimizing the effect of the hierarchical pruning through a hybrid sparse allocation method based on forward propagation includes weight matrix grouping and reconstruction error control.

[0007] Furthermore, the decoder pruning is: pruning the decoder layer by layer through a calibration data set and generating calibration features for pruning; the encoder pruning is: after the decoder is pruned, the encoder part of the model is pruned layer by layer.

[0008] Furthermore, the decoder pruning is specifically as follows:

[0009] Use the transcriptions generated by the speech recognition model itself as a calibration dataset for the text modality to preserve the original semantic information and format of the model;

[0010] For the audio modality calibration dataset, the dense features output by the encoder are used as input;

[0011] The self-attention, cross-attention, and feed-forward networks in the decoder are pruned layer by layer, and the reconstruction error of each layer is recorded after each pruning.

[0012] Furthermore, the encoder pruning is specifically as follows:

[0013] Use speech waveforms from open source datasets as a calibration dataset for the encoder to obtain feature responses from real data;

[0014] Record the reconstruction error and control the error range to ensure that the model's audio feature extraction ability is not affected;

[0015] Each layer in the encoder is processed layer by layer, and only the reconstruction error is recorded through forward propagation, without the need for gradient calculation.

[0016] Furthermore, the weight matrix grouping is as follows: the weight matrices of each layer in the model are divided into multiple groups according to their functions, including the key, query and value projection matrices of the self-attention layer, the projection matrix of the cross-attention layer, and the feedforward network weights of each layer; the reconstruction error control is as follows: forward propagation calculation is performed on each group of weight matrices under different sparsity conditions to control the reconstruction error.

[0017] Furthermore, the weight matrix grouping is specifically as follows: each weight matrix in the encoder and the decoder is divided into eight groups; and a suitable sparsity is determined according to the importance of each weight matrix, so that the sparsity of different groups is flexibly adjustable.

[0018] Furthermore, the reconstruction error control is specifically as follows:

[0019] The reconstruction error under different sparsity conditions is calculated by forward propagation, which is used as an indicator to judge the degree of parameter preservation under the sparsity.

[0020] A reconstruction error threshold is set for each weight matrix, and the maximum sparsity configuration is selected from the candidate sparsities to ensure that the reconstruction error is within the threshold;

[0021] A unified threshold is set for each group of weight matrices to simplify the complexity of sparsity selection.

[0022] An electronic device, comprising:

[0023] at least one processor; and

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0026] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the aforementioned method.

[0027] A computer program product comprises a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0028] Compared with the prior art solutions, the technical effects of the present invention are:

[0029] The present invention proposes a hierarchical pruning method without backpropagation, which is designed for large-scale sequence-to-sequence speech recognition models with encoder-decoder architecture. This method avoids heavy backpropagation calculations by pruning the decoder and encoder in sequence. This method can reduce the parameters of the Whisper-large model by about 60% without backpropagation or retraining, and has almost no effect on the performance of the model on various data sets. At the same time, this method is applicable to multilingual data sets, and the pruned model maintains good robustness and generalization in multilingual capabilities. This innovation greatly reduces the threshold for large-scale model deployment, making it easier to apply in resource-constrained environments.

[0030] The present invention proposes a hybrid sparse allocation strategy that only requires forward propagation. By optimizing and retaining important parameters in the weight matrix, the sparse allocation is made more flexible and effective, further improving the compression efficiency of the model. This method can complete the sparse optimization of the model only through forward propagation, avoiding the additional retraining process. Experiments show that under the condition of 60% sparsity, this method performs better than traditional uniform sparse pruning on multiple test sets, especially when the sparsity is high, it can effectively reduce the decline in model performance.

[0031] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of hierarchical pruning of the present invention.

[0033] Figure 2 It is a schematic diagram of the mixed sparse allocation based on forward propagation of the present invention. DETAILED DESCRIPTION

[0034] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0035] The present invention discloses a speech recognition model compression method, which reduces the parameters of the speech recognition model through hierarchical pruning; optimizes the effect of the hierarchical pruning through a hybrid sparse allocation method based on forward propagation; the reducing the parameters of the speech recognition model through hierarchical pruning includes encoder pruning and decoder pruning; the optimizing the effect of the hierarchical pruning through the hybrid sparse allocation method based on forward propagation includes weight matrix grouping and reconstruction error control.

[0036] like Figure 1 As shown in the figure, the decoder pruning is to prune the decoder layer by layer through the calibration data set and generate calibration features for pruning. The decoder part is the part with large parameters in the Seq2Seq model and has a significant impact on the output performance. The encoder pruning is to prune the encoder part of the model layer by layer after the decoder is pruned. The encoder is responsible for extracting features from the input speech and is crucial to the recognition effect of the model.

[0037] The decoder pruning is specifically as follows:

[0038] Use the transcriptions generated by the speech recognition model itself as a calibration dataset for the text modality to preserve the model’s original semantic information and format, thus ensuring that the model retains important text information;

[0039] For the audio modality calibration dataset, the dense features output by the encoder are used as input to ensure that the model's ability to capture speech features is maintained during the pruning process;

[0040] The self-attention, cross-attention, and feedforward networks in the decoder are pruned layer by layer. The reconstruction error of each layer is recorded after each pruning to ensure that the error after pruning is within a reasonable range.

[0041] In traditional pruning methods, the decoder and encoder are usually not processed separately, and there are few practices for generating calibration features endogenously in the model. This embodiment avoids the performance degradation caused by inconsistent external data formats by adaptively generating a text modality calibration dataset that is consistent with the model format. This method effectively reduces the loss of model performance during the pruning process. After the decoder is pruned, the overall number of parameters of the model is significantly reduced and the inference speed is significantly improved. Experiments show that this separate pruning strategy can ensure that the model maintains a good recognition rate at high sparsity.

[0042] The encoder pruning is specifically as follows:

[0043] Use speech waveforms from open source datasets as a calibration dataset for the encoder to obtain feature responses from real data;

[0044] Record the reconstruction error and control the error range to ensure that the model's audio feature extraction ability is not affected;

[0045] Each layer in the encoder (including self-attention and feedforward networks) is processed layer by layer, and only the reconstruction error is recorded through forward propagation without gradient calculation.

[0046] This embodiment is the first to separate the encoder and decoder during the pruning process, and in particular uses real audio waveforms for the encoder calibration data, ensuring that the feature extraction of the model remains stable during the pruning process, which is suitable for refined pruning in large-scale Seq2Seq speech models. By pruning the encoder, the model parameters can be further reduced without significantly reducing the recognition effect, making the model more applicable in environments with limited computing resources.

[0047] like Figure 2 As shown, a hybrid sparse allocation strategy based on forward propagation is used to further optimize the pruning effect of the model. By allocating different sparsities, important weights in each layer can be retained, improving the performance of the model after pruning. The weight matrix grouping is as follows: the weight matrices of each layer in the model are divided into multiple groups according to their functions, including the key, query and value projection matrices of the self-attention layer, the projection matrix of the cross-attention layer, and the feedforward network weights of each layer; the reconstruction error control is as follows: forward propagation calculations are performed on each group of weight matrices under different sparsity conditions to control the reconstruction error.

[0048] The weight matrix grouping is specifically as follows: each weight matrix in the encoder and decoder is divided into eight groups:

[0049] (self_attn.k / q / v_proj), (self_attn.out_proj), (fc1), (fc2),

[0050] (self / encoder_attn.k / q / v_proj), (self / encoder_attn.out_proj), (fc1), (fc2); determine the appropriate sparsity according to the importance of each weight matrix, so that the sparsity of different groups can be flexibly adjusted.

[0051] The traditional uniform sparse pruning strategy ignores the importance differences of each weight matrix. This embodiment achieves differentiated processing by grouping weight matrices, and configures different sparsities for weight matrices of different functions, thereby improving overall performance. The sparse allocation after grouping makes the recognition effect of the model better than the traditional uniform sparse pruning method at the same pruning rate, especially when the sparsity is high, it can still maintain the stability of performance.

[0052] The reconstruction error control is specifically as follows:

[0053] The reconstruction error under different sparsity conditions is calculated by forward propagation, which is used as an indicator to judge the degree of parameter preservation under the sparsity.

[0054] A reconstruction error threshold is set for each weight matrix, and the maximum sparsity configuration is selected from the candidate sparsities to ensure that the reconstruction error is within the threshold;

[0055] A unified threshold is set for each group of weight matrices to simplify the complexity of sparsity selection.

[0056] The reconstruction error under different sparse conditions is calculated through forward propagation, which avoids the calculation of complex Hessian matrix and greatly improves the pruning efficiency without losing model accuracy.

[0057] Reconstruction error control enables the model to ensure output stability under high sparsity conditions, which is particularly suitable for resource-constrained application scenarios and meets actual production deployment needs.

[0058] The large-scale Seq2Seq speech model efficient pruning method proposed in the present invention has significant advantages in industrial applications, and is particularly suitable for deployment requirements in resource-constrained environments. The practicability of the present invention is analyzed in detail below from the perspectives of technical advantages, performance indicators and production implementation.

[0059] The large-scale Seq2Seq speech model efficient pruning method proposed in the present invention has significant advantages in industrial applications, and is particularly suitable for deployment requirements in resource-constrained environments. The practicability of the present invention is analyzed in detail below from the perspectives of technical advantages, performance indicators and production implementation.

[0060] Technical advantages

[0061] The present invention effectively overcomes the limitations of traditional large-scale speech model pruning methods in terms of computational overhead and complexity through the hierarchical pruning technology without retraining and the hybrid sparse allocation strategy based on forward propagation. Compared with the existing pruning methods that rely on backpropagation and a large amount of retraining, the present invention can directly achieve a parameter reduction of about 60% on a large-scale Seq2Seq speech model, while maintaining the high recognition rate and cross-domain and cross-language robustness of the model. The backpropagation-free feature of this solution lowers the hardware threshold of the device, making it possible to deploy advanced speech recognition technology on low-computing resource devices, and provides technical support for speech recognition in scenarios such as IoT devices and mobile applications.

[0062] Performance Indicators

[0063] The present invention achieves a parameter pruning rate of 60% on the Whisper-large model, and maintains the high recognition performance of the original model on multiple public data sets such as LibriSpeech, Common Voice, and Tedlium. The pruned model significantly reduces the number of parameters and memory usage without sacrificing accuracy, which helps to improve the response speed and stability of the speech recognition system. In addition, in multilingual speech recognition tasks, the pruned model can still maintain excellent multilingual recognition capabilities, further expanding its application potential in the global market. This combination of compression ratio and performance indicators makes the present invention have great market potential and application prospects in scenarios such as smart terminals, vehicle-mounted systems, and edge computing.

[0064] Production Implementation

[0065] The pruning method adopted by the present invention does not require retraining of the model, so it can effectively reduce the consumption of computing resources and the production cycle during the production implementation process. By utilizing the calibration data set to achieve efficient sparse configuration, this technology can complete the pruning operation on a standard GPU within 6 hours, thereby greatly reducing the time cost of the model compression process. In addition, the pruned model is compatible with the current deep learning framework and is easy to integrate into existing speech recognition products, shortening the conversion cycle from research and development to productization and improving the production efficiency of the enterprise. At the same time, the pruning method of the present invention supports combination with other model compression technologies such as knowledge distillation, further improving the lightweight degree of the model, which will provide application developers with more flexibility and expand the application boundaries of large-scale speech recognition models in a wide range of application scenarios.

[0066] In summary, the efficient pruning solution of the present invention has shown significant advantages in terms of technological innovation, performance indicators and production implementation, and has great potential for industrialization. Its convenient deployment process, significant model compression effect and strong robustness in multiple languages ​​and multiple fields not only meet the current industrial demand for lightweight speech recognition technology, but also lay a solid foundation for speech recognition applications in future smart devices and the Internet of Things.

[0067] The present invention discloses an electronic device, comprising:

[0068] at least one processor; and

[0069] a memory communicatively connected to the at least one processor; wherein,

[0070] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the aforementioned method.

[0071] The present invention discloses a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the aforementioned method.

[0072] The present invention discloses a computer program product, comprising a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0073] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0074] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0075] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0076] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A speech recognition model compression method, characterized in that: Reducing the parameters of a speech recognition model through layer pruning; optimizing the effect of the layer pruning through a hybrid sparse allocation method based on forward propagation; reducing the parameters of a speech recognition model through layer pruning includes encoder pruning and decoder pruning; optimizing the effect of the layer pruning through a hybrid sparse allocation method based on forward propagation includes weight matrix grouping and reconstruction error control.

2. The speech recognition model compression method according to claim 1, characterized in that: The decoder pruning is: pruning the decoder layer by layer through a calibration data set and generating calibration features for pruning; the encoder pruning is: after the decoder is pruned, the encoder part of the model is pruned layer by layer.

3. The compression method of the speech recognition model according to claim 2, characterized in that: The decoder pruning is specifically as follows: Use the transcriptions generated by the speech recognition model itself as a calibration dataset for the text modality to preserve the original semantic information and format of the model; For the audio modality calibration dataset, the dense features output by the encoder are used as input; The self-attention, cross-attention, and feed-forward networks in the decoder are pruned layer by layer, and the reconstruction error of each layer is recorded after each pruning.

4. The speech recognition model compression method according to claim 2, characterized in that: The encoder pruning is specifically as follows: Use speech waveforms from open source datasets as a calibration dataset for the encoder to obtain feature responses from real data; Record the reconstruction error and control the error range to ensure that the model's audio feature extraction ability is not affected; Each layer in the encoder is processed layer by layer, and only the reconstruction error is recorded through forward propagation, without the need for gradient calculation.

5. The speech recognition model compression method according to claim 1, characterized in that: The weight matrices are grouped as follows: the weight matrices of each layer in the model are divided into multiple groups according to their functions, including the key, query and value projection matrices of the self-attention layer, the projection matrix of the cross-attention layer, and the feedforward network weights of each layer; The reconstruction error control is as follows: forward propagation calculation is performed on each group of weight matrices under different sparsity conditions to control the reconstruction error.

6. The method for compressing a speech recognition model according to claim 5, characterized in that: The weight matrix grouping is specifically as follows: each weight matrix in the encoder and the decoder is divided into eight groups; and a suitable sparsity is determined according to the importance of each weight matrix, so that the sparsity of different groups is flexibly adjustable.

7. The method for compressing a speech recognition model according to claim 5, characterized in that: The reconstruction error control is specifically as follows: The reconstruction error under different sparsity conditions is calculated by forward propagation, which is used as an indicator to judge the degree of parameter preservation under the sparsity. A reconstruction error threshold is set for each weight matrix, and the maximum sparsity configuration is selected from the candidate sparsities to ensure that the reconstruction error is within the threshold; A unified threshold is set for each group of weight matrices to simplify the complexity of sparsity selection.

8. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images