Lightweight speech recognition method of unstructured pruning compression based on second-order information

Abstract

The invention discloses an unstructured pruning compression lightweight speech recognition method based on second-order information, and the method comprises the following steps: carrying out the sensitivity analysis based on a Hessian matrix on different components of each layer of a target speech recognition model, and carrying out the mixed sparsity pruning compression search; searching the model layer by layer from shallow to deep according to the obtained sparseness, calculating the contribution of each parameter to the model performance according to the input data and the values of the model parameters, cutting off the parameters with the bottom contribution, and updating the residual parameters; and generating a final pruning compression model by adopting cyclic pruning, pruning to a relatively low sparseness lower than the target sparseness, and then finely adjusting the model so as to correct the model parameters to the optimal performance. According to the method, unimportant parameters can be efficiently removed from the pre-trained large voice model on the premise of not sacrificing the performance, the calculation and storage requirements of the model are reduced, and the good performance is achieved under the high sparse condition.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a lightweight speech recognition method based on unstructured pruning compression of second-order information. Background Art

[0002] At present, with the accumulation of large-scale speech annotation data, the improvement of graphics processor computing speed, and the maturity of deep neural network technology, the effect of large-vocabulary continuous speech recognition has been significantly improved in recent years, becoming an important way of human-computer interaction. The rapid development of mobile terminals and the increasingly rich use scenarios of voice interaction without the Internet require the voice recognition system to reduce its demand for computing resources and adapt to mobile terminals with limited hardware computing resources. At the same time, the rich use needs also put forward higher requirements for the efficiency of speech recognition.

[0003] Traditional speech recognition systems use a variety of methods, including knowledge distillation, structured pruning, and unstructured pruning based on parameter size. Knowledge distillation uses the original model to guide the behavior of the compressed model, but it requires manual pre-setting of the network structure and consumes a lot of time and computing power to retrain the model; structured pruning prunes the neural network parameters according to the pruning pattern, which is usually a whole row or a whole column. Since the parameters need to be pruned in whole rows or columns, structured pruning has great limitations. Speech recognition systems that are usually pruned and compressed based on this method have poor performance and are difficult to meet usage needs; unstructured pruning based on parameter size judges the importance of the parameter based on its absolute value and removes parameters with small absolute values. However, the absolute value cannot accurately measure the importance of the parameter and ignores the influence of the input data. Speech recognition systems that are pruned and compressed based on this method have poor performance and are also difficult to meet usage needs.

[0004] Those skilled in the art need to develop a lightweight speech recognition method based on an unstructured pruned compressed neural network model of second-order information. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides a lightweight speech recognition method based on unstructured pruning compression of second-order information, characterized in that it includes the following steps:

[0006] S1: Get the sparsity of each layer of the target speech recognition model;

[0007] S2: Use the sparsity in step 1 to search the model layer by layer, calculate the contribution of each parameter to the model performance, and cut off the parameters with lower contribution;

[0008] S3: Compensate the above-mentioned trimmed parameter set;

[0009] The above steps S1 to S3 are repeated to obtain the final pruning compression model.

[0010] Furthermore, in step S1, a sensitivity analysis is performed on at least one layer in the model, and the sparsity under a certain compression ratio is obtained by combining a hybrid sparsity pruning compression search algorithm.

[0011] Furthermore, in step S3, the variable of the loss function that is clipped due to a parameter is estimated; at the same time, the selection of the least important parameter and the update of the remaining parameters are modeled as a conditional optimization problem, and the variable of the remaining parameters is obtained using Lagrange multipliers, thereby achieving compensation of the parameter set.

[0012] The present invention discloses a lightweight speech recognition method based on unstructured pruning compression of second-order information, which is characterized by comprising the following steps:

[0013] S1. Perform sensitivity analysis on different components of each layer of the target speech recognition model for mixed sparsity pruning and compression search;

[0014] S2. Search the model layer by layer from shallow to deep according to the sparsity obtained above, calculate the contribution of each parameter to the model performance according to the input data and the value of the model parameters, trim the parameters with lower contribution and update the remaining parameters;

[0015] S3. Use cyclic pruning to generate the final pruned compression model, prune to a lower sparsity than the target sparsity, and then fine-tune the model to correct the model parameters to the optimal performance.

[0016] Furthermore, the sensitivity analysis in step S1 is the sum of the diagonal elements of the corresponding area of ​​the Hessian matrix of the layer parameter relative to the loss function.

[0017] Furthermore, the variable in step S3 is obtained by Taylor expansion.

[0018] Furthermore, the sensitivity analysis in step S1 is based on the Hessian matrix.

[0019] Furthermore, the step S3 further includes a step S4:

[0020] S4. Repeat the above pruning and fine-tuning process until the model is pruned to the target sparsity and the model is fine-tuned to convergence.

[0021] Furthermore, the size of the model after compression in step S4 is reduced to 35% of the original size.

[0022] Furthermore, at least one layer in step S1 performs sensitivity analysis on each layer.

[0023] The present invention proposes a lightweight speech recognition method based on unstructured pruning compression of second-order information. The present invention can efficiently remove up to 80% of unimportant parameters from a pre-trained large speech model without sacrificing performance, thereby significantly reducing the computation and storage requirements of the model. Compared with the model training parameter conditions in the prior art, the present invention achieves a performance improvement of up to 30%, and has good performance under high sparsity conditions.

[0024] The present invention uses widely used deep learning toolkits such as PyTorch and Hugging Face to achieve easy integration into existing speech recognition systems. The present invention helps to reduce the demand for computing resources of speech recognition models, allowing high-quality speech recognition models to be deployed in resource-constrained environments;

[0025] The present invention is applicable to a variety of model architectures and to a variety of pre-trained speech models including wav2vec2-base, large and wavlm-base-plus. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a lightweight speech recognition method based on unstructured pruning and compression of second-order information provided in an embodiment of the present invention.

[0027] Figure 2 A flowchart of a specific implementation method of a lightweight speech recognition method based on an unstructured pruned compressed neural network model of second-order information provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0029] In the drawings, components with the same structure are indicated by the same numerical labels, and components with similar structures or functions are indicated by similar numerical labels. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component.

[0030] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0031] The lightweight speech recognition method based on unstructured pruning compression of second-order information implemented in the present invention mainly includes the following steps:

[0032] S1: Get the sparsity of each layer of the target speech recognition model;

[0033] The acquisition method is to perform a sensitivity analysis based on the Hessian matrix on each layer in the model, and combine the mixed sparsity pruning compression search algorithm to obtain the sparsity under a certain compression ratio;

[0034] S2: Use the sparsity in step 1 to search the model layer by layer, calculate the contribution of each parameter to the model performance, and cut off the parameters with lower contribution;

[0035] S3: Compensate the above-mentioned trimmed parameter set;

[0036] The above steps S1 to S3 are repeated to obtain the final pruning compression model.

[0037] Wherein, in step S3, the specific compensation method is:

[0038] Use Taylor expansion to estimate the loss function for a variable that is clipped by a parameter, i.e. set to zero;

[0039] At the same time, the selection of the least important parameter and the updating of the remaining parameters are modeled as a conditional optimization problem, and the Lagrange multiplier is used to obtain the variable of the remaining parameters to achieve the compensation of the parameter set.

[0040] like Figure 2 As shown, a flowchart of a lightweight speech recognition method based on an unstructured pruned compressed neural network model of second-order information provided by an embodiment of the present invention is shown. Specifically, the method mainly includes the following steps:

[0041] S1. First, the different components of each layer of the target speech recognition model are used for mixed sparsity pruning compression search; the mixed sparsity pruning compression search algorithm obtains the optimal sparsity allocation under an appropriate compression ratio.

[0042] Among them, sensitivity analysis can measure the importance of a certain layer. The specific calculation method is the sum of the diagonal elements of the corresponding area of ​​the Hessian matrix of the layer parameters relative to the loss function. The higher the sensitivity, the more parameters the layer retains and the smaller the sparsity.

[0043] S2. Search the model layer by layer from shallow to deep according to the sparsity obtained in step S1, calculate the contribution of each parameter to the model performance based on the input data and the value of the model parameters, prune the parameters with lower contribution and update the remaining parameters to compensate for the performance degradation caused by pruning.

[0044] The loss function of the model is modeled as a multivariate function with the dependent variable as the parameter and the input, and the Taylor expansion is used to estimate the loss function when one parameter is clipped, i.e., set to zero.

[0045] At the same time, the selection of the least important parameter and the updating of the remaining parameters are modeled as a conditional optimization problem, and the variables of the remaining parameters are obtained using Lagrange multipliers.

[0046] S3. Use cyclic pruning to generate the final pruned compression model. Cyclic pruning first prunes the model to a lower sparsity than the target sparsity, and then fine-tunes the model to correct the model parameters to the position of optimal performance.

[0047] S4. Repeat the above pruning and fine-tuning process until the model is pruned to the target sparsity and the model is fine-tuned to convergence.

[0048] Thus, the model for speech recognition pruned using the above method can be compressed to 35% of its original size while maintaining the same performance. Compared with the prior art, the performance of the model for speech recognition pruned using the above method has a 30% performance advantage.

[0049] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A lightweight speech recognition method based on unstructured pruning compression of second-order information, characterized in that: The steps include: S1: Get the sparsity of each layer of the target speech recognition model; S2: Use the sparsity in step 1 to search the model layer by layer, calculate the contribution of each parameter to the model performance, and cut off the parameters with lower contribution; S3: Compensate the above-mentioned trimmed parameter set; The above steps S1 to S3 are repeated to obtain the final pruning compression model.

2. The lightweight speech recognition method based on unstructured pruning compression of second-order information as described in claim 1 is characterized in that: In the step S1, a sensitivity analysis is performed on at least one layer in the model, and the sparsity under a certain compression ratio is obtained by combining a hybrid sparsity pruning compression search algorithm.

3. The lightweight speech recognition method based on unstructured pruning compression of second-order information as described in claim 1, characterized in that: In step S3, the variable of the loss function that is clipped due to a parameter is estimated; at the same time, the selection of the least important parameter and the update of the remaining parameters are modeled as a conditional optimization problem, and the variable of the remaining parameters is obtained using Lagrange multipliers, thereby achieving compensation of the parameter set.

4. A lightweight speech recognition method based on unstructured pruning compression of second-order information, characterized in that: The steps include: S1. Perform sensitivity analysis on different components of each layer of the target speech recognition model for mixed sparsity pruning and compression search; S2. Search the model layer by layer from shallow to deep according to the sparsity obtained above, calculate the contribution of each parameter to the model performance according to the input data and the value of the model parameters, trim the parameters with lower contribution and update the remaining parameters; S3. Use cyclic pruning to generate the final pruned compression model, prune to a lower sparsity than the target sparsity, and then fine-tune the model to correct the model parameters to the optimal performance.

5. The lightweight speech recognition method based on unstructured pruning compression of second-order information as described in claim 4, characterized in that: The sensitivity analysis in step S1 is the sum of the diagonal elements of the corresponding area of ​​the Hessian matrix of the layer parameter relative to the loss function.

6. The lightweight speech recognition method based on unstructured pruning compression of second-order information as described in claim 3, characterized in that: The variable in step S3 is obtained by Taylor expansion.

7. The lightweight speech recognition method based on unstructured pruning compression of second-order information as described in claim 1 or 4, characterized in that: The sensitivity analysis in step S1 is based on the Hessian matrix.

8. The lightweight speech recognition method based on unstructured pruning compression of second-order information as described in claim 4 is characterized in that: The step S3 further includes a step S4: S4. Repeat the above pruning and fine-tuning process until the model is pruned to the target sparsity and the model is fine-tuned to convergence.

9. The lightweight speech recognition method based on unstructured pruning compression of second-order information as described in claim 8, characterized in that: The size of the model after compression in step S4 is reduced to 35% of the original size.

10. The lightweight speech recognition method based on unstructured pruning compression of second-order information as described in claim 2, characterized in that: At least one layer in step S1 performs sensitivity analysis on each layer.

Images