Lightweight speech recognition method based on unstructured pruning compression of second-order information

Abstract

The application discloses a lightweight speech recognition method based on unstructured pruning compression of second-order information, which comprises the following steps: performing sensitivity analysis on each layer of different components of a target speech recognition model based on a Hessian matrix, and performing mixed sparsity pruning compression search; searching the model layer by layer from shallow to deep according to the obtained sparsity, calculating the contribution of each parameter to the model performance according to the values of input data and model parameters, cutting off the parameters with low contribution, and updating the remaining parameters; adopting cyclic pruning to generate a final pruning compression model, pruning to a lower sparsity lower than the target sparsity, and then fine-tuning the model to correct the model parameters to the optimal performance. The application can efficiently remove unimportant parameters from a pre-trained large speech model without sacrificing performance, reduces the calculation and storage requirements of the model, and has good performance under high sparsity conditions.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a lightweight speech recognition method based on unstructured pruning compression of second-order information. Background Technology

[0002] Currently, the accumulation of large-scale speech annotation data, the improvement of graphics processing unit (GPU) processing speed, and the maturity of deep neural network technology have led to significant improvements in the performance of large-vocabulary continuous speech recognition in recent years, making it an important method of human-computer interaction. The rapid development of mobile terminals and the increasing variety of voice interaction scenarios without internet access necessitate reducing the computational resource requirements of speech recognition systems to adapt to mobile terminals with limited hardware computing resources. At the same time, the diverse usage demands also place higher demands on the efficiency of speech recognition.

[0003] Traditional speech recognition systems employ various methods, including knowledge distillation, structured pruning, and parameter-size-based unstructured pruning. Knowledge distillation uses the original model to guide the behavior of the compressed model, but it requires manual pre-setting of the network structure and consumes significant time and computational resources for model retraining. Structured pruning trims neural network parameters according to pruning patterns, typically entire rows or columns. Because it requires trimming entire rows or columns, structured pruning has significant limitations, and speech recognition systems based on this method generally perform poorly and fail to meet usage requirements. Parameter-size-based unstructured pruning judges the importance of parameters based on their absolute values, removing parameters with small absolute values. However, absolute values ​​do not accurately measure parameter importance and ignore the influence of input data. Speech recognition systems based on this method also perform poorly and fail to meet usage requirements.

[0004] Those skilled in the art need to develop a lightweight speech recognition method based on an unstructured pruning and compressed neural network model with second-order information. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention provides a lightweight speech recognition method based on unstructured pruning compression of second-order information, characterized by comprising the following steps:

[0006] S1: Obtain the sparsity of each layer of the target speech recognition model;

[0007] S2: Use the sparsity from step 1 to search the model layer by layer, calculate the contribution of each parameter to the model performance, and prune the parameters with lower contributions.

[0008] S3: Compensate the parameter set after the above trimming;

[0009] The steps S1-S3 are cycled to obtain a final pruned compression model.

[0010] Further, the sensitivity analysis of at least one layer in the model in the step S1 is combined with a mixed sparsity pruning compression search algorithm to obtain sparsity under a certain compression ratio.

[0011] Further, in the step S3, the variable of the loss function is estimated due to one parameter being pruned; meanwhile, selecting the least important parameter and updating the remaining parameters are modeled as a conditional optimization problem, and the variable of the remaining parameters is obtained by using a Lagrange multiplier, so as to realize compensation of the parameter set.

[0012] The application discloses a lightweight speech recognition method based on unstructured pruning compression of second-order information.

[0013] S1. Sensitivity analysis is performed on different components of each layer of the target speech recognition model, and is used for mixed sparsity pruning compression search;

[0014] S2. According to the obtained sparsity, the model is searched layer by layer from shallow to deep, the contribution of each parameter to the model performance is calculated according to the input data and the value of the model parameter, the parameter with lower contribution is pruned, and the remaining parameters are updated;

[0015] S3. A final pruned compression model is generated by using cyclic pruning, pruning to a lower sparsity lower than the target sparsity, and then fine-tuning the model, so as to modify the model parameters to optimal performance.

[0016] Further, the sensitivity analysis in the step S1 is the sum of the diagonal elements of the corresponding region of the Hessian matrix of the loss function with respect to the parameters of the layer.

[0017] Further, the variable in the step S3 is obtained by Taylor expansion.

[0018] Further, the sensitivity analysis in the step S1 is based on the Hessian matrix.

[0019] Further, the step S3 further includes a step S4:

[0020] S4. The pruning and fine-tuning process is repeated until the model is pruned to the target sparsity and the model is fine-tuned to convergence.

[0021] Further, the size of the model compressed in the step S4 is reduced to 35% of the original size.

[0022] Further, at least one layer in the step S1 is subjected to sensitivity analysis.

[0023] The application provides a lightweight speech recognition method based on unstructured pruning compression of second-order information, which can efficiently remove up to 80% of unimportant parameters from a pre-trained large speech model without sacrificing performance, thereby significantly reducing the computational and storage requirements of the model, and has a performance improvement of up to 30% compared to the model training parameters in the prior art, and has good performance under high sparsity conditions.

[0024] The application uses widely used deep learning toolkits such as PyTorch and Hugging Face, and is easy to integrate into existing speech recognition systems, which helps to reduce the computational resource requirements of speech recognition models and enables deployment of high-quality speech recognition models in resource-constrained environments.

[0025] The application is suitable for various model architectures and various pre-trained speech models including wav2vec2-base, large and wavlm-base-plus. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 A flowchart of a lightweight speech recognition method based on unstructured pruning compression of second-order information is provided for an embodiment of the application.

[0027] Figure 2 A flowchart of a lightweight speech recognition method based on unstructured pruning compression of second-order information is provided for an embodiment of the application. DETAILED DESCRIPTION

[0028] The following reference description of the drawings introduces several preferred embodiments of the application, making the technical content of the application clearer and easier to understand. The application can be embodied in many different forms, and the scope of protection of the application is not limited to the embodiments mentioned in the text.

[0029] In the drawings, components with the same structure are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the application does not limit the size and thickness of each component.

[0030] In the drawings, components with the same structure are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the application does not limit the size and thickness of each component. In order to make the drawing clearer, the thickness of some components is appropriately exaggerated in some places in the drawing.

[0031] The lightweight speech recognition method based on unstructured pruning compression using second-order information, as implemented according to this invention, mainly includes the following steps:

[0032] S1: Obtain the sparsity of each layer of the target speech recognition model;

[0033] The method of obtaining sparsity is to perform sensitivity analysis based on the Hessian matrix on each layer of the model, and then combine the hybrid sparsity pruning compression search algorithm to obtain the sparsity at a certain compression ratio.

[0034] S2: Use the sparsity from step 1 to search the model layer by layer, calculate the contribution of each parameter to the model performance, and prune the parameters with lower contributions.

[0035] S3: Compensate the parameter set after the above trimming;

[0036] Repeat steps S1 to S3 above to obtain the final pruning and compression model.

[0037] In step S3, the specific compensation method is as follows:

[0038] Taylor expansion is used to estimate the loss function of the variable that is set to zero when one parameter is pruned.

[0039] Meanwhile, the selection of the least important parameter and the updating of the remaining parameters are modeled as a conditional optimization problem. The Lagrange multipliers are used to obtain the variable of the remaining parameter, thereby achieving the compensation of the parameter set.

[0040] like Figure 2 The flowchart shown is a lightweight speech recognition method based on an unstructured pruned compressed neural network model with second-order information, according to an embodiment of the present invention. Specifically, the method mainly includes the following steps:

[0041] S1. First, perform a hybrid sparsity pruning compression search on each different component of the target speech recognition model; the hybrid sparsity pruning compression search algorithm obtains the optimal sparsity allocation under an appropriate compression ratio.

[0042] Sensitivity analysis can measure the importance of a certain layer. Specifically, it is calculated as the sum of the diagonal elements of the corresponding region of the Hessian matrix of the loss function for the parameters of that layer. The higher the sensitivity, the more parameters that layer retains and the smaller the sparsity.

[0043] S2. Search the model layer by layer from shallow to deep according to the sparsity obtained in step S1. Calculate the contribution of each parameter to the model performance based on the input data and the values ​​of the model parameters. Prune the parameters with lower contributions and update the remaining parameters to compensate for the performance degradation caused by pruning.

[0044] modeling the loss function of the model as a multivariate function of the parameters and the input, using Taylor expansion to estimate the loss function for the variable for which one parameter is clipped, i.e., set to zero;

[0045] and modeling the selection of the least important parameter and the update of the remaining parameters as a conditional optimization problem, using Lagrange multipliers to solve for the variables of the remaining parameters.

[0046] S3. Using cyclic pruning to produce a final pruned and compressed model, cyclic pruning first prunes the model to a lower sparsity than the target sparsity, and then fine-tunes the model to correct the model parameters to the position of optimal performance.

[0047] S4. Repeating the above pruning and fine-tuning process until the model is pruned to the target sparsity and the model is fine-tuned to convergence.

[0048] Thus, a model for speech recognition pruned using the above method can be compressed to 35% of its original size while maintaining the performance unchanged. Compared with the prior art, the model for speech recognition pruned using the above method has a 30% performance advantage.

[0049] The preferred embodiments of the present application are described in detail above. It should be understood that modifications and variations can be made by those of ordinary skill in the art without departing from the spirit and scope of the present application. Therefore, it is intended that the scope of the present application be defined by the scope of the claims and not by the detailed description.

Claims

1. A lightweight speech recognition method based on unstructured pruning compression using second-order information, characterized in that, Includes the following steps: S1: Perform sensitivity analysis based on the Hessian matrix on different components of each layer of the target speech recognition model, and combine the hybrid sparsity pruning compression search algorithm to obtain the sparsity of each layer under a certain compression ratio. S2: Using the sparsity obtained in step S1, search the model layer by layer from shallow to deep, and use unstructured pruning with a single parameter as the pruning object. Calculate the contribution of each parameter to the model performance based on the input data and the values ​​of the model parameters, and prune the parameters with lower contributions. S3: The least important parameter and the updated remaining parameters are modeled as a conditional optimization problem. The changes in the remaining parameters are obtained using Lagrange multipliers, thereby compensating for the parameter set after the above-mentioned pruning. S4: Repeat steps S1~S3 above, using a cyclic pruning strategy: first prune the model to a lower sparsity than the target sparsity, then fine-tune the pruned model to correct the model parameters to optimal performance. Repeat the above pruning and fine-tuning process until the model is pruned to the target sparsity and the model is fine-tuned to convergence, so as to obtain the final pruned and compressed model.

2. The lightweight speech recognition method based on unstructured pruning compression using second-order information as described in claim 1, characterized in that, The sensitivity analysis in step S1 is the sum of the diagonal elements of the corresponding region of the Hessian matrix of the loss function for the parameters of that layer.

3. The lightweight speech recognition method based on unstructured pruning compression using second-order information as described in claim 1, characterized in that, In step S2, the change in the loss function due to parameter pruning is estimated by Taylor expansion, thereby calculating the contribution of each parameter to the model performance.

4. The lightweight speech recognition method based on unstructured pruning compression using second-order information as described in claim 1, characterized in that, The size of the compressed model in step S4 is reduced to 35% of its original size.

Images