Multi-dimensional pruning method for deep convolutional neural network model

Abstract

The invention discloses a multi-dimensional pruning method for a deep convolutional neural network model, and the method can automatically complete the processes of model pruning, performance recovery and the like, achieves the efficient compression and optimization of a deep learning model, carries out the fine adjustment of the pruned model, and sets the learning rate to be one tenth or less of the original training. And unstable model performance caused by excessive parameter updating is avoided. Meanwhile, the performance change on the verification data set is monitored, and when the performance on the verification data set is stable or no longer remarkably improved, fine adjustment is stopped. On the other hand, more accurate parameter importance evaluation is realized by comprehensively considering the local importance and global structure information of the parameters.

Description

Technical Field

[0001] The invention relates to the field of digital application technology, and in particular to a multi-dimensional pruning method for a deep convolutional neural network model. Background Art

[0002] In the field of deep learning, as the complexity of the model increases, the number of model parameters and the amount of computation increase dramatically, resulting in storage and computing power challenges when the model is deployed on resource-constrained devices (such as mobile devices, embedded systems, etc.). Model pruning, as an effective model compression technology, aims to reduce the size and computing requirements of the model by removing redundant parameters or structures in the model while maintaining the performance of the model as much as possible.

[0003] Existing model pruning methods mainly include channel pruning, layer pruning, etc. These methods usually rely on some kind of importance evaluation criteria to determine which parts can be safely removed. Among them, the L2 norm, as a simple and effective measurement method, is widely used to evaluate the importance of parameters. However, the direct application of the L2 norm for pruning has the following defects: First, there is a lack of global consideration. Pruning is only based on the L2 norm of local parameters, which may ignore the interaction between parameters and the influence of the overall model structure; second, the pruning granularity is single, and the redundancy of the model in different dimensions is not comprehensively considered, which limits the pruning effect; third, it is difficult to recover performance after pruning. Direct pruning may cause a significant decline in model performance, and there is a lack of effective mechanism to restore or improve the performance of the pruned model. Summary of the invention

[0004] The purpose of the present invention is to address the shortcomings of the prior art and propose a multi-dimensional pruning method for a deep convolutional neural network model. On the one hand, this method can automatically complete the processes of model pruning and performance recovery to achieve efficient compression and optimization of the deep learning model. On the other hand, by comprehensively considering the local importance of parameters and the global structural information, a more accurate parameter importance evaluation can be achieved.

[0005] The present invention discloses a multi-dimensional pruning method for a deep convolutional neural network model, comprising the following steps:

[0006] A1 extracts all parameters to be evaluated from the trained deep learning model, including convolution kernel weights, fully connected layer weights, bias terms, etc.; ensures that the extracted parameters cover all important components in the model;

[0007] A2 calculates the L2 norm of the convolutional neural network model layer by layer and channel by channel as a preliminary indicator of local importance; the L2 norm can reflect the amplitude and overall importance of the parameters;

[0008] A3 trains the model once and extracts the global importance based on the contribution of the parameters in the forward and backward propagation of the model;

[0009] A4 calculates the correlation between parameters and uses methods such as Pearson correlation coefficient to measure the redundancy between parameters;

[0010] A5 weightedly sums the results of local importance, global importance, and correlation analysis to obtain a comprehensive importance score for each parameter;

[0011] A6 sets the pruning ratio and sorts the model parameters according to the comprehensive importance score. The parameters with the lowest ranking are considered as unimportant parameters and pruned accordingly.

[0012] A7 sets the pruning ratio based on the comprehensive importance score, for example, 0.3; all parameters with a comprehensive importance score below this threshold will be considered unimportant parameters and serve as candidates for pruning;

[0013] A8 prunes unimportant parameters. For channel pruning, the entire convolution kernel channel will be pruned, while for layer pruning, the layers with less impact on the output of the entire model will be pruned. Pruning uses soft pruning, which resets the weights to be removed to zero.

[0014] A9 fine-tunes the pruned model and sets the learning rate to one-tenth or less of the original training to avoid unstable model performance caused by excessive parameter updates. At the same time, it monitors the performance changes on the validation dataset and stops fine-tuning when the performance on the validation dataset reaches stability or no longer improves significantly.

[0015] Preferably, the pruning ratio set in A7 is 0.3, and the pruning ratio needs to be adjusted according to experiments and model characteristics.

[0016] Preferably, before performing multi-dimensional pruning, a deep learning model importance scoring based on L2 norm and global information fusion is required, and the scoring method includes the following steps:

[0017] B1 calculates the local importance score; for each parameter in the model, its L2 norm is calculated by formula (1) as a preliminary importance indicator:

[0018]

[0019] Among them, w i represents the weight vector of the i-th parameter, n represents the dimension of the vector, w ij Represents vector w j The jth component of

[0020] B2 extracts the global importance score; extracts global information based on the contribution of the parameter in the forward propagation and back propagation of the model; here, as shown in formula (2), the absolute value of the gradient of the calculated parameter is selected as the measurement standard:

[0021]

[0022] in, Represents the loss function value of the kth sample, y k and Represent the label and predicted value of the kth sample respectively, represents the gradient of the loss function with respect to the i-th parameter, and m represents the total number of training samples;

[0023] B3 constructs a correlation matrix; calculates the correlation between parameters, constructs a correlation matrix R, and uses the Pearson correlation coefficient to calculate the correlation between parameters. The specific calculation process is expressed as formula (3):

[0024]

[0025] Among them, Cov(w i ,w j ) represents the parameter w i and w j The covariance of Var(w i ) and Var(w j ) represent the parameters w i and w j The variance of Pearson's correlation coefficient R ij The value range of is between [-1,1]. The larger the absolute value, the stronger the correlation between the two parameters.

[0026] B4 integrates local and global information; weighted fusion of local importance score, global importance score and correlation matrix is ​​performed to obtain the comprehensive importance score of each parameter; the specific calculation can be expressed as shown in formula (4):

[0027]

[0028] Among them, α, β and γ are weight coefficients, which are used to adjust the contribution of local importance, global information and correlation matrix in the comprehensive score; note that the negative sign here means that the redundancy between parameters is considered in the comprehensive score, that is, the parameters with higher correlation will be appropriately reduced in the comprehensive score; the weight coefficients α, β and γ are determined by experiments or experience.

[0029] Preferably, the parameters in the model include channel weights and layer weights.

[0030] The advantages and technical effects of the present invention are:

[0031] By introducing a multi-dimensional importance scoring mechanism, the present invention can accurately evaluate the importance of model parameters and implement precise pruning operations accordingly. This multi-dimensional pruning method not only effectively reduces the redundant parameters of the model, but also retains the key information of the model to the maximum extent through a detailed pruning strategy, so that good performance can be maintained after pruning. Through fine-tuning technology, the performance of the pruned model can be further restored and optimized, achieving the dual goals of efficient model compression and performance preservation;

[0032] The importance scoring and multi-dimensional pruning methods of the present invention provide a high degree of flexibility and can be customized according to the characteristics of different models and application requirements. Whether for complex large models or for specific application scenarios, the present invention can achieve targeted model optimization by adjusting the calculation method and pruning strategy of importance scoring. This customization capability enables the present invention to be widely used in various deep learning fields, providing personalized model compression and optimization solutions for users with different needs. DETAILED DESCRIPTION

[0033] The following is a detailed description of an embodiment of the invention, which is exemplary and is only used to explain the invention, and is not to be construed as limiting the invention.

[0034] The first object of the present invention is to provide a multi-dimensional precise pruning method based on the above-mentioned importance scoring method, which can automatically complete model pruning, performance recovery and other processes to achieve efficient compression and optimization of deep learning models.

[0035] The technical solution to achieve the first object of the present invention is: a multi-dimensional precise pruning method, the specific steps are as follows:

[0036] ① Extract all parameters to be evaluated from the trained deep learning model, including convolution kernel weights, fully connected layer weights, bias terms, etc. Make sure the extracted parameters cover all important components in the model;

[0037] ② Calculate the L2 norm of the convolutional neural network model layer by layer and channel by channel as a preliminary indicator of local importance. The L2 norm can reflect the amplitude and overall importance of the parameters;

[0038] ③ Perform one training iteration on the model and extract the global importance based on the contribution of the parameters in the forward and backward propagation of the model;

[0039] ④ Calculate the correlation between parameters and use methods such as Pearson correlation coefficient to measure the redundancy between parameters;

[0040] ⑤ Weighted sum of the results of local importance, global importance and correlation analysis to obtain the comprehensive importance score of each parameter;

[0041] ⑥ Set the pruning ratio and sort the model parameters according to the comprehensive importance score. The parameters with the lower ranking are regarded as unimportant parameters and pruned accordingly;

[0042] ⑦ Based on the comprehensive importance score, set the pruning ratio, for example 0.3 (this value needs to be adjusted according to the experiment and model characteristics).

[0043] All parameters with a comprehensive importance score below this threshold will be considered unimportant parameters and will be candidates for pruning. ⑧ Prune unimportant parameters. For channel pruning, the entire convolution kernel channel will be pruned, while for layer pruning, the layers that have little impact on the output of the entire model will be pruned. Pruning uses soft pruning, which resets the weights to be removed to zero;

[0044] Fine-tune the pruned model, and set the learning rate to one-tenth or less of the original training to avoid unstable model performance caused by excessive parameter updates. Meanwhile, monitor the performance changes on the validation dataset, and stop fine-tuning when the performance on the validation dataset reaches stability or no longer improves significantly.

[0045] The technical solution to achieve the second purpose of the present invention is to propose a deep learning model importance scoring method based on L2 norm and global information fusion, and the specific steps are as follows:

[0046] ① Calculate the local importance score. For each parameter in the model (such as channel weight, layer weight, etc.), calculate its L2 norm as a preliminary importance indicator using formula (1):

[0047]

[0048] Among them, w i represents the weight vector of the i-th parameter, n represents the dimension of the vector, w ij Represents vector w j The jth component of

[0049] ② Extract global importance scores. Extract global information based on the contribution of parameters in the forward and backward propagation of the model.

[0050] Here, as shown in formula (2), the absolute value of the gradient of the calculated parameter is selected as the criterion:

[0051]

[0052] in, Represents the loss function value of the kth sample, y k and Represent the label and predicted value of the kth sample respectively, represents the gradient of the loss function with respect to the i-th parameter, and m represents the total number of training samples;

[0053] ③ Construct a correlation matrix. Calculate the correlation between parameters, construct the correlation matrix R, and use the Pearson correlation coefficient to calculate the correlation between parameters. The specific calculation process is expressed as formula (3):

[0054]

[0055] Among them, Cov(w i ,w j ) represents the parameter w i and w j The covariance of Var(w i ) and Var(w j ) represent the parameters w i and w j The variance of Pearson's correlation coefficient R ij The value range of is between [-1,1]. The larger the absolute value, the stronger the correlation between the two parameters.

[0056] ④ Fusion of local and global information. The local importance score, global importance score and correlation matrix are weighted fused.

[0057] The comprehensive importance score of each parameter is obtained. The specific calculation can be expressed as shown in formula (4):

[0058]

[0059] Among them, α, β, and γ are weight coefficients used to adjust the contribution of local importance, global information, and correlation matrix in the comprehensive score. Note that the negative sign here means that the redundancy between parameters is considered in the comprehensive score, that is, the parameters with higher correlation will be appropriately reduced in the comprehensive score. The weight coefficients α, β, and γ are determined by experiments or experience.

[0060] In order to more clearly illustrate the specific implementation mode of the present invention, an embodiment is provided below:

[0061] A multi-dimensional pruning method for a deep convolutional neural network model of the present invention introduces a multi-dimensional importance scoring mechanism to accurately evaluate each parameter of the deep learning model, and sets the pruning threshold accordingly to implement precise pruning operations, thereby effectively reducing the redundant parameters of the model. After pruning, the model performance is restored and possibly improved through fine-tuning technology, achieving the dual goals of efficient model compression and performance retention. This technical solution is highly flexible and widely applicable, and can perform customized pruning according to the characteristics and application requirements of different models to meet the needs of various deep learning fields while maintaining the efficiency and accuracy of the model. Based on this, a multi-dimensional precise pruning method is proposed, and the specific steps are as follows:

[0062] ① Existing model training: The model to be pruned should have been trained on the original data set and reached convergence;

[0063] ② Comprehensive parameter extraction: systematically extract all parameters to be evaluated from the trained deep learning model, which covers key components such as convolution kernel weights, fully connected layer weights, and bias terms. Ensure that nothing is missed so that subsequent analysis can fully reflect the important features of the model;

[0064] ③ Local importance evaluation (L2 norm calculation): Calculate the L2 norm of the convolutional neural network model layer by layer and channel by channel as a preliminary indicator to measure the local importance of the parameters. The L2 norm can intuitively reflect the amplitude and overall importance of the parameters, providing a strong basis for subsequent pruning operations;

[0065] ④ Global importance evaluation: During the iteration of model training, the contribution of parameters in forward propagation and back propagation is analyzed to extract global importance information. This helps to more fully understand the role of parameters in the model and provide additional reference for pruning decisions;

[0066] ⑤ Correlation analysis: In order to identify the redundancy between parameters, the correlation between parameters is calculated using methods such as the Pearson correlation coefficient. By quantifying the degree of correlation between parameters, redundant parameters can be effectively identified and removed, further improving the compactness and efficiency of the model;

[0067] ⑥ Comprehensive importance score calculation: The results of local importance, global importance and correlation analysis are weighted and summed to generate a comprehensive importance score for each parameter, providing a more accurate basis for subsequent pruning operations;

[0068] ⑦ Pruning strategy formulation: set the pruning ratio (the specific value needs to be adjusted according to the experiment and model characteristics), and sort the model parameters according to the comprehensive importance score. The parameters (channels or layers) ranked later are considered unimportant parameters in proportion and will be pruned later;

[0069] ⑧Model pruning execution: Sort the model parameters according to the set pruning ratio, and prune the parameters with the lowest order as unimportant parameters. For channel pruning, directly remove the entire convolution kernel channel; for layer pruning, prune the layers that have little impact on the output of the entire model. The pruning operation adopts soft pruning, that is, reset the weights to be removed to zero;

[0070] Model fine-tuning: After pruning is completed, the model is fine-tuned to restore its performance. During this process, the learning rate is set to one-tenth or less of the original training to avoid excessive parameter updates that lead to unstable model performance. At the same time, the performance changes on the validation dataset are monitored, and fine-tuning is stopped when the performance reaches stability or no longer improves significantly.

[0071] Finally, all the unexplained parts of the invention adopt mature products and mature technical means in the existing technology.

[0072] In the description of this specification, reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in the embodiment or example of the invention.

[0073] Although embodiments of the invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A multi-dimensional pruning method for a deep convolutional neural network model, characterized in that: The following steps are involved: A1 extracts all parameters to be evaluated from the trained deep learning model, including convolution kernel weights, fully connected layer weights, bias terms, etc.; ensures that the extracted parameters cover all important components in the model; A2 calculates the L2 norm of the convolutional neural network model layer by layer and channel by channel as a preliminary indicator of local importance; the L2 norm can reflect the amplitude and overall importance of the parameters; A3 trains the model once and extracts the global importance based on the contribution of the parameters in the forward and backward propagation of the model; A4 calculates the correlation between parameters and uses methods such as Pearson correlation coefficient to measure the redundancy between parameters; A5 weightedly sums the results of local importance, global importance, and correlation analysis to obtain a comprehensive importance score for each parameter; A6 sets the pruning ratio and sorts the model parameters according to the comprehensive importance score. The parameters with the lowest ranking are considered as unimportant parameters and pruned accordingly. A7 sets the pruning ratio based on the comprehensive importance score, for example, 0.3; all parameters with a comprehensive importance score below this threshold will be considered unimportant parameters and serve as candidates for pruning; A8 prunes unimportant parameters. For channel pruning, the entire convolution kernel channel will be pruned, while for layer pruning, the layers with less impact on the output of the entire model will be pruned. Pruning uses soft pruning, which resets the weights to be removed to zero. A9 fine-tunes the pruned model and sets the learning rate to one-tenth or less of the original training to avoid unstable model performance caused by excessive parameter updates. At the same time, it monitors the performance changes on the validation dataset and stops fine-tuning when the performance on the validation dataset reaches stability or no longer improves significantly.

2. A multi-dimensional pruning method for a deep convolutional neural network model according to claim 1, characterized in that: The pruning ratio set in A7 is 0.3, and the pruning ratio needs to be adjusted according to the experiment and model characteristics.

3. The multi-dimensional pruning method of a deep convolutional neural network model according to claim 1, characterized in that: Before performing multi-dimensional pruning, it is necessary to first perform a deep learning model importance score based on the L2 norm and global information fusion. The scoring method includes the following steps: B1 calculates the local importance score; for each parameter in the model, its L2 norm is calculated by formula (1) as a preliminary importance indicator: Among them, w i represents the weight vector of the i-th parameter, n represents the dimension of the vector, w ij Represents vector w j The jth component of B2 extracts the global importance score; extracts global information based on the contribution of the parameter in the forward propagation and back propagation of the model; here, as shown in formula (2), the absolute value of the gradient of the calculated parameter is selected as the measurement standard: in, Represents the loss function value of the kth sample, y k and Represent the label and predicted value of the kth sample respectively, represents the gradient of the loss function with respect to the i-th parameter, and m represents the total number of training samples; B3 constructs a correlation matrix; calculates the correlation between parameters, constructs a correlation matrix R, and uses the Pearson correlation coefficient to calculate the correlation between parameters. The specific calculation process is expressed as formula (3): Among them, Cov(w i ,w j ) represents the parameter w i and w j The covariance of Var(w i ) and Var(w j ) represent the parameters w i and w j The variance of Pearson's correlation coefficient R ij The value range of is between [-1,1]. The larger the absolute value, the stronger the correlation between the two parameters. B4 integrates local and global information; weighted fusion of local importance score, global importance score and correlation matrix is ​​performed to obtain the comprehensive importance score of each parameter; the specific calculation can be expressed as shown in formula (4): Among them, α, β and γ are weight coefficients, which are used to adjust the contribution of local importance, global information and correlation matrix in the comprehensive score; note that the negative sign here means that the redundancy between parameters is considered in the comprehensive score, that is, the parameters with higher correlation will be appropriately reduced in the comprehensive score; the weight coefficients α, β and γ are determined by experiments or experience.

4. The multi-dimensional pruning method of a deep convolutional neural network model according to claim 1, characterized in that: The parameters in the model include channel weights and layer weights.

Images