A model pruning method of an interpretable CNN classification model

Abstract

The application relates to a model pruning method of an interpretable CNN classification model, belongs to the field of image compression, and solves the problems of high operation complexity, large time and memory consumption and difficulty in deployment on terminal equipment of an existing deep CNN model, and solves the problem of lack of interpretability of an existing model pruning algorithm. The method comprises the following steps: inputting a training picture into a neural network model to be pruned, and extracting a feature map matrix of each convolution layer; upsampling the feature map matrix to the size of the input picture, and then performing a normalization operation to construct a saliency map; multiplying the saliency map and the input picture element by element to construct a weighted input picture; subtracting the input picture from the weighted picture element by element to construct an attention region occlusion map; inputting the attention occlusion map into the model to be pruned, observing the change of model accuracy as an importance score of the channel, and pruning the channel to obtain a pruned lightweight model. The application realizes high pruning rate of the model and improves the interpretability of the pruning process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep model model compression, in particular to a model pruning method of an interpretable CNN classification model. BACKGROUND

[0002] As one of the most important deep models, the convolutional neural network (CNN) has achieved great success in image processing, target tracking and detection, natural language processing and many other fields due to its good feature extraction ability and generalization ability. However, this also brings a network structure that is getting deeper and wider, which brings great storage resource consumption and computing resource consumption to the deployment of the convolutional neural network on edge devices. For example, the common VGG-16 model has a parameter quantity of up to 138M, and 15 billion floating-point operations are required in the forward inference process to classify a picture.

[0003] Model pruning is a commonly used model compression method, which can effectively alleviate the over-parameterization phenomenon of deep models and does not require special hardware and software support. The commonly used pruning method mainly uses the weight size of the convolution kernel in the convolution network to evaluate the importance of the convolution kernel. Due to the influence of input data floating, the compression rate, interpretability and accuracy that can be achieved by the existing method have further space for improvement. SUMMARY

[0004] In order to improve the deficiencies of the existing method, the present application provides a model pruning method of an interpretable CNN classification model, which aims to compress the calculation amount and parameter amount of the existing deep learning model, improve the identification ability of redundant parameters in the deep learning model, solve the problem of low compression rate and great influence on model accuracy of the existing method, and realize the visual explanation of the pruning process through the organic combination with the model explanation method.

[0005] The present application realizes the visual explanation of the model pruning process by combining the model explanation method with the model pruning process, and realizes the importance evaluation of each filter of the model only relying on the forward inference process of the model through the disturbance processing of the input picture. This method combines the traditional model pruning process with the model explanation method, and simultaneously evaluates the importance of each filter by using the forward inference process of the model, which has the advantages of large compression degree and low influence on the original accuracy of the model.

[0006] The technical scheme of the present application is as follows:

[0007] A model pruning method of an interpretable CNN classification model, comprising the following steps:

[0008] 1) Construction of the baseline model: By training a predetermined deep neural network CNN classification model on a predefined data set, a baseline model with certain generalization ability but containing redundant parameters is obtained. Specifically: first, use the python programming language and pytorch deep learning framework to build a predetermined deep neural network CNN classification model; then use the predetermined training parameters to train the model on the data set, and record the accuracy of the model as the baseline score, to obtain the baseline model.

[0009] 2) Obtaining filter attention region occlusion picture: In order to evaluate the role of each filter in the inference process of the CNN classification model, the attention region of each filter in the input picture needs to be obtained, and then the region is occluded to simulate the deletion of the filter's influence on the model. Specifically: first, input a picture in the training set into the baseline model obtained in step 1), extract the feature map matrix of a certain convolution layer, and the classification score of the model for the picture; then, take the up-sampling operation on each feature map matrix, enlarge the matrix to the size of the input picture; normalize the up-sampled picture to limit the value of its elements to the range of 0-1, and then obtain the saliency map of each filter; further, multiply the obtained saliency map with the input picture element by element to obtain the weighted input picture; input the weighted picture into the ReLU activation function, and retain the part of the weighted picture greater than zero (the part of interest of the filter), to obtain the attention region map; finally, by respectively subtracting the original input picture from the obtained attention region map element by element, the attention region occlusion picture retaining all information of the original input picture except the attention region of a certain filter is obtained.

[0010] 3) Filter importance score generation: By observing the accuracy change of the model to be pruned, the importance of the filters of the convolution layer is evaluated by using the forward inference process of the model. Specifically: first, input the attention region occlusion picture obtained in step 2) into the baseline model obtained in step 1), and record the classification score of each occlusion picture input into the model; subtract the baseline score recorded in step 1) from the score of the occlusion picture to obtain the score of the corresponding filter in the inference of the input picture. Different filters have different activation degrees for different class pictures, and the score calculated from a single picture cannot reflect the role of each filter in identifying all class pictures. Therefore, n pictures belonging to different classes are randomly sampled from the training data set, and the above process is repeated to obtain n scores for each filter; further, the scores are summed and then normalized. Finally, the comprehensive contribution score of the filter to the identification of each class picture is obtained.

[0011] 4) Pruning: the scores of the integrated contribution degree obtained from step 3) are sorted from small to large, all filters with scores less than a predefined pruning threshold are marked as redundant parts, and all parameters related to the filters are directly deleted, including the parameters of the convolution kernel, the parameters of the adjacent batch normalization layer, the input channels of the next convolution layer, etc. For the baseline model obtained in step 1), pruning is performed on each layer of convolution from input to output, and after pruning each layer, fine-tuning is performed to restore the generalization ability of the model, and a lightweight model is obtained.

[0012] Wherein, the predefined data set in step 1) includes but is not limited to common classification data sets such as CIFAR-10, CIFAR-100 and ImageNet, and can also be a classification data set collected by the user himself.

[0013] Wherein, the network in step 1) is a CNN classification network constructed using a machine learning method, which has a structure sufficient to support feature mining on the predefined data set, without specific restrictions on the specific structure of the convolution network, the type of convolution layer and the depth of convolution.

[0014] Wherein, the training parameters in step 1) include learning rate, learning rate decay mode, learning round, batch size, etc.

[0015] Wherein, the upsampling operation in step 2) is specifically implemented by using an interpolation algorithm such as bilinear interpolation; and the normalization operation is specifically implemented by using maximum and minimum normalization.

[0016] Wherein, each element of the saliency map in step 2) represents the attention score of a certain filter on the corresponding pixel point of the input picture.

[0017] Wherein, the filter importance score of a single picture obtained in step 3) can be positive or negative. If the score is positive, it means that the content concerned by the filter has a positive effect on the model inference process; if the score is negative, it means that the filter interferes with the model inference process.

[0018] Wherein, the normalization operation in step 3) is specifically implemented by using maximum and minimum normalization.

[0019] Wherein, the pruning threshold in step 4) is a decimal number in the range of 0-1, and is generally set to 0.5 or 0.6.

[0020] The beneficial effects of the present application are: compared with the existing technical method, the model pruning process and the model explanation method are combined to realize the compression of the deep CNN model, effectively improve the compression rate of the model, and the influence on the accuracy of the model is very small. Through the model explanation method, the attention area of each filter in the input picture is obtained, which can display the information obtained by the filter in a visual way; by using the filter attention area to block the picture to disturb the input picture, the importance degree of the model to each filter can be evaluated, which eliminates the disadvantage of insufficient single layer data; the present application takes the filter in the convolution layer as the operation unit of pruning, which does not destroy the overall structure of the model, does not need special software and hardware support, and has good application prospect for the deployment of deep CNN network on edge devices. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The overall flowchart of the model pruning method of the interpretable CNN classification model of the present application.

[0022] Figure 2 The example diagram of the input picture described in the present example.

[0023] Figure 3 The example diagram of the filter saliency map described in the present example.

[0024] Figure 4 The example diagram of the filter attention area described in the present example.

[0025] Figure 5 The example diagram of the filter attention area described in the present example. DETAILED DESCRIPTION

[0026] The specific embodiments of the present application are further illustrated in combination with the drawings and technical solutions.

[0027] As shown in Figure 1 A model pruning method of an interpretable CNN classification model, including the steps of constructing a benchmark model, obtaining a filter attention area blocked picture, generating a filter importance score, and pruning, etc., specifically as follows:

[0028] Step 1, training of the baseline model. This example constructs a ResNet-50 network to implement a classification task on an ImageNet dataset, and the initial model structure of the ResNet-50 is shown in Table 1: random horizontal flipping and random cropping are performed on the pictures in the ImageNet dataset to enhance the diversity of the data, and normalization is performed on the pictures; a neural network is built according to the network structure of the ResNet-50, and the input specification of the network is set to 224*224*3, where the three channels represent RGB channels; the batch size, momentum and weight decay in the training are set to 256, 0.9 and 0.0001, respectively; the random gradient algorithm is used, and the training is performed for 120 rounds, the initial learning rate is set to 0.5, and the cosine annealing strategy is used to update the learning rate; thus, a baseline model capable of completing the classification task on the ImageNet is obtained, but the baseline model contains redundant parameters (the Top-1 accuracy of the model is 76.15%, and the Top-5 accuracy is 92.87%).

[0029] Table 1 ResNet-50 model structure before pruning

[0030] Convolutional layer Number of output channels Conv-1 64 Layer1_bottleneck0 64，64，256 Layer1_bottleneck1 64，64，256 Layer1_bottleneck2 64，64，256 Layer2_bottleneck0 128，128，512 Layer2_bottleneck1 128，128，512 Layer2_bottleneck2 128，128，512 Layer2_bottleneck3 128，128，512 Layer3_bottleneck0 256，256，1024 Layer3_bottleneck1 256，256，1024 Layer3_bottleneck2 256，256，1024 Layer3_bottleneck3 256，256，1024 Layer3_bottleneck4 256，256，1024 Layer3_bottleneck5 256，256，1024 Layer4_bottleneck0 512，512，2048 Layer4_bottleneck1 512，512，2048 Layer4_bottleneck2 512，512，2048

[0031] Step 2, obtaining of the occlusion attention region picture: taking the layer4_bottleneck2_conv3 convolutional layer of the ResNet-50 (the third convolutional layer of the last bottleneck structure of the fourth layer of the model) as an example, assuming that the input picture of the model is a picture X of a snake. Then the input of the convolutional layer is the 7*7*512 intermediate picture generated by the upper layer, and each filter generates a 7*7 size feature map matrix, and all the feature maps constitute a set The following bilinear interpolation formula is used for each feature map to upsample the feature map to 224*224 size to obtain the upsampled feature map set A 3up The maximum and minimum values of each matrix in A 3up are calculated, and the following formula is used to perform a normalization operation on each matrix to obtain the saliency map set A 3sal of the filter. Further, each matrix in A 3sal is multiplied with the input picture element by element to obtain the weighted input picture A 3weig hted , and ReLU is applied to the weighted picture to remove the negative part of the picture to obtain the final saliency map. Because all elements in A 3weig hted have been normalized to the range [0, 1], each matrix in A 3weig hted can be directly subtracted by 1 to obtain the final occlusion picture A 3mask .

[0032]

[0033]

[0034] A 3weig hted = X·A 3sal

[0035] A 3mask = 1-A 3weig hted

[0036] where (x, y) represents the point of the matrix which we want to estimate, the feature map matrix is enlarged to the size of the input picture by the formula; A 3up represents the matrix after up-sampling operation, minA 3up represents the minimum value in the matrix, maxA 3up represents the maximum value in the matrix; A 3sal represents the saliency map, each element of which represents the attention score of a filter on each pixel in the input picture; X·A 3sal represents the element-wise multiplication of the input picture and the saliency map; A 3weig hted represents the weighted input picture; A 3mask represents the occlusion picture; 1 represents a full 1 matrix with the same size as the input picture;

[0037] where the input picture in the present example is shown in Figure 2 , the filter saliency map is shown in Figure 3 , the filter attention region is shown in Figure 4 , and the filter occlusion picture is shown in Figure 5 .

[0038] Step 3, generation of filter importance scores. Take the 7th, 27th and 31st filters in the layer4_bottleneck2_conv3 convolutional layer as an example: the accuracy of ResNet-50 in correctly classifying the picture of the snake is 0.71, and after inputting the attention occlusion pictures of the three filters into the model, the probabilities of the model successfully classifying the picture become 0.30, 0.56 and 0.46 respectively. The changes in accuracy are 0.41, 0.15 and 0.25 respectively. It can be seen that the 7th filter mainly focuses on the body part of the snake, and after occluding its attention region, the accuracy of the model in correctly classifying the picture decreases by 0.41, indicating that the part of interest of the filter is important to the model inference process; while the 27th filter mainly focuses on the texture on the sand, and deleting the attention region of the filter has little effect on the model, which is consistent with our way of thinking. After repeating the above process on 1000 pictures sampled from the training data set, the scores of the three filters are 0.9, 0.4 and 0.6 respectively, and the importance ranking of the three filters is: 7th > 31st > 9th.

[0039] Step 4, Pruning. The pruning threshold is set to 0.46, so the 31st filter in step 2 will be removed. Repeat the above steps 2-3 for the first two convolutional layers in each bottleneck structure from input to output, in order, to remove filters with importance scores less than 0.46, to achieve pruning of the model. The convolutional layers close to the input are not the target of pruning, because that will affect the extraction of information from the input image; the third convolutional layer in each bottleneck is also not in the scope of pruning, because the presence of the residual structure changes the number of its output channels, which will affect the connection of each layer of the model. The process of pruning for ResNet-50 can be described as:

[0040] Input: a trained ResNet-50 model

[0041] Pruning:

[0042] Repeat the following process for the first two convolutional layers in each bottleneck from input to output:

[0043] Sample 1000 images from the input dataset;

[0044] Calculate the score of each filter in the convolution of this layer according to steps 2 and 3;

[0045] Rank the filters according to the scores, and remove the filters with scores less than 0.46;

[0046] Get the new model structure, and inherit the parameters of the remaining filters from the old model;

[0047] Fine-tune the new model on the training dataset;

[0048] Output: a pruned ResNet-50 model

[0049] After pruning, the ResNet-50 model has a 56.4% reduction in the number of floating-point operations per second (FLOPs) and a 55.5% reduction in the number of parameters. The Top-1 accuracy of the model is 72.45% (a decrease of 3.67%); the Top-5 accuracy is 90.63% (a decrease of 2.24%). The structure of the pruned model is shown in Table 2:

[0050] Table 2 Structure of the pruned ResNet-50 model

[0051]

[0052]

[0053] The model pruning process is combined with the model explanation method, the attention area of the filter in the input picture can be accurately found, and the information obtained by the filter can be displayed in a visual manner; the input picture is disturbed by shielding the picture by using the filter attention area, the importance degree of each filter can be evaluated by using the model as a whole, and the disadvantage of insufficient single-layer data is eliminated. The redundant structure in the model can be more efficiently and accurately identified, the compression rate of the calculation amount and the storage resource consumption of the model is improved, and the influence of the model pruning on the accuracy of the model is minimized.

Claims

1. A method for model pruning of an interpretable CNN classification model, characterized in that, The method comprises the following steps: 1) Construction of a baseline model: a predetermined deep neural network CNN classification model is trained on a predefined data set to obtain a baseline model; specifically: first, a predetermined deep neural network CNN classification model is built using the python programming language and the pytorch deep learning framework; then the model is trained on the data set using predetermined training parameters, and the accuracy of the model is recorded as a baseline score to obtain the baseline model; 2) Obtain the attention region occlusion picture of the filter: in order to evaluate the role of each filter in the inference process of the CNN classification model, the attention region of each filter in the input picture needs to be obtained, and then the region is occluded to simulate the deletion of the filter to the model; specifically: first, input a picture in the training set into the baseline model obtained in step 1), extract the feature map matrix of a certain convolution layer, and the classification score of the model for the picture; then, perform upsampling operation on each feature map matrix, and enlarge the matrix to the size of the input picture; normalize the upsampling picture to limit the value of its elements to the range of 0-1, and then obtain the saliency map of each filter; further, multiply the obtained saliency map with the input picture element by element to obtain a weighted input picture; input the weighted picture into the ReLU activation function, and retain the part greater than zero in the weighted picture to obtain the attention region map; finally, by respectively subtracting the original input picture from the obtained attention region map element by element, the attention region occlusion picture retaining all information of the original input picture except the attention region of a certain filter is obtained; 3) Filter importance score generation: by observing the accuracy change of the model to be pruned, the importance of the filters of the convolution layer is evaluated by using the forward inference process of the model; specifically: first, input the attention region occlusion picture obtained in step 2) into the baseline model obtained in step 1), and record the classification score of each occlusion picture input into the model; subtract the baseline score recorded in step 1) from the score of the occlusion picture to obtain the role of the corresponding filter in inferring the input picture; different filters have different activation degrees for different category pictures, and the score calculated from a single picture cannot reflect the role of each filter in identifying all category pictures; therefore, n pictures belonging to different categories are randomly sampled from the training data set, and the above process is repeated to obtain n scores of each filter; Further, the scores are summed and then normalized; finally, the comprehensive contribution score of the filter to the identification of the category picture is obtained. 4) Pruning: the comprehensive contribution scores obtained from step 3) are sorted from small to large, all filters whose comprehensive contribution scores are less than a predefined pruning threshold are marked as redundant parts, and all parameters related to the filters are directly deleted; for the base model obtained in step 1), pruning is performed on each layer from input to output, and after pruning is completed for each layer, fine-tuning is performed to restore the generalization ability of the model, and a lightweight model is obtained; the parameters include the parameters of the convolution kernel, the parameters of the adjacent batch normalization layer, and the input channels of the next convolution layer.

2. The model pruning method of an interpretable CNN classification model according to claim 1, characterized in that, The predefined data set of step 1) includes common classification data sets such as CIFAR-10, CIFAR-100 and ImageNet, and classification data sets collected by the user himself.

3. The model pruning method of an interpretable CNN classification model according to claim 1, characterized in that, The training parameters of step 1) include learning rate, learning rate decay mode, learning round and batch size.

4. The model pruning method of an interpretable CNN classification model according to claim 1, characterized in that, The up-sampling operation of step 2) is a bilinear interpolation algorithm; the normalization operation is a maximum and minimum normalization.

5. The model pruning method of an interpretable CNN classification model according to claim 1, characterized in that, The filter importance score of step 3) may be positive or negative. If the score is positive, it means that the content concerned by the filter has a positive effect on the model inference process; if the score is negative, it means that the filter interferes with the model inference process.

6. The model pruning method of an interpretable CNN classification model according to claim 1, characterized in that, The normalization operation of step 3) is a maximum and minimum normalization.

7. The model pruning method of an interpretable CNN classification model according to claim 1, characterized in that, The pruning threshold of step 4) is a decimal in the range of 0-1.

8. The model pruning method of an interpretable CNN classification model according to claim 1, characterized in that, Step 2) realizes the generation of the attention mask picture based on the following formula: ； ； ； ； wherein, represents the point of the matrix that one wants to estimate, by which the feature map matrix is up-scaled to the size of the input picture; represents the matrix after the up-sampling operation, represents the minimum value in the matrix, represents the maximum value in the matrix; represents the saliency map, each element of which represents the attention score of a certain filter on each pixel point in the input picture; represents the element-wise multiplication of the input picture and the saliency map; represents the weighted input picture; represents the occlusion picture; represents the all-ones matrix with the same size as the input picture.

Images