Deep neural network compression method based on image classification task category set characteristics

Abstract

The invention provides a deep neural network compression method based on image classification task category set characteristics, and the method comprises the steps: carrying out the pruning of a model according to the category expected to be encountered by the model, enabling the model to achieve a higher cutting rate, judging the importance of a corresponding convolution kernel through the rank size of a feature graph, and cutting off the convolution kernel with low importance. When the number of categories to be identified by the classification task is reduced, the method can effectively reduce model parameters. Compared with other model cutting methods, the model cut by the method is lighter. According to the method, the importance of the convolution kernel under the sub-classification task can be obtained by combining the ranks of the feature maps of all categories, the network does not need to be completely trained again, and compared with training a small model from the beginning, the method is faster.

Description

Technical Field

[0001] This invention relates to the field of deep neural network compression technology, specifically a deep neural network compression method based on the characteristics of image classification task category sets. Background Technology

[0002] Convolutional Neural Networks (CNNs) have demonstrated outstanding performance in various fields, including image classification, object detection, and image segmentation. However, the computational intensity of CNNs poses a significant challenge to their deployment on resource-constrained systems, such as edge devices. Running full CNNs on these systems is often time-consuming and power-intensive, and the limited computing power and energy make these models difficult to deploy. Therefore, it is often necessary to compress neural network models before deploying them to edge devices.

[0003] Current model compression methods include pruning, quantization, low-rank matrix and tensor decomposition, knowledge distillation, and their combinations.

[0004] Pruning includes structured pruning and unstructured pruning. Unstructured pruning refers to pruning individual weights to 0 without considering the network structure. Unstructured pruning does not follow specific geometry or constraints and, in most cases, requires additional information to represent sparse locations. Structured pruning refers to removing entire parts of the model, such as convolutional kernels and channels. The difference between the two lies in the granularity of the pruning. Unstructured pruning has a finer granularity but usually requires specific hardware and acceleration algorithms, while structured pruning is simpler, more flexible, and easier to apply to real-world systems. Quantization refers to the process of representing values ​​with fewer bits. In neural networks, this corresponds to weights, activation values, and gradient values. Typically, during training on a GPU, values ​​are stored in 32-bit floating-point (FP) single-precision. Floating-point half-precision (FP-16) and integer arithmetic (INT-16) are also commonly considered. In FP-16 operations, multiplication results are often accumulated to FP-32 precision before being converted back to FP-16. While maintaining accuracy, this approach still incurs significant overhead in terms of memory and bandwidth usage. To further accelerate training, improve inference speed, and reduce memory and bandwidth requirements, current research tends to employ lower-precision integer representations for training and inference, such as INT-8, INT-4, INT-2, or even 1-bit binary precision. These low-precision representations can significantly compress the model while achieving a better balance between performance and resource consumption. Designing such networks makes training on CPUs, FPGAs, ASICs, and GPUs much easier. Low-rank matrices and tensor decomposition refer to the compression of deep neural networks by decomposing the weight tensors (second-order tensors in the case of matrices) into lower-rank approximations, which also removes redundancy in the parameters. Many studies applying tensor decomposition to deep neural networks are based on the use of singular value decomposition. Knowledge distillation involves learning a smaller network from a large network, through supervision from the large network, and minimizing the entropy, distance, or divergence between their probability estimates. Bucilua et al. (2006) first explored the idea of ​​reducing model size by learning student networks from model ensembles. They used a teacher network to label a large amount of unlabeled data and used pseudo-labels provided by the teacher to train a student network. They found that using a network 1000 times smaller than the original ensemble achieved performance close to that of the original ensemble. In summary, the choice of compression method depends primarily on the specific application scenario and hardware support, and they may be used in combination. Summary of the Invention

[0005] The technical problem to be solved:

[0006] As described in the background section, current deep neural network model compression methods are designed for complete tasks. Taking image classification as an example, in practical applications, some categories may not need to be identified, and fewer recognition targets mean that the model can be more lightweight.

[0007] For example, in the context of autonomous driving, recognizing and avoiding obstacles such as cats and dogs is crucial, while recognizing birds is unnecessary because birds do not obstruct the vehicle's path. In this case, the neural network pruned for the full task will still have redundant parts. If a certain category is excluded from the subclassification task, removing channels that are only important to that category will not affect the model's performance, and we can further prune the CNNs. Of course, we can use methods such as knowledge distillation to train a small model from scratch, but this method is quite time-consuming, as each small model must go through the entire (train-prune-fine-tuning) process. Therefore, for subclassification tasks, there is currently a lack of a method that directly prunes a large model to obtain a small model through (pruning-fine-tuning).

[0008] The technical solution of this invention:

[0009] To further compress neural networks applied to classification tasks and after complete task pruning, making neural networks deployed on edge devices more lightweight, this invention proposes a deep neural network compression method based on the characteristics of the category set in image classification tasks, specifically including the following steps:

[0010] Step 1: Based on the first training set containing complete category image samples, train the set deep learning neural network model and perform preliminary cropping to obtain the first deep learning neural network model after preliminary cropping of the complete task.

[0011] Step 2: For each category in the complete category, select M image samples; input the M image samples of the i-th category into the first deep learning neural network model, and calculate the importance index of each convolution kernel in the first deep learning neural network model corresponding to the i-th category. in Let K be the importance index of the i-th category corresponding to the first convolutional kernel in the first deep learning neural network model, and K be the number of convolutional kernels in the first deep learning neural network model.

[0012] Step 3: For a specific sub-classification task, determine the N categories that need to be classified in that sub-classification task; superimpose the importance indices of the K convolutional kernels in the first deep learning neural network model corresponding to the N categories to obtain the importance scores of the K convolutional kernels [I1,I2,I3,I4,…,I…]. K ];

[0013] Step 4: Based on the importance scores of the K convolutional kernels obtained in Step 3, prune the convolutional kernels in the first deep learning neural network model, removing several convolutional kernels with lower importance scores. Then, retrain and fine-tune the deep learning neural network model obtained after pruning. Calculate the performance degradation value of the retrained and fine-tuned deep learning neural network model relative to the first deep learning neural network model. If the set performance degradation threshold is not reached, further prune the convolutional kernels in the first deep learning neural network model and retrain and fine-tune the deep learning neural network model obtained after further pruning until the set performance degradation threshold is reached.

[0014] Furthermore, the importance index is the rank of the feature map generated from the image after passing through the first deep learning neural network model.

[0015] Furthermore, in step 4, when retraining and fine-tuning the deep learning neural network model obtained after cropping, the M image samples corresponding to each of the N categories in step 2 are used as training samples.

[0016] Furthermore, in step 2, M image samples corresponding to each category are selected from the first training set.

[0017] Furthermore, in step 2, we take M = 128.

[0018] Furthermore, in step 3, the importance score of the k-th convolutional kernel...

[0019] Furthermore, in step 1, the process of training and initially pruning the established deep learning neural network model is as follows:

[0020] A second deep learning neural network model with Batch Normalization (BN) layers is sparsely trained using the L1 paradigm. The convolutional kernels corresponding to smaller scaling values ​​γ in the BN layers are pruned. The pruned deep learning neural network model is then retrained and fine-tuned. The performance degradation of the retrained and fine-tuned deep learning neural network model relative to the second deep learning neural network model is calculated. If the performance degradation threshold is not reached, the convolutional kernels of the BN layers in the second deep learning neural network model are further pruned, and the further pruned deep learning neural network model is retrained and fine-tuned until the set performance degradation threshold is reached.

[0021] Furthermore, in step 1, the deep learning neural network model set is either a VGG network model or a ResNet network model.

[0022] Beneficial effects

[0023] This invention proposes a deep neural network compression method based on the characteristics of the category set in image classification tasks. It prunes the model according to the categories the model is expected to encounter, achieving a higher pruning rate. The importance of the corresponding convolutional kernel is determined by the rank of the feature map, and low-importance kernels are pruned. When the number of categories to be identified in the classification task decreases, this method can effectively reduce the number of model parameters. Compared to other model pruning methods, the model pruned using this invention is more lightweight.

[0024] Since this invention can obtain the importance of the convolutional kernel in the subclassification task by combining the rank of the feature maps of each category, there is no need to train the entire network again. Compared with training a small model from scratch, this invention is faster.

[0025] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0026] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0027] Figure 1 : Framework diagram of Task-Specific Pruning (TSP) algorithm;

[0028] Figure 2 The pruning efficiency of the algorithm under different network models and datasets: (a) using the VGG16 network model and the CIFAR100 dataset; (b) using the VGG19 network model and the CIFAR100 dataset; (c) using the VGG16 network model and the CIFAR10 dataset; (d) using the ResNet56 network model and the CIFAR10 dataset.

[0029] Figure 3 Comparison of model performance before and after fine-tuning under different category combinations;

[0030] Figure 4 : The first and 16th convolutional layers of a VGG19 network trained on the CIFAR100 dataset; (a) the first convolutional layer, (b) the 16th convolutional layer. Detailed Implementation

[0031] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0032] This embodiment considers a deep learning neural network model deployed on an edge device for an image classification task. Because it's deployed on an edge device, the deep learning neural network model needs to be compressed for the full image classification task. However, for edge devices, in specific application scenarios, certain image categories don't need to be identified. Compared to a deep learning neural network model for the full classification task, the model deployed here can be further compressed. For such sub-classification tasks, there are many combinations; for example, CIFAR100 has 2... 100 With only one category combination, training a compressed small model for each sub-classification task would be very time-consuming. To address this issue, this embodiment proposes a deep neural network compression method based on the characteristics of image classification task category sets. It focuses on CNNs deployed on edge devices that only need to complete a portion of the classification task. Based on different sub-classification task combinations, it dynamically identifies and prunes parts that are irrelevant to the current classification task, further compressing the model size, making the deployed CNNs more lightweight, and improving the running efficiency of the neural network pruning algorithm.

[0033] The deep neural network compression method based on the characteristics of image classification task category sets proposed in this embodiment includes the following steps:

[0034] Step 1: Based on the first training set containing complete category image samples, train and initially crop the set deep learning neural network model to obtain the first deep learning neural network model after initial cropping of the complete task.

[0035] Here, we first use the L1 norm to train the network, making the initial complete network sparse and performing preliminary pruning. Specifically, for a deep learning neural network model with Batch Normalization (BN) layers (hereinafter referred to as the second deep learning neural network model), we use the L1 norm to sparsify and train the second deep learning neural network model, that is, we apply the following sparsity-regularized loss function to train the network weights and these scaling factors together until convergence:

[0036]

[0037] in It is the cross-entropy loss used for image classification. It is a sparsity-induced penalty term for the scaling factor, and λ is used to balance these two terms.

[0038] In deep learning neural network models, batch normalization (BN) is a commonly used technique to improve the convergence speed and accuracy of neural network models. BN layers primarily stabilize the training process by standardizing the input of each layer in the neural network. Next, we will use the scaling factor γ in the BN layer as a reference for initial network pruning.

[0039] The initial pruning method involves iteratively pruning channels associated with smaller scaling factor γ values ​​in the Batch Normalization (BN) layers, and then fine-tuning the pruned model to compensate for the performance loss caused by the removal of convolutional kernels. This iterative pruning process continues until the model's performance falls below a specified threshold. Through this process, a deep learning neural network model after initial pruning of the complete task is obtained (hereinafter referred to as the first deep learning neural network model).

[0040] Next, we will determine the importance of the same convolutional kernel for different image categories. In convolutional neural networks used for image classification tasks, the same convolutional kernel has different importance for different image categories. In this embodiment, the rank of the feature map generated from the image is used as an indicator to measure the importance of the channels that generated the feature map. The higher the rank of the feature map, the richer the information it contains, and therefore the more important the corresponding channels are. The specific principle is analyzed as follows:

[0041] Importance is defined as:

[0042]

[0043] Here, Rank(·) is the rank of the feature map generated from the input image P. By performing Singular Value Decomposition (SVD) on the input image, it can be found that a feature map with rank r can be decomposed into a lower-rank feature map with rank r′ and some additional information. The implementation process of SVD is as follows:

[0044]

[0045] In this formula, r′ <r。δ i u i and v i These are the top-i singular values, left singular vector, and right singular vector of the corresponding feature map, respectively. The SVD results show that higher-order feature maps do indeed contain more information than lower-order feature maps. This indicates that using the rank of a feature map to measure the importance of the corresponding channel is reliable and well-founded.

[0046] Images from the training set are input into the model according to their respective categories, and the rank of the corresponding feature map generated by each channel during the forward propagation is calculated. The rank of the feature map corresponding to each category is defined as...

[0047] However, these ranks are significantly affected by the distribution of the input images. But we found that the ranks can be stabilized by calculating the average rank using the batch size. Our experiments input images with a batch size of 128 into the model, and the results show that the ranks obtained from different batches are almost identical. Therefore, even with a limited number of images obtained from a single category, calculating the rank of the corresponding feature map remains effective. This stability highlights the robustness of the proposed method.

[0048] Figure 4 This diagram shows the first and 16th convolutional layers of a VGG19 network trained on the CIFAR-100 dataset. It illustrates the rank of the feature maps for different classes corresponding to different channels, obtained by training VGG19 on CIFAR-100. In this diagram, the yellower colors represent higher rank values. In the lower layers of the network, completely black columns indicate that the feature maps for these channels are unimportant to any class. In the higher layers of the network, the importance of different classes to different channels begins to change, while some completely black columns still exist. These changes in rank values ​​form the basis for determining which channels should be pruned.

[0049] Based on the requirements of the subclassification task, we performed subclassification on each category. Summation,

[0050]

[0051] This is to determine the importance of each channel in this subcategorization task. Within an acceptable performance degradation range, channels with lower ranks will be pruned and fine-tuned.

[0052] The specific steps are as follows:

[0053] Step 2: For each category in the complete category, select M image samples. In this embodiment, M image samples corresponding to each category are selected from the first training set. Input the M image samples of the i-th category into the first deep learning neural network model, and calculate the rank distribution of each convolution kernel in the first deep learning neural network model under different category image inputs, where the rank distribution corresponding to the i-th category is... in Let K be the rank of the i-th category corresponding to the first convolutional kernel in the first deep learning neural network model, and K be the number of convolutional kernels in the first deep learning neural network model.

[0054] Step 3: Based on the set of sub-classification tasks that the edge device needs to process, for a certain sub-classification task, determine the N categories that need to be classified in that sub-classification task; sum the ranks of the K convolutional kernels in the first deep learning neural network model corresponding to the N categories respectively to obtain the importance scores of the K convolutional kernels [I1,I2,I3,I4,…,I…]. K The importance score of the k-th convolutional kernel.

[0055] For example, if you need to process categories 1, 3, and 7, then select... The importance scores of each convolution kernel are obtained by adding them up digit by digit.

[0056] Step 4: Based on the importance scores of the K convolutional kernels obtained in Step 3, prune the convolutional kernels in the first deep learning neural network model, removing several kernels with lower importance scores. Retrain and fine-tune the pruned deep learning neural network model, calculating the performance degradation of the retrained and fine-tuned model relative to the first deep learning neural network model. If the performance degradation threshold is not reached, further prune the convolutional kernels in the first deep learning neural network model and retrain and fine-tune the further pruned model until the set performance degradation threshold is reached. Note that due to the reduced workload of the subtasks, it is normal for model performance to increase rather than decrease when the number of pruned convolutional kernels is small. In this case, we can continue to prune more convolutional kernels according to the conventional process to achieve a higher pruning rate.

[0057] like Figure 2 As shown, the results of pruning VGG16, VGG19, and ResNet56 trained on CIFAR10 / 100 using the method of this invention are illustrated. The pruning rate here refers to the value achieved when the model performance loss after fine-tuning is within 1%. We measure the pruning rate using the number of channels and FLOPs, and compare the method of this invention (TSP algorithm) with CAP, HRank, and TTP algorithms. It can be seen that on complete datasets, the pruning rate achieved by TSP is slightly lower than that of current mainstream pruning methods. However, when the number of classes retained in the classification task gradually decreases, the pruning rate achieved by TSP is higher than other methods. Figure 2In the CIFAR100-trained VGG16 dataset, when the number of retained classes is <76, the TSP-pruned model is more compact, surpassing the TTP model. On the CIFAR100-trained VGG19 dataset, when the number of classes is <90, the TSP-pruned model's pruning rate exceeds that of the CAP model. Furthermore, we trained a ResNet56 model on CIFAR10, and the experimental results show that the TSP algorithm is very effective for both VGG and ResNet structures. Additionally, the figure shows that increasing the number of target classes leads to an earlier drop in sub-model accuracy, demonstrating a relationship between task complexity and the model parameters required by the sub-model. Sub-models that perceive more classes require more model parameters, and the same pruning ratio results in a greater accuracy loss for models that perceive more classes. Therefore, when applying this method, the more classes the sub-model needs to perceive, the more we need to appropriately reduce the pruning rate to ensure the sub-model's performance.

[0058] Figure 3 The display shows a performance comparison of the model before and after fine-tuning for different class combinations when there are only three classification tasks. It can be seen that regardless of the combination, the Top-1 accuracy of the model before fine-tuning is only about 63%, while the accuracy after fine-tuning can reach about 94%, even higher than the original model's accuracy. This is because TSP explored a better network structure during the pruning process, and the peak performance of the model trained under this structure is slightly improved. Clearly, due to the performance improvement from fine-tuning, it is necessary to perform fine-tuning before deploying the actual model.

[0059] The above results demonstrate the effectiveness of the proposed method for pruning networks. When the number of subclasses in a classification task is small, our algorithm can achieve a higher pruning rate. Furthermore, compared to training and pruning a small model from scratch, this invention is less time-consuming and more efficient.

[0060] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention.

Claims

1. A deep neural network compression method based on the characteristics of image classification task category set, Features: The following steps are involved: Step 1: Based on a first training set containing complete category image samples, the set deep learning neural network model is trained and preliminarily pruned to obtain a first deep learning neural network model after preliminarily pruned for the complete task; Step 2: For each category in the complete category, select M image samples respectively; input the M image samples of the i-th category into the first deep learning neural network model, and calculate the importance index of each convolution kernel in the first deep learning neural network model corresponding to the i-th category in is the importance index of the first convolution kernel in the first deep learning neural network model corresponding to the i-th category, and K is the number of convolution kernels in the first deep learning neural network model; Step 3: For a certain sub-classification task, determine the N categories that need to be classified in the sub-classification task; superimpose the importance indicators of the K convolution kernels in the first deep learning neural network model corresponding to the N categories to obtain the importance scores of the K convolution kernels [I 1 , I 2 , I 3 , I 4 , ..., I K ]; Step 4: According to the importance scores of the K convolution kernels obtained in step 3, the convolution kernels in the first deep learning neural network model are pruned, and several convolution kernels with low importance scores are pruned. The deep learning neural network model obtained after pruning is retrained and fine-tuned, and the model performance degradation value of the retrained and fine-tuned deep learning neural network model relative to the first deep learning neural network model is calculated. If the set model performance degradation threshold is not reached, the convolution kernels in the first deep learning neural network model are further pruned, and the deep learning neural network model obtained after further pruned is retrained and fine-tuned until the set model performance degradation threshold is reached.

2. According to claim 1, a deep neural network compression method based on the characteristics of the image classification task category set, Features: The importance indicator is the rank of the feature map generated by the image after passing through the first deep learning neural network model.

3. A deep neural network compression method based on the characteristics of image classification task category set according to claim 1 or 2, Features: In step 4, when retraining and fine-tuning the deep learning neural network model obtained after cropping, the M image samples corresponding to the N categories mentioned in step 3 in step 2 are used as training samples.

4. A deep neural network compression method based on the characteristics of image classification task category set according to claim 1 or 2, Features: In step 2, M image samples corresponding to each category are selected from the first training set.

5. According to claim 4, a deep neural network compression method based on the characteristics of the image classification task category set, Features: In step 2, take M=128.

6. A deep neural network compression method based on the characteristics of image classification task category set according to claim 1 or 2, Features: In step 3, the importance score of the kth convolution kernel is 7. A deep neural network compression method based on the characteristics of image classification task category set according to claim 1 or 2, Features: In step 1, the process of training and preliminarily pruning the set deep learning neural network model is as follows: Using L 1 Paradigm sparse training is performed on a second deep learning neural network model with a BN layer, and the convolution kernels corresponding to the smaller scaling value γ in the BN layer are trimmed. The deep learning neural network model obtained after the trimming is retrained and fine-tuned, and the model performance degradation value of the retrained and fine-tuned deep learning neural network model relative to the second deep learning neural network model is calculated. If the set model performance degradation threshold is not reached, the convolution kernels of the BN layer in the second deep learning neural network model are further trimmed, and the deep learning neural network model obtained after the further trimming is retrained and fine-tuned until the set model performance degradation threshold is reached.

8. A deep neural network compression method based on the characteristics of image classification task category set according to claim 1 or 2, Features: In step 1, the set deep learning neural network model is a VGG network model or a ResNet network model.

Images