A compression method for image classification models based on a greedy strategy

Abstract

This invention discloses a compression method for image classification models based on a greedy strategy. By using an iterative greedy approach, this invention sets relatively small training iterations, accuracy thresholds, and pruning rate thresholds, thus mitigating the pain points of pruning. Furthermore, the initial neural network is used as the teacher network in knowledge distillation for model fine-tuning, allowing the pruning process to select the network structure, reducing the dependence of knowledge distillation on network structure selection. The reliability and generality of this invention are verified using the small image classification dataset CIFAR-10 and the large dataset ImageNet. Experimental results, using different pruning strategies and models, demonstrate that the proposed compression method, compared to existing methods, can maximize the search for lightweight network models with almost no loss of high accuracy. The resulting classification model not only has lower parameter and computational costs but also faster inference speed, making it applicable to scenarios with strict requirements for real-time performance and spatial resources.

Description

Technical Field

[0001] This invention belongs to the field of network compression technology, and specifically refers to a compression method based on an image classification model using a greedy strategy. Background Technology

[0002] While complex convolutional neural network (CNN) models offer superior performance in image classification, their increasing depth and complexity present challenges. Because complex CNN models often require significant computation and memory, they cannot be directly applied in industrial settings. Researchers have proposed model compression methods to improve computational efficiency and conserve memory by minimizing model complexity without compromising accuracy, thereby enabling the practical application of neural networks across various fields. This invention primarily involves two model compression methods: pruning and knowledge distillation.

[0003] Neural network pruning methods preserve some important network parameters while removing the rest, thereby simplifying the network structure and achieving model compression. Research on neural network pruning focuses on two aspects: fine-grained pruning of network nodes and connections, and coarse-grained pruning of channels and filters. Han (Source: Song Han, Jeff Pool, John Tran, and William J. Daily. Learning both weights and connections for efficient neural networks. 2015.) was one of the first to propose a fine-grained pruning strategy: first, learn the network connections, and then prune the connections according to a threshold. This pruning transforms dense, fully connected layers into sparse layers. Therefore, unstructured pruning can result in irregular sparse weight matrices in the network, which places high demands on hardware. Consequently, more researchers have focused on structural pruning. Many scholars have proposed a filter-level pruning strategy, setting the pruning rate based on the differences in the sensitivity of each convolutional layer in the network to pruning. ThiNet (Source: Luo, Jian-Hao, Jianxin Wu, and Weiyao Lin. Thinnet: A filter-level pruning method for deep neural network compression. Proceedings of the IEEE international conference on computer vision. 2017) further proposed the idea of ​​fine-tuning the model, training for only one or two epochs after structural pruning of each layer, and then training for a sufficient number of epochs after the entire network has been pruned.

[0004] Unlike traditional methods of training network models, knowledge distillation trains models using a "teacher-student network" approach (Source: Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. 2015.). This involves first training a complex model as the teacher model, and then designing a small neural network as the student model. During training, the student model is influenced not only by the "hard labels" of the samples but also by the "soft labels" of the teacher model's output. This method aims to make the student model's output distribution as similar as possible to the teacher model, thereby achieving model compression. Ba (Source: Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep?, 2013.) proposed a training method for student networks: using a pre-trained deep network (teacher network) and unlabeled data to jointly train the student network. Romero et al. (Source: Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. 2014.) further proposed that not only should the output of the teacher network be used as the content for the student network to learn, but the student network should also learn from the teacher network in the representation of the intermediate hidden layers. The knowledge distillation training mode will result in a lightweight network with high network performance, but it does not solve the problem of potential parameter redundancy in the network.

[0005] Both of these traditional model compression methods have their shortcomings. For example, pruning methods usually require resources to train a large neural network first and a given pruning rate. In experiments, the following phenomenon is often observed: if the pruning rate is too high, the network cannot be recovered, while if the pruning rate is too low, there are too many redundant parameters. As for knowledge distillation, although it can train a small neural network, it relies too much on the choice of network structure and requires repeated parameter tuning to find a high-quality network structure. Summary of the Invention

[0006] This invention proposes a compression method for image classification models based on a greedy strategy, which can effectively reduce the computational cost and parameter count of conventional classification models. Moreover, the compressed model does not depend on specific pruning methods or hyperparameters during training.

[0007] The technical solution steps of this invention are as follows:

[0008] Step (1): Select the CIFAR-10 ten-class dataset and the ImageNet thousand-class dataset as the training set, validation set and test set of the network.

[0009] Step (2): Load the pre-trained parameters into the classic image classification model with residual structure.

[0010] Step (3): Train the image classification model network until it converges, and use it as the teacher network and the initial network.

[0011] Step (4): Use a channel-based or convolutional kernel-based structured pruning method to prune the initial network, set the pruning rate for each pruning, and set the accuracy loss threshold and the total pruning rate threshold.

[0012] Step (5): The teacher network is used to restore the initial network after the current pruning, and the image classification model restored after pruning is obtained.

[0013] Step (6): Enter the iterative model compression framework. After each pruning, the teacher network is retrained for several epochs to restore accuracy before entering the next pruning. This continues until the accuracy loss reaches a preset threshold or the pruning rate meets the requirements, at which point the iterative framework is exited and the compressed network is obtained.

[0014] Step (7): The computational load is measured in FLOPs, i.e., the computational load of forward inference; the number of parameters is measured in M ​​(megabytes); the accuracy evaluation method varies depending on the dataset used, as detailed below:

[0015] CIFAR-10 calculates accuracy by comparing the network's predictions with the actual labels, and the percentage of correct predictions out of the total number of tests is considered accurate. ImageNet, on the other hand, evaluates accuracy based on the first and top five predicted categories, namely Top-1Acc and Top-5Acc. Top-1Acc means the prediction with the highest probability is correct, while Top-5Acc means that at least one of the top five predicted categories is correct.

[0016] Step (8): Compare the number of parameters, computational cost, and accuracy of the network compressed by this method, the initial network, and the network compressed using the basic method.

[0017] Furthermore, the outputs of the teacher network and student network are first normalized using the softmax function of the output layer. Let x... i The input image data is f(x), which represents the output of the entire network. T is the temperature coefficient, used in knowledge distillation to soften the output.

[0018]

[0019] In classification problems, t u Let represent the output of the u-th category, and N represent the number of categories. Therefore, the softmax function used is as follows:

[0020]

[0021] The model fine-tuning based on knowledge distillation is used for retraining the pruned network. In this invention, only 5 epochs are trained. The loss function during training involves the temperature coefficient T, KL divergence, and cross-entropy, as follows:

[0022] L kd =α×T 2 ×KLDivLoss(Q S Q T )+(1-α)×H(y s ,y T )

[0023] Where α is the smoothing coefficient, which is set to 0.95 in this invention; T is the temperature coefficient in knowledge distillation, which is set to 1 in this invention; KLDivLoss is the first loss function, calculated based on the KL divergence of the softmax output Q of the student network and the teacher network; and H is the second loss function, calculated based on the cross-entropy of the initial output y of the student network and the teacher network.

[0024] The beneficial effects of this invention are:

[0025] Compared to existing technologies, the image classification model compression method based on a greedy strategy can maximize the search for lightweight network models with almost no loss of high accuracy. The resulting classification model not only has fewer parameters and lower computational cost but also faster inference speed, making it applicable to scenarios with strict requirements for real-time performance and spatial resources. While the teacher network in the classic knowledge distillation architecture is directly used to lightweight the network, in this invention, the teacher network is used to restore the accuracy of the pruned network, resulting in the pruned and restored network model.

[0026] (1) Network models with lower parameter and computational complexity

[0027] After using the image classification model compression method based on the greedy strategy of this invention, whether it is ResNet or DenseNet, compared with the existing pruning framework, the method proposed in this invention reduces the computational cost and number of parameters of the network model as much as possible, whether using pruning filters or network slimming, and the accuracy does not decrease significantly. On the contrary, the accuracy is sometimes improved due to the effect of knowledge distillation. For example, when using the pruning filter method to prune ResNet-56 and setting the pruning rate of the convolutional layer to 0.1 for each pruning, this invention can ultimately reduce the number of network parameters to two-thirds of the original, while improving the accuracy by 0.5%.

[0028] (2) Accuracy is maintained within an acceptable range

[0029] Figure 5 This is a comparative experiment using three methods on the ImageNet dataset. This experiment is based on ResNet-56, and the pruning rate and training process are the same as those used on the CIFAR-10 dataset. The difference is that a model fine-tuning experiment with the removal of the teacher network was added for comparison. Taking PFEC-A as an example, when the model was fine-tuned without using this method, the Top-5Acc dropped from 91.42% to 89.19%, while the model compression method of this invention can maintain the accuracy to 90.43%. It can be seen that this invention maintains the accuracy of the model while based on greedy pruning. Attached Figure Description

[0030] Figure 1 This is the flow chart of the image classification model compression method based on the greedy strategy of the present invention;

[0031] Figure 2 This invention employs the principle of knowledge distillation.

[0032] Figure 3 This is the principle behind the fine-tuning of the image classification model after each pruning in this invention;

[0033] Figure 4 This invention is based on the small dataset CIFAR-10, comparing the performance of different image classification models under different compression methods;

[0034] Figure 5 This invention is based on a performance comparison of ImageNet and ResNet-34 on different compression methods using large datasets. Detailed Implementation

[0035] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. The embodiments described are only a part of the embodiments of the present invention and cannot encompass all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0036] This invention discloses a compression method for image classification models based on a greedy strategy. This invention alleviates the pain points of traditional classification model compression methods: pruning methods typically require training a large neural network first, and the pruning rate is fixed, which is quite limiting. In experiments, the following phenomena are frequently observed: an excessively high pruning rate makes the network unrecoverable, while an excessively low pruning rate results in too many redundant parameters; while knowledge distillation can train a small neural network, it relies excessively on network structure selection, requiring repeated parameter tuning to continuously find a high-quality network structure. This invention alleviates the pain points of pruning by using an iterative greedy approach, setting a smaller number of training iterations, accuracy, and pruning rate thresholds. Furthermore, the initial neural network is used as the teacher network in knowledge distillation for model fine-tuning, allowing the pruning process to select the network structure, thus reducing the dependence of knowledge distillation on network structure selection. The reliability and general applicability of this invention have been verified using the small image classification dataset CIFAR-10 and the large dataset ImageNet. Different pruning strategies and models have been selected. Experimental results show that, compared with existing compression methods, the compression method proposed in this invention can search for lightweight network models to the greatest extent with almost no loss of high accuracy. The resulting classification model not only has a lower number of parameters and computational cost, but also a faster inference speed, and can be applied to scenarios with strict requirements for real-time performance and spatial resources.

[0037] See Figure 1-3 The compression method for image classification models based on greedy strategies includes two aspects: a coarse-grained pruning method based on channels and convolution kernels, and a model fine-tuning method based on knowledge distillation.

[0038] The specific process of coarse-grained pruning based on channels and convolution kernels is as follows:

[0039] Step 1: Importance assessment of channels and convolutional kernels

[0040] Step 2: Remove unimportant channels and convolution kernels

[0041] Step 3: Model fine-tuning based on knowledge distillation

[0042] Step 4: After pruning, use the model as the new initial model and repeat the first three steps.

[0043] The specific process of the model fine-tuning method based on knowledge distillation is as follows:

[0044] Step 1: Extract the softmax values ​​from the student network and the teacher network outputs.

[0045] Step 2: Calculate the loss based on both hard and soft labels.

[0046] Step 3: Fine-tuning network parameters

[0047] Step 4: Repeat the above steps several times.

[0048] Specific process:

[0049] (1) Dataset

[0050] CIFAR-10 (Source: KRIZHEVSKY A. Learning multiple layers of features from tiny images[D]. University of Toronto, 2009.) and ImageNet (Source: RUSSAKOVSKY O, DENGJ, SU H, et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115(3):211-252.) contains RGB color images in 10 categories, such as airplanes, cars, etc. The dataset contains 50,000 training images and 10,000 test images. The ImageNet dataset is a classification dataset released by ILSVRC in 2012. The training set contains 1.3 million images and labels, and the test and validation sets contain 50,000 images and labels. Regarding the processing of the dataset, the samples in the training set, validation set, and test set were first normalized and preprocessed. In order to enrich the training samples, data augmentation was performed on the CIFAR-10 and ImageNet datasets. Augmentation methods included random flipping and random cropping.

[0051] (2) Model selection

[0052] The compression method based on the greedy strategy image classification model in this invention was tested using ResNet (Source: HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition(CVPR).2016:770-778.) and DenseNet (Source: Gao Huang, Zhuang Liu, Laurens Van, DerMaaten, and Kilian Q Weinberger. Densely connected convolutional networks[C].In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708,2017.) network models, covering the mainstream residual neural networks. Because the residual structure effectively prevents overfitting and network degradation, it is currently a widely used convolutional neural network.

[0053] (3) Pruning methods

[0054] Network slimming (Source: Liu, Zhuang, et al. Learning efficient convolutional networks through network slimming[C]. Proceedings of the IEEE International Conference on Computer Vision.2017.): The importance of channels is evaluated by the gamma coefficient in the Batch Normalization (BN) layer, and an L1 regularization constraint for the gamma coefficient is added to the loss function to make the model automatically sparsify. After training, many gamma coefficient values ​​in the network model are fixed to zero.

[0055] Pruning filters (Source: Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, Hans Peter Graf. Pruning filters for efficient convnets. 2017.): Evaluate the importance of the convolution kernel based on the absolute sum of all parameters of the convolution kernel, and perform pruning operations by sorting them according to the absolute sum.

[0056] (4) Implementation details

[0057] The experimental code for this invention is implemented in Python 3.8, using the PyTorch deep learning framework, deployed on an Ubuntu 20.04 LTS operating system, and equipped with four Nvidia GTX 1080Ti graphics cards and an 11.2 CUDA processor. The entire model compression process is analyzed below. All training processes use the steepest gradient descent algorithm with momentum, with a learning rate of 0.1, momentum of 0.9, and weight decay set to 10. -4 The total training iterations (excluding model fine-tuning) consisted of 160 epochs, with the learning rate decaying by a factor of 10 every 50 epochs. First, a pre-trained ResNet-34 model was read from the PyTorch website and trained on the CIFAR-10 and ImageNet datasets respectively. The resulting network parameters were stored locally as the initial network and the teacher network.

[0058] like Figure 2 In this invention, the outputs of the teacher network and student network are first normalized using the softmax function of the output layer. xi is the input image data, f(x) represents the entire ResNet network, and T is a temperature coefficient used in knowledge distillation to soften the output; in this experiment, it is set to 1.

[0059]

[0060] The softmax function is as follows, in classification problems t u This represents the output of the u-th category, where N represents the number of categories:

[0061]

[0062] The model fine-tuning based on knowledge distillation is trained for only 5 epochs. The loss function involves the temperature coefficient T, KL divergence, and cross-entropy, as follows:

[0063] L kd =α×T 2 ×KLDivLoss(Q S Q T )+(1-α)×H(ys ,y T )

[0064] Where α is the smoothing coefficient, which is set to 0.95 in this invention; T is the temperature coefficient in knowledge distillation, which is set to 1 in this invention; KLDivLoss is the first loss function, calculated based on the KL divergence of the softmax outputs of the student network and the teacher network; and H is the second loss function, calculated based on the cross-entropy of the initial outputs of the student network and the teacher network.

[0065] The pruning methods used in the pruning framework are all coarse-grained, meaning they determine which layers need pruning based on the importance of channels and convolutional kernels. This invention uses two methods: Pruning filters (hereinafter referred to as PFEC) and Network slimming (hereinafter referred to as Slimming), to prune convolutional kernels and channels respectively. Since pruning a layer has varying effects on the network, and pruning one layer can affect the next, the pruning rate needs to be set according to the pruning sensitivity for different network models. For example, in ResNet-56, the pruning rate for layers 1-18 is higher than that for layers 19-56. Different pruning rates set in the experiments are distinguished by uppercase letters, such as PFEC-A.

[0066] The comparative experiments are based on the following scenarios: only pruning without using the algorithm of this invention, using the compression method of this invention, and using the compression method of this invention but without using a teacher network in the process of retraining to restore accuracy.

[0067] Figure 4This section describes the network performance difference between the CIFAR-10 dataset using general pruning methods and the method described in this invention. When using the PFEC method, the number of layers corresponding to the pruning rate varies depending on the selected model: for ResNet-56, it corresponds to layers 1-18, 19-36, and 37-56, with PFEC-A indicating pruning rates of 0.1, 0.1, and 0.1 respectively, and PFEC-B indicating pruning rates of 0.6, 0.3, and 0.1; for ResNet-110, it corresponds to layers 1-36, 37-72, and 73-108, skipping the last two layers, with PFEC-A indicating pruning rates of 0.5, 0, and 0 respectively, and PFEC-B indicating pruning rates of 0.5, 0.4, and 0.3. When using the Sliming-A method, the scaling factor is set to 0.5. After using the greedy strategy-based model compression method of this invention, the computational cost and number of parameters of the network model are significantly reduced, while the accuracy does not decrease significantly. On the contrary, the accuracy is occasionally improved due to the effect of knowledge distillation (ResNet-56). Taking ResNet-110 as an example, the number of parameters of the basic model is 1.73M, the number of Flops is 0.51G, and the accuracy reaches 93.79%. After using the method of this invention, the number of parameters is compressed to 0.39M (only 1.17M without this algorithm), and the accuracy decreases by 0.65% (0.12% without this algorithm).

[0068] and Figure 5 This is a comparative experiment using three methods on the ImageNet dataset. This experiment is based on ResNet-56, and the pruning rate and training process are the same as those used on the CIFAR-10 dataset. The difference is that a model fine-tuning experiment with the removal of the teacher network was added for comparison. Taking PFEC-A as an example, when the model was fine-tuned without using this method, the Top-5Acc dropped from 91.42% to 89.19%, while the model compression method of this invention can maintain the accuracy at 90.43%.

[0069] Although embodiments of the present invention have been shown, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for compressing an image classification model based on a greedy strategy, characterized in that The method comprises the following steps: Step (1), CIFAR-10 ten-class data set and ImageNet thousand-class big data set are selected as the training set, the verification set and the test set of the network; Step (2), the pre-training parameters of a classical image classification model with a residual structure are loaded; Step (3), the image classification model network is trained to convergence, and is used as a teacher network and an initial network; Step (4), a structured pruning method based on a channel or a convolution kernel is used to prune the initial network, a pruning rate of each pruning is set, an accuracy loss threshold and a total pruning rate threshold are set; Step (5), the teacher network is used to restore the initial network after the current pruning, and a pruned image classification model is obtained; Step (6), an iterative model compression framework is entered, after each pruning, the teacher network is used to retrain the restored network for several epochs to recover the network accuracy, and then the next pruning is entered; until the accuracy loss reaches the pre-set threshold or the pruning rate reaches the requirement, the iterative framework is exited and the compressed network is obtained.

2. The compression method of the image classification model based on the greedy strategy according to claim 1, wherein the recovery of the network accuracy in step (6) is specifically implemented as follows: The outputs of the teacher network and the student network are first normalized using the softmax function of the output layer, let x i be the input image data, f(x) represent the output of the entire network, and T be the temperature coefficient, which is used to soften the output in knowledge distillation: In the classification problem t u represents the output of the u-th classification, N represents the number of classifications, and thus the softmax function used is as follows: The model fine-tuning based on knowledge distillation is used for the retraining of the pruned network, and the loss function involved in the training includes a temperature coefficient T, a KL divergence and a cross entropy, as follows: L kd = a x T 2 x KLDivLoss(Q S , Q T ) + (1 - a) x H(y s , y T ) wherein α is a smoothing coefficient, KLDivLoss is a first loss function, and is calculated according to the KL divergence of the softmax output Q of the student network and the teacher network; H is a second loss function, and is calculated according to the cross entropy of the initial output y of the student network and the teacher network.

Images