An image classification model compression method based on pruned sub-network features

Abstract

The application discloses a kind of based on pruning subnetwork feature image classification model compression method, this method first selects random original network, and pretraining is obtained in award lottery network.Second, according to the set pruning rate corresponding to the award lottery network selected, using function fitting, obtain network feature representation.Then for random original network (target network), using weight sparsity degree division target network weight frequency, and using fitting function to transform weight frequency, obtain preliminary pruning result, target network parameter sampling in weight range, obtain target pruning subnetwork.Finally, the training of target pruning subnetwork is carried out, the classification of image is completed, and the training accuracy and compression effect of pruning subnetwork with different pruning retention rate are obtained.The application can compress the model to the greatest extent and does not lose too much accuracy, and can be better applied to the edge environment with limited resources.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to an image classification model compression method based on pruned subnetwork features. Background Technology

[0002] With the development of deep learning technology, it has been found that for the same task in a given scenario, large neural networks generally perform better than small neural networks in training. However, large neural networks require higher system performance, higher training energy consumption, and higher training costs. Therefore, the concept of model compression was proposed, aiming to reduce the size of the network while maintaining its accuracy.

[0003] Network pruning is a common technique in model compression. It posits that the importance of each parameter in a neural network varies depending on the task context. Reducing unnecessary weights in the neural network can decrease network size and energy consumption.

[0004] The sparse network obtained by the standard Fine-Tune pruning method (Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135–1143, 2015.), which involves training, pruning, and fine-tuning, can remove more than 90% of the network parameters. However, the network structure obtained by this method does not perform well when trained from scratch. Even if the original training weights are inherited and fine-tuned, more training epochs are required to restore accuracy.

[0005] However, most pruning techniques focus on specifying criteria to find specific subnetworks, neglecting to explore the internal relationships between dense networks and their criterion-compliant candidate subnetworks. Current pruning research points to a pruning theory, the lottery ticket hypothesis (Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.), where subnetworks conforming to this hypothesis are called winning lottery networks. However, winning lottery networks only focus on finding a sparse structure at the cost of full pre-training, which is not common for practical use and for studying the relationships between dense networks and their subnetworks during sparse network training.

[0006] Subsequent researchers have built upon the lottery hypothesis to explore ways to reduce training overhead, including the EB (Early Bird) phenomenon, where winning lottery networks can be quickly acquired during training using extremely low-cost training algorithms. Through research on IMP (Integrated Positive Flow) to find winning lottery networks, the Iterative Synaptic Flow Pruning (SynFlow) algorithm was proposed to prune the initial network without requiring training or data. Yue Bai, Huan Wang, and others proposed the DLTH (Dual Lottery Ticket Hypothesis) as a supplement to the lottery hypothesis, which can transform any sparse network into a winning lottery network using RST (Random Sparse Network Transformation).

[0007] However, current pruning methods based on the lottery hypothesis still focus only on the Prune Criterion to reduce the training overhead of the winning lottery network. Research on the relationship between dense networks and the winning lottery network is relatively superficial, and another advantage of the winning lottery network—that it can compress the network size significantly without sacrificing much accuracy compared to other pruning techniques—has not been discovered. This characteristic is suitable for application scenarios with limited resources, such as edge environments. Summary of the Invention

[0008] To address the aforementioned issues, this invention proposes a method for extracting sparse network features and applying them to image classification model compression. This method introduces a lottery-themed network into the edge domain, extracts features from the pre-trained lottery-themed network, and applies them to the target network. Pruning is performed before training the target network, avoiding the additional training overhead of conventional pruning techniques. This allows the resulting sparse network to inherit the advantages of the lottery-themed network, effectively reducing the number of parameters in conventional classification models and making it well-suited for resource-constrained edge environments.

[0009] The performance of the sub-network determines the performance of the extracted features; therefore, the lottery winning network is used as a sub-network for feature extraction. This invention proposes a fitting pruning strategy: by extracting features from the lottery winning network, these features can be used to prune other networks. This allows other networks to have a large number of parameters pruned before training, and the resulting sparse network can achieve test accuracy that meets or even exceeds that of the original network during retraining. Simultaneously, the model can be compressed to within 0.1% to 1% of the original number of parameters, reducing the burden on edge nodes for training and inference of large model networks.

[0010] The technical solution of the present invention includes the following steps:

[0011] Step (1): The VGG16 model, initialized using batch normalization and Xavier normal distribution weights, is used as the randomized original network. A publicly available image dataset is selected to obtain the training and test sets.

[0012] Step (2): Pre-train the random original network to obtain the winning lottery network.

[0013] Furthermore, the pre-training process is as follows: the randomized original network employs the Iterative Amplitude Pruning (IMP) algorithm, requiring multiple rounds of iterative training. After each round of training, the network is pruned according to the pruning ratio. The size of the randomized original network decreases systematically based on the number of training rounds and the pruning ratio. For example, after 5 rounds of training, with 20% of the weights pruned after each round, the network size decreases sequentially to: 80%, 64%, 51.2%, 40.96%, and 32.77%. The winning lottery network trained using IMP produces different winning lottery network outputs after each round of iteration, and each winning lottery network has a different size.

[0014] Step (3): Set the pruning rate and select the corresponding size of the winning lottery network according to the pruning rate. The pruning rate is implicit in the features extracted from the winning lottery networks of different sizes.

[0015] We use a fitting function to fit the winning lottery network to obtain the representation of the features acquired by the winning lottery network. The fitting function is a bimodal Gaussian distribution, as shown below:

[0016]

[0017] The input to the fitting function is the weight frequency, which represents how many winning lottery network weights exist within a specific weight range. The weight range is determined by weight sparsity. For example, if a lottery network has a weight sparsity of 10000, and a certain weight range is defined as [0.1901, 0.1902], then weights 0.19011 and 0.19012 exist within this range, resulting in a weight frequency of 2. Additionally, the fitting function requires initial parameter values; initial values ​​that are too large or too small will affect the fitting speed. The meanings of the parameters in the fitting function are as follows:

[0018] Where x represents the parameters based on sparse partitioning, μ represents the expected value of the parameter weights, σ ​​represents the standard deviation of the parameter weights, and A is the coefficient. Since the parameter weights of the winning lottery network exhibit a bimodal distribution, a bimodal Gaussian fitting function is used for fitting, where A1,μ1,σ1 are the fitting parameters for the left peak value; and A2,μ2,σ2 are the fitting parameters for the right peak value.

[0019] Step (4): For the random original network, i.e. the target network (the network to be pruned), divide the weight frequencies of the target network using the same weight sparsity as in step (3). Use the fitting function from step (3) to transform the weight frequencies of the target network. The transformation result is the preliminary pruning result. Pruning is applied to the weight frequencies within the weight range.

[0020] The initial pruning targets the weight frequencies. Then, the target network parameters within the weight range need to be sampled to ensure the number of parameters within that range matches the pruned weight frequencies. Random sampling is used here. Parameters not selected for sampling have their weights set to 0 according to the standard pruning criteria (Prune Criterion), thus obtaining the target pruned subnetwork.

[0021] Step (5): Train the target pruning subnetwork using the training set images to complete image classification and obtain the training accuracy and compression effect corresponding to the pruning subnetwork with different pruning rates. Input the test set images into the trained target pruning subnetwork to verify the effect.

[0022] This invention has the following characteristics and beneficial effects:

[0023] 1. Compared with existing technologies, the fitting pruning strategy based on subnetwork feature extraction can compress the model to the greatest extent without losing too much accuracy. It maintains similar accuracy to the original model within the commonly used 10%–20% compression range, and can compress the network by more than ten times, reaching a compression range of less than 1%, based on existing compression rates. Furthermore, the pruning stage occurs before training, i.e., training the sparse network from scratch without additional fine-tuning steps, reducing pruning overhead and making it well-suited for resource-constrained edge environments.

[0024] 2. A network model with maximum parameter compression: Using the fitting and pruning strategy based on feature extraction of pruned sub-networks proposed in this invention, when using the VGG16 model, based on the existing pruning framework, the method proposed in this invention can compress the model to 0.3% to 1% of the original number of parameters while maintaining high accuracy. Even considering significant accuracy loss, the model parameters can be compressed to less than one-thousandth of the original number of parameters. In contrast, the standard Fine-Tune pruning strategy, after compressing to 3% of the original number of parameters, will result in gradient explosion or inability to fit, making training impossible.

[0025] 3. Accuracy is maintained within an acceptable range. Using the fitting and pruning strategy proposed in this invention, the accuracy is similar to that of the original model within a compression range of 10% to 20% (within 0.5% accuracy loss), and training accuracy comparable to the original network is achieved when compressed to 16%. The accuracy loss does not exceed 3% within a compression range of 1% to 10%, and does not exceed 7% within a compression range of 0.3% to 1%. Attached Figure Description

[0026] Figure 1 This is the complete compression method process;

[0027] Figure 2 Initialize the parameter distribution of the network with random weights;

[0028] Figure 3 Initialize the network parameter distribution for Xavier Normal weights;

[0029] Figure 4 The parameter distribution of the winning lottery network;

[0030] Figure 5 This is a representation of the network features of winning lottery tickets extracted through the fitting function;

[0031] Figure 6 The target network parameter distribution obtained using the fitting pruning strategy;

[0032] Figure 7This invention presents a performance comparison of different compression ranges under VGG16 and CIFAR10. Detailed Implementation

[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. The embodiments described are only a part of the embodiments of the present invention and cannot encompass all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0034] This invention discloses an image classification model compression method based on pruned subnetwork features, employing a fitting pruning strategy. This invention addresses the pain points of traditional classification model compression methods: pruning methods typically require training a large neural network first, followed by fine-tuning to restore accuracy. This not only incurs training overhead for model fine-tuning but also often maintains a pruning retention rate above 10%. When the retention rate is too low, it damages the network structure, making training difficult. This invention leverages the advantages of lottery network analysis: it can train a sparse network from scratch and achieve the same training accuracy as the original network within the same iteration time. It discovers that the weight distribution of lottery networks generally follows a specific bimodal distribution. Based on the ideas of knowledge distillation and transfer learning, the features of the lottery network are transferred to different models with the same architecture. The extraction method is through function fitting; this approach of applying extracted lottery network features to other networks is called the fitting pruning strategy. In the fitting pruning strategy, the pruning stage of the target model occurs before the training process, i.e., training a sparse network from scratch, whereas the traditional training-pruning-fine-tuning process prunes after training.

[0035] The reliability and generalizability of this invention were verified using the CIFAR-10 and VGG16 image classification datasets. The Fine-Tune pruning method (Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Advances in neural information processing systems, pp. 1135–1143, 2015.) was selected as a comparison method to test the accuracy of fine-tuning and retraining (Fine-Tune) and retraining accuracy (fitting pruning) under different compression ratios. Experimental results show that, compared with existing compression methods, the proposed compression method can compress the network model to the maximum extent with almost no loss of high accuracy. It can compress the model to three-thousandths of the original model size without losing much accuracy (within 7%). Furthermore, the pruning process occurs before training, without the additional training overhead of fine-tuning steps, making it applicable to edge scenarios with limited computing power and strict space resource requirements.

[0036] See Figure 1 The compression method based on pruning subnetwork feature extraction includes two aspects: a lottery network feature extraction method and a fitting pruning method.

[0037] The specific process of extracting network features from winning lottery tickets is as follows:

[0038] Step 1: Select a random original network and obtain the winning lottery network through IMP.

[0039] The VGG16 model, which uses batch normalization and Xavier normal weight initialization methods, is used as the random primitive network.

[0040] Step 2: Set the parameters for dividing the winning lottery network by weight sparsity.

[0041] Step 3: Fit the network characteristics of the winning lottery ticket using a bimodal Gaussian distribution function.

[0042] The specific process of the fitting pruning method is as follows:

[0043] Step 1: Set the weight sparsity to divide the target network parameters.

[0044] Step 2: Use the fitted function obtained to trim the target network and get the preliminary trimming results.

[0045] Step 3: Random sampling is used within the preliminary pruning results to obtain the final target pruning subnetwork.

[0046] Step 4: Train the acquired target pruning subnetwork to complete image classification, obtain the training accuracy and compression effect of the pruning subnetwork with different pruning retention rates, and verify them.

[0047] Specific process:

[0048] Dataset: CIFAR-10 (Source: This dataset was released in 2009 by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton of the Department of Computer Science at the University of Toronto. The related paper is "Learning Multiple Layers of Features from Tiny Images") is a classic dataset for image classification in the field of computer vision. It contains 60,000 32*32 images in 10 categories, divided into 50,000 training images and 10,000 test images.

[0049] Model selection: The classic VGG16 image classification model was selected (Source: Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556,2014. and Liu Z, Sun M, Zhou T, et al. Rethinking the value of network pruning[J]. arXiv preprint arXiv:1810.05270,2018.), and Batch Normalization and Xavier Normal weight initialization methods were used.

[0050] Pruning methods: The pruning method uses unstructured pruning techniques to trim individual elements of parameters, such as a single weight in a fully connected layer, a single convolution kernel parameter element in a convolutional layer, or a floating-point number in a custom layer.

[0051] The experimental code in this embodiment is implemented in Python 3.9, the deep learning framework is based on PyTorch, and it is deployed on the Ubuntu 18.04LTS operating system, using two Nvidia GTX 2080Ti graphics cards and a 10.2 CUDA processor. The entire model compression process is analyzed below.

[0052] The optimizer used in all training processes is the steepest gradient descent algorithm with momentum, with a learning rate of 0.1, momentum of 0.9, and weight decay set to 10. -4 The method for finding the winning lottery network uses IMP (Iterative Amplitude Pruning), with each IMP iteration taking 160 EPOCH and a pruning rate of 20%, using the original initialization method. To enable finding the lottery with a large learning rate, a Warm-up mode with a learning rate of 0.01 is used for the first 25 EPOCH. A total of 25 IMP iterations are performed, and the winning lottery network after each IMP iteration is recorded for feature extraction.

[0053] All training processes employed Batch Normal layers and Xavier Normal weight initialization. The purpose of weight initialization is to prevent layer activation outputs from exploding or vanishing during the forward propagation of the deep neural network. Using weight initialization effectively improves network training performance. Figure 2 The network parameter distribution without the use of weight initialization techniques shows that the weights are discrete and their upper and lower limits differ significantly. Figure 3 To illustrate the parameter weight distribution of the neural network model after using the Xavier Normal weight initialization technique, the weights are inversely proportional to the square root of the number of units in the previous layer to ensure that the weights remain within a reasonable range. The final parameter distribution of the winning lottery network is shown below. Figure 4 As shown.

[0054] The first step in feature extraction is to set the sparsity to classify the parameters of the neural network model. Setting Sparse = 10000 means that parameters with weight differences within 0.0001 are grouped into the same class. Parameters within the same weight range are represented by the same weight and used as the input variables of the fitting function. The final number of parameter weight ranges obtained is calculated using the following formula:

[0055]

[0056] Where Divides represents the number of weight ranges, Tensor represents the weights of all neural network model parameters, and Sparse represents sparsity. Sparse determines the number of weight ranges: the larger the Sparse, the higher the sparsity, the more weight ranges are divided, and the more difficult the fitting becomes, especially in the case of an extremely sparse network, such as a pruning retention rate of less than 1% and fewer than 100 remaining parameters in some pruned layers. To avoid fitting failures, Sparse is set to 10000.

[0057] After dividing the entire neural network parameters using the above formula, we use all weight ranges as independent variables and the number of parameters within each weight range as the dependent variable to fit a function. The fitting function uses a bimodal Gaussian distribution function.

[0058]

[0059] In addition, initial values ​​for the fitting function parameters need to be provided to facilitate fitting. μ represents the expected value of the neural network model parameter weights, σ ​​represents the standard deviation, and A is the coefficient. μ and σ need to be calculated manually, while A is set manually. The final fitted A is... ′ 1,μ ′ 1,σ1 ′ A ′ 2,μ ′ 2,σ2 ′ This can characterize the features of the winning lottery network, such as Figure 5 As shown.

[0060] After obtaining the features of the winning lottery network, the target model can be fitted and pruned. The first step of fitting and pruning is to divide the model's neural network parameters into weights using the method described above. The frequency of the target model's weights is then used as the input to the fitting function. After mapping by the fitting function, a preliminary pruning result is obtained. The preliminary pruning targets the frequency of the target model's weights, i.e., how many parameters should be retained within the weight range. For example, after dividing the model's parameter weight range, if there are 100 parameters within the weight range [0.1901, 0.1902), after mapping by the fitting function, a new weight frequency of 20 is obtained, meaning that the target model should retain 20 parameters within the weight range [0.1901, 0.1902).

[0061] Next, since the initial pruning results only pertain to the weight frequencies, it is necessary to select parameters based on the new weight frequencies. Here, random sampling is used to randomly retain model parameters that match the new weight frequencies within the weight range. The remaining parameters are frozen as "pruned weights," and the freezing method follows the conventional pruning method of setting them to zero. The final pruning result is shown below, with the parameter weight distribution as follows: Figure 6 As shown.

[0062] and Figure 7To compare the retraining accuracy of the sparse network after fitting and the accuracy after fine-tuning, this example is based on experiments using VGG16 and CIFAR10 with identical hyperparameter settings. The original network training accuracy was 93.86%. Without the technical solution of this invention, retraining after model fine-tuning can only retain about 3% of the parameters. When the compression ratio drops above 97%, the model becomes untrainable. However, using the model compression method of this invention, the accuracy remains close to the original network training accuracy within a compression range of 15%–20%; within 10%–15%, the accuracy loss is within 0.5%; within 1%–10%, the accuracy loss is within 3%; and within 0.3%–1%, the accuracy loss is within 7%, which is within an acceptable range. Furthermore, 99.7% compression is not the compression limit; further pruning is possible, but the accuracy loss will exceed 7%. Therefore, 99.7% can be considered an acceptable level of compression.

[0063] Although embodiments of the present invention have been shown, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A compression method for image classification models based on pruned subnet features, characterized in that, Includes the following steps: Step 1: Use the VGG16 model, which uses batch normalization and Xavier normal distribution weight initialization, as the randomized original network; select a public image dataset to obtain the training set and test set; Step 2: Pre-train the random original network to obtain the winning lottery network; Step 3: Set the pruning rate, select a winning lottery network of the corresponding size according to the pruning rate, and use the fitting function to fit the winning lottery network to obtain the representation of the features acquired by the winning lottery network. The input to the fitting function is the weight frequency, which is the number of winning lottery networks within the weight range. The weight range is divided according to the sparsity of the weights. The pruning rate is implicit in the features extracted from winning lottery networks of different sizes; The fitting function for the network of the winning lottery ticket is a bimodal Gaussian distribution, as shown below: ； in, This represents the parameters based on sparse partitioning. This represents the expected value of the parameter weights. The standard deviation of the parameter weights. For coefficients; These are the fitting parameters for the left peak value; These are the fitting parameters for the right peak value; Step 4: For the random original network, i.e. the target network, use the same weight sparsity as in Step 3 to divide the weight frequency of the target network; use the fitting function in Step 3 to transform the weight frequency of the target network to obtain the preliminary pruning result. The pruning is aimed at the weight frequency within the weight range. The target network parameters within the weight range are sampled to obtain the target pruned subnet; Step 5: Use the training set images to train the target pruning subnet, complete the image classification, and obtain the training accuracy and compression effect of the pruning subnet with different pruning rates; input the test set images into the trained target pruning subnet to verify the effect.

2. The image classification model compression method based on pruned subnet features according to claim 1, characterized in that, In step 2, the pre-training process is as follows: the random original network is trained using the Iterative Amplitude Pruning (IMP) algorithm for multiple rounds of iterative training. After each round of iterative training, the network is pruned based on the pruning ratio. The size of the random original network decreases regularly according to the number of training rounds and the pruning ratio.

3. The image classification model compression method based on pruned subnet features according to claim 2, characterized in that, In step 2, the winning lottery network trained using the IMP algorithm produces different winning lottery network outputs after each iteration, and each winning lottery network has a different size.

4. The image classification model compression method based on pruned subnet features according to claim 1, characterized in that, In step 4, the sampling adopts a random sampling method, and the parameters that are not selected for sampling are weighted to 0 according to the pruning criteria.

Images