An Iterative Random Pruning Method and System Based on Meta-Transfer Learning

Abstract

This invention belongs to the field of neural network technology, specifically relating to an iterative random pruning method and system based on meta-transfer learning. The pruning method of this invention includes the following steps: S1, a transfer learning step, pre-training the convolutional neural network using a large-scale dataset to initialize the network parameters; S2, a meta-learning step, meta-training the network using a small sample dataset; S3, a pruning step, pruning the network weights after meta-training; S4, a testing step, testing the network using a small sample dataset; and S5, a fine-tuning stage, retraining the network to restore its accuracy. Compared to existing technologies, this invention can reduce the number of parameters in a convolutional neural network with small sample datasets while maintaining the accuracy of the network model, achieving the goal of model compression.

Description

Technical Field

[0001] This invention belongs to the field of neural network technology, specifically relating to an iterative random pruning method and system based on meta-transfer learning. Background Technology

[0002] Deep learning is a representative technology in the field of artificial intelligence, with its typical network model being the deep neural network. It has achieved significant breakthroughs in areas such as image processing, speech recognition, and autonomous driving. However, deep learning network models also have some drawbacks, such as a large number of network parameters, computational complexity, and high hardware resource consumption. This makes the deployment of these models in embedded mobile devices extremely difficult and challenging. Therefore, researchers have extensively studied methods for compressing and accelerating deep learning models to address the resource consumption and operational efficiency issues of deep neural networks in practical applications. The main methods include parameter quantization, low-rank decomposition, knowledge distillation, compact network design, and network pruning. Among these, network pruning can remove redundant parameters from the model, effectively reducing network complexity and thus achieving model compression. It can also alleviate overfitting to some extent, making it the most widely used method.

[0003] Existing pruning methods are performed under conditions of large training data volume and relatively complete datasets. However, in real-world applications, a sufficient amount of labeled data cannot be guaranteed, and applying general pruning methods to the trained network model in such scenarios often yields unsatisfactory results, failing to achieve the expected performance. Therefore, in recent years, experts and scholars have begun to research model compression methods for small sample sizes. Based on this, this invention proposes an iterative random pruning method and system based on meta-transfer learning. Summary of the Invention

[0004] To address the shortcomings of the existing technologies, this invention proposes an iterative random pruning method and system based on meta-transfer learning. The aim is to improve the accuracy of convolutional neural networks by leveraging the advantages of meta-learning and transfer learning in small-sample scenarios, while effectively reducing the number of network parameters and achieving model compression through iterative random pruning.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0006] An iterative random pruning method based on meta-transfer learning includes the following steps:

[0007] S1. Pre-train the convolutional neural network using a large-scale dataset to obtain well-initialized weight parameters;

[0008] S2. Use a small sample dataset to perform meta-training on the fine-tuned transfer network to learn features;

[0009] S3. Perform iterative random pruning on the network weights after meta-training to reduce the number of model parameters;

[0010] S4. Test the pruned network using a small sample dataset;

[0011] S5. Retrain the network after testing and restore some of the deleted weights to restore the accuracy of the network model.

[0012] As a preferred embodiment, step S1 includes the following steps in the pre-training process:

[0013] S11. Select a network model, randomly initialize the weight parameters W, pre-train the model on a large-scale dataset D, such as a convolutional neural network on ImageNet, and then calculate the loss L:

[0014]

[0015] Where l(y,f(x)) represents the cross-entropy loss. During training, the weights W are updated using gradient descent, and the updated weights W' are:

[0016]

[0017] S12. Determine the network layers to be trained. Since the subsequent small sample datasets contain different classification targets, the final classifier part of the network needs to be discarded, while freezing all the previous convolutional layers and maintaining the weight parameters.

[0018] S13. Retrain the model to restore its accuracy.

[0019] As a preferred embodiment, step S2, the meta-training process includes the following steps:

[0020] S21. After obtaining the pre-trained network, perform meta-training on it. In this stage, the model will be trained on a small sample training set T. train A set of meta-tasks T is randomly sampled for learning, where T follows the distribution p(T). Each meta-task is then divided into support sets. and query set

[0021] S22. Initialize the new network classifier and use it as the base learner Θ for meta-learning, traversing the support set. For each task, calculate the corresponding loss value and update the base learner parameters θ1 using gradient descent:

[0022]

[0023] Where θ1' represents the updated base learner parameters, and α is the learning rate. for The loss value for θ1.

[0024] S23. A portion of the frozen convolutional layers is used as a meta-learner Φ. The parameters in Φ are learnable, while the parameters of other layers remain unchanged. Then, the query set is processed... Perform forward testing and calculate the corresponding test loss value for Φ. And update its weight parameter θ2. It's worth noting that when updating θ2, the following method is used: and The sum, and the updated weight θ2' is:

[0025]

[0026] As a preferred embodiment, step S3, the iterative random pruning process, includes the following steps:

[0027] S31. Before performing pruning, perform meta-training on the model for several epochs to generate a relatively good initialization, creating a binary mask matrix M for the network. Then, the weight parameters of each layer can be expressed as:

[0028] θ t '=θ t *M

[0029] Where, θ t Let θ be the weight parameters of the t-th convolutional layer before pruning. t ' represents the pruned parameters, and * represents the Hadamard product, which indicates the retention and deletion of weights;

[0030] S32. The pruning is performed in multiple steps. Each time, a value in M ​​is randomly selected according to the proportion p and set to zero to obtain a sparse matrix.

[0031] S33, Using a small sample training set T train Retrain the network to restore the model's accuracy until it converges.

[0032] As a preferred embodiment, step S4, the testing process, includes the following steps:

[0033] To test the network, select test set T. test Divide it into support sets and query set use Train the pruned model, and then... Test the model's ability to quickly adapt to the target task.

[0034] As a preferred embodiment, step S5, the retraining process, includes the following steps:

[0035] Calculate the accuracy decrease ΔAcc of the model before and after pruning. For a small value ε, if the accuracy decrease is too high and ΔAcc > ε, then restore some of the deleted weights, and finally retrain the model to restore the accuracy.

[0036] The present invention also provides an iterative random pruning system based on meta-transfer learning, which is based on the neural network pruning method described in any of the above claims, wherein the neural network pruning system includes:

[0037] The transfer learning module is used to pre-train convolutional neural networks using large-scale datasets to obtain well-initialized weight parameters.

[0038] The meta-learning module is used to perform meta-training on the fine-tuned transfer network using a small sample dataset to learn features.

[0039] The pruning module is used to iteratively and randomly prune the network weights after meta-training in order to reduce the number of model parameters.

[0040] The testing module is used to test the pruned network using a small sample dataset;

[0041] The fine-tuning module is used to retrain the network after testing and restore some of the removed weights to restore the accuracy of the network model.

[0042] Compared with the prior art, the advantages of the present invention are as follows:

[0043] 1. This invention combines the advantages of transfer learning and meta-learning. Transfer learning leverages knowledge and experience learned from large-scale datasets to aid in learning new tasks, thus saving training time and improving model performance. Meta-learning enables the learning of a general learning strategy, allowing this strategy to quickly adapt and learn on unknown tasks. Transfer learning and meta-learning together enable models to converge faster and achieve higher accuracy.

[0044] 2. This invention employs an iterative random pruning method, which effectively reduces the number of model parameters on small sample datasets, while mitigating overfitting and improving accuracy. Most existing network pruning methods are performed when training data is abundant (on large datasets), allowing the model to explore sufficient space and form a policy network to adapt to the task. In this case, the pruning method can accurately determine which weights are important and which are redundant. However, in small sample scenarios, the support set may only contain a few hundred samples at most, insufficient to support the entire model's exploration. Therefore, ordinary pruning methods struggle to assess the importance of network weights, and classification networks trained using these methods often perform poorly when processing query sets. The iterative random pruning method of this invention avoids calculating the importance of model structure, directly generating a sparse structure in the model, thereby reducing the number of model parameters and achieving model compression.

[0045] 3. This invention can improve the accuracy of the model while reducing the number of model parameters. Simulation experiments show that on the miniImageNet dataset, the pruning rates of the ResNet-12 model and the 4CONV model reached 42.04% and 38.39%, respectively, and the accuracy was higher than the benchmark method; on the Omniglot dataset, the pruning rates of the ResNet-12 model and the 4CONV model reached 51.63% and 56.25%, respectively, and the accuracy was improved. Attached Figure Description

[0046] Figure 1 This is a flowchart of an iterative random pruning method based on meta-transfer learning according to a preferred embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the transfer learning steps of the present invention;

[0048] Figure 3 This is a schematic diagram of the testing and fine-tuning steps of the present invention;

[0049] Figure 4 This is a block diagram of an iterative random pruning system based on meta-transfer learning, according to a preferred embodiment of the present invention. Detailed Implementation

[0050] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0051] The preferred embodiment of the present invention is an iterative random pruning method based on meta-transfer learning, which includes first performing transfer learning on a convolutional neural network, pre-training the network using a large-scale dataset, then performing meta-training on the network after transfer learning using a small sample dataset to learn features, then iteratively and randomly pruning the network weights to reduce the number of model parameters, testing the pruned network using a small sample dataset, and finally retraining the tested network and restoring some of the deleted weights to restore the accuracy of the network model.

[0052] Reference Figure 1 This invention provides an iterative random pruning method based on meta-transfer learning, the specific process of which includes the following steps:

[0053] Step 1: Train the convolutional neural network using transfer learning.

[0054] The specific steps are as follows:

[0055] S11. Select a network model, randomly initialize the weight parameters W, download a dedicated large-scale dataset from an open-source website, such as ImageNet, pre-train the convolutional neural network, and then calculate the loss L:

[0056]

[0057] Where l(y,f(x)) represents the cross-entropy loss. During training, the weights W are updated using gradient descent, and the updated weights W' are:

[0058]

[0059] S12. Determine the network layers to be trained. Since the subsequent small sample datasets contain different classification targets, the final classifier part of the network (fully connected layers) needs to be discarded, while freezing all preceding convolutional layers and maintaining the weight parameters, such as... Figure 2 As shown, the weight parameters are saved locally.

[0060] S13. Retrain the model to restore its accuracy.

[0061] Step 2: Perform meta-learning on the trained network. The specific steps are as follows:

[0062] S21. Download dedicated small sample datasets from open-source websites, such as Omniglot and miniImageNet. Divide the small sample dataset into a training set T. train and test set T test T train and T testThe sample classes in the dataset have no overlap, and then the pre-trained network undergoes meta-training. At this stage, the model is trained on a small sample training set T. train A set of randomized meta-tasks T is selected for training, where T follows the distribution p(T). Each meta-task is then divided into support sets. and query set

[0063] S22. Initialize the new network classifier and use it as the base learner Θ for meta-learning, traversing the support set. For each task, calculate the corresponding loss value and update the base learner parameters θ1 using gradient descent:

[0064]

[0065] Where θ1' represents the updated base learner parameters, and α is the learning rate. for The loss value for θ1.

[0066] S23. A portion of the frozen convolutional layers is used as a meta-learner Φ. The parameters in Φ are learnable, while the parameters of other layers remain unchanged. Then, the query set is processed... Perform forward testing and calculate the corresponding test loss value for Φ. And update its weight parameter θ2. It's worth noting that when updating θ2, the following method is used: and The sum, and the updated weight θ2' is:

[0067]

[0068] Step 3: Perform iterative random pruning on the network.

[0069] S31. Create a binary mask matrix M for the network, then the weight parameters of each layer can be expressed as:

[0070] θ t '=θ t *M

[0071] Where, θ t Let θ be the weight parameters of the t-th convolutional layer before pruning. t ' represents the pruned parameters, and * represents the Hadamard product, which indicates the retention and deletion of weights;

[0072] S32. Using an iterative approach, the pruning is divided into multiple executions. Each time, a value in M ​​is randomly selected and set to zero according to the p ratio to obtain a sparse matrix.

[0073] S33, Using a small sample training set T train Retrain the network to restore the model's accuracy until it converges.

[0074] Step 4: Test the pruned network, referring to... Figure 3 .

[0075] S41. Select the test set T from the small sample dataset. test Divide it into support sets and query set

[0076] S42, Utilization The pruned model is trained and its parameters are updated using gradient descent to learn the features of a small sample dataset.

[0077] S43, Through The test model's ability to quickly adapt to the target task, among which... The class is an unknown class, and The classes in the model have no overlap, and the accuracy of the model is calculated.

[0078] Step 5: Make final fine-tuning of the network, referring to... Figure 3 .

[0079] S51. In order to prevent the accuracy of the model from dropping too much after pruning, which would lead to irrecoverable performance, it is necessary to calculate the accuracy drop ΔAcc of the model before and after pruning.

[0080] S52. For a small value ε, if the accuracy of the model drops too much, resulting in ΔAcc > ε, then some of the deleted weights are restored, and the model is retrained for several cycles to restore accuracy.

[0081] S53, Reuse Test the model by repeating step S52 until ΔAcc < ε, then output the model.

[0082] The effects of the above embodiments of the present invention are further illustrated by the following simulations:

[0083] 1. Simulation experimental conditions:

[0084] The neural network pruning method of this invention can be implemented in Python, and the encoding and calculation involved in the algorithm are implemented using the NumPy library.

[0085] The publicly available datasets used in the embodiments of this invention include: ImageNet, miniImageNet, and Omniglot.

[0086] The network structures mainly used in the embodiments of this invention are ResNet-12 and 4CONV, both of which can be built using the PyTorch library.

[0087] 2. Simulation Experiment Content:

[0088] The convolutional neural network was pre-trained using the large-scale ImageNet dataset, and then the experimental results for 5-way 1-shot and 5-way 5-shot tests were performed on the miniImageNet few-shot dataset, as shown in Table 1. The experimental results for 20-way 1-shot and 20-way 5-shot tests were performed on the Omniglot few-shot dataset, as shown in Table 2.

[0089] Table 1. Pruning Results on miniImageNet

[0090]

[0091] As shown in Table 1, on the miniImageNet dataset, the present invention achieves a pruning rate of 42.04% on the ResNet-12 model, with accuracy improvements of approximately 9.24% and 9.98% compared to the baseline method in 5-way 1-shot and 5-way 5-shot tests, respectively. On the 4CONV model, the present invention achieves a pruning rate of 38.39%, with accuracy improvements of approximately 4.17% and 4.48% compared to the baseline method in 5-way 1-shot and 5-way 5-shot tests, respectively. This indicates that the present invention can effectively reduce the number of model parameters and alleviate overfitting problems.

[0092] Table 2. Pruning results on Omniglot

[0093]

[0094] As shown in Table 2, on the Omniglot dataset, the present invention achieves a pruning rate of 51.63% on the ResNet-12 model, with accuracy improvements of approximately 7.33% and 0.4% compared to the baseline method in 20-way 1-shot and 20-way 5-shot tests, respectively. On the 4CONV model, the present invention achieves a pruning rate of 56.25%, with accuracy improvements of approximately 4.17% and 2.48% compared to the baseline method in 20-way 1-shot and 20-way 5-shot tests, respectively. This indicates that the present invention can effectively reduce the number of model parameters and alleviate overfitting problems.

[0095] like Figure 4As shown, a preferred embodiment of the present invention also provides an iterative random pruning system based on meta-transfer learning. Based on the above pruning method, the pruning system includes:

[0096] The transfer learning module is used to pre-train convolutional neural networks using large-scale datasets to obtain well-initialized weight parameters.

[0097] The meta-learning module is used to perform meta-training on the fine-tuned transfer network using a small sample dataset to learn features.

[0098] The pruning module is used to iteratively and randomly prune the network weights after meta-training in order to reduce the number of model parameters.

[0099] The testing module is used to test the pruned network using a small sample dataset;

[0100] The fine-tuning module is used to retrain the network after testing and restore some of the removed weights to restore the accuracy of the network model.

[0101] Other aspects of this embodiment can be found in the foregoing pruning method embodiment.

[0102] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. An iterative random pruning method based on meta-transfer learning, characterized in that, Includes the following steps: S1. Pre-train the convolutional neural network using a large-scale dataset to obtain the initialized weight parameters; S2. Use a small sample dataset to perform meta-training on the network after transfer learning to learn features; S3. Perform iterative random pruning on the network weights after meta-training; S4. Test the pruned network using a small sample dataset; S5. Retrain the network after testing and restore some of the deleted weights; Step S1 specifically includes: S11. Select a network model and randomly initialize the weight parameters W on a large-scale dataset. Use a pre-trained model and then calculate the loss. : in, This represents the cross-entropy loss; during training, the weights W are updated using gradient descent, and the updated weights... for: S12. Determine the network layers to be trained: Discard the last classifier part of the network, while freezing all the previous convolutional layers and maintaining the weight parameters. S13. Retrain the model to restore its accuracy; Step S2 specifically includes: S21, The model is trained on a small sample set. Randomly sample a set of meta-tasks To learn, each meta-task is divided into support sets. and query set ; S22. Initialize the new network classifier and use it as the base learner for meta-learning. traverse the support set For each task, calculate the corresponding loss value and update the base learner parameters using gradient descent. : in, These are the updated base learner parameters, where α is the learning rate. for about The loss value; S23. Separate a portion from the frozen convolutional layer as a meta-learner. , The parameters in the query set can be learned, while those in other layers remain unchanged; then the query set... Perform forward testing and calculate Corresponding test loss value and update its weight parameters. In updating When using and The sum, the updated weights for: Step S3 includes: S31. Create a binary mask matrix M for the network, then the weight parameters of each layer are represented as follows: in, Let be the weight parameters of the t-th convolutional layer before pruning. The parameters are after pruning, and * represents the Hadamard product, which indicates the retention and deletion of weights. S32. Randomly select values ​​with a proportion of p in the mask matrix M and set them to zero to obtain a sparse matrix; S33. Using a small sample training set Retrain the network to restore the model's accuracy.

2. The iterative random pruning method based on meta-transfer learning according to claim 1, characterized in that, Step S4 is as follows: Test the trained meta-learner by selecting a test set. Divided into support sets and query set ,use Train the pruned model, and then... Test the model's ability to quickly adapt to the target task.

3. The iterative random pruning method based on meta-transfer learning according to claim 1, characterized in that, Step S5 is as follows: Calculate the decrease in model accuracy after pruning. If the accuracy drops too much and leads to , If the threshold is set, then the deleted weights are recovered, and finally the model is retrained to restore accuracy.

4. A system based on the iterative random pruning method according to any one of claims 1-3, characterized in that, Includes the following modules: The transfer learning module is used to pre-train convolutional neural networks using large-scale datasets to obtain initialized weight parameters; The meta-learning module is used to perform meta-training on the fine-tuned transfer network using a small sample dataset to learn features. The pruning module is used to perform iterative random pruning of the network weights after meta-training. The testing module is used to test the pruned network using a small sample dataset; The fine-tuning module is used to retrain the network after testing and restore some of the removed weights.

Images