A knowledge distillation-based class-incremental learning method
By transferring knowledge from the teacher network to the Mobilenet_v2 student network using the knowledge distillation method, the problem of high computational cost and high resource consumption in incremental learning is solved. This achieves efficient knowledge transfer and improved prediction performance between small and large models, and is suitable for mobile deployment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIANGSU UNIV OF SCI & TECH
- Filing Date
- 2022-06-23
- Publication Date
- 2026-05-22
AI Technical Summary
Existing technologies in deep learning, especially incremental learning-like technologies, involve large computational demands, high resource consumption, and significant privacy risks, making it difficult to achieve efficient knowledge transfer between small and large models and improve prediction performance.
We employ a knowledge distillation method, using MobileNet_v2 as the student network and ResNet34 as the teacher network. With a tenfold reduction in the number of parameters, we transfer knowledge from the teacher network to the student network through knowledge distillation, thereby compressing the network model. We then train the model within a fine-tuning framework, reducing computational and memory overhead.
While reducing computing resources and memory overhead, student networks can achieve predictive performance comparable to teacher networks, are suitable for mobile deployment, and avoid privacy issues.
Smart Images
Figure CN115170872B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision, and specifically to a class incremental learning method based on knowledge distillation. Background Technology
[0002] In the context of deep learning, to achieve better prediction results, the following measures are typically taken to train the network: 1. Using overparameterized deep neural networks, i.e., expanding the network model by increasing the number of layers as much as possible. 2. Ensemble models, i.e., integrating multiple weak models to enhance prediction performance. However, both methods have obvious drawbacks, namely, the large amount of computation and resources required. Driven by this drawback, related research has turned to the field of model compression, i.e., using a smaller, shallower network model to achieve the same or comparable prediction results as a larger model. Obviously, training a small model from scratch is difficult to achieve the desired effect. Therefore, the technical approach is to train a large model and transfer its knowledge to a smaller model, which is the specific practice of knowledge distillation.
[0003] Incremental learning refers to a learning system's ability to continuously learn new knowledge from new samples while retaining most of the previously learned knowledge. The stability-plasticity curve is typically used to measure the incremental learning capability of a network model. If a network model is required to simultaneously learn new tasks and retain performance on old tasks, larger network models generally perform better. However, under incremental settings, the computational cost and resource requirements increase linearly. Therefore, reducing computational cost and conserving computational resources has always been a major research focus in this field. This invention is based on knowledge distillation, using MobileNet v2 as the student network and ResNet34 as the teacher network. With a tenfold reduction in the number of parameters, it achieves prediction performance comparable to the teacher network. Furthermore, this invention does not require storing samples or expanding the model, significantly reducing memory overhead and avoiding privacy issues.
[0004] Currently, incremental learning is mainly divided into two settings: task-based incremental learning and class-based incremental learning. The training process for both settings is identical; the main difference lies in the testing phase: task-based incremental learning can obtain the task ID of the current image, while class-based incremental learning cannot. Clearly, class-based incremental learning is a more challenging learning task. Therefore, this invention primarily focuses on improving the prediction performance of class-based incremental learning. Simultaneously, the prediction results of the task-based incremental learning setting are calculated during testing, and experimental results show that it also has a good improvement effect on the task-based incremental learning setting. The simplest fine-tuning is chosen within the training framework of the student network to minimize computational cost. Summary of the Invention
[0005] Purpose of the invention: The purpose of this invention is to provide a knowledge distillation-based incremental learning algorithm that reduces computational load, saves computational resources and memory overhead, and avoids privacy leaks.
[0006] Technical Solution: A knowledge distillation-based incremental learning algorithm involves training a teacher network model using ResNet34 on datasets such as CIFAR-100 and Tiny-ImageNet under incremental learning settings, and then transferring the knowledge of the teacher network to the student network Mobilenet_v2 using knowledge distillation. This method achieves network model compression, thereby reducing computational load and saving computational resources.
[0007] An incremental learning method based on knowledge distillation includes the following steps:
[0008] (1) Segment the dataset into long task sequences;
[0009] (2) Deploy and train the teacher network model, and save the corresponding network model parameters after each training of the current task;
[0010] (3) Deploy the student network, train the student network model through fine-tuning, load the teacher network model trained in step (2), and use the knowledge distillation method to make the output of the student network model fit the output of the teacher network model, thereby realizing the knowledge transfer of the teacher network model and completing the training of the student network.
[0011] (4) Remove the teacher network model loaded in step (3), release memory, and test the prediction effect of the student network.
[0012] The method for splitting the dataset in step (1) is to divide each dataset into n tasks, with no overlap between the data of the tasks, and in the incremental phase, the data of the old tasks cannot be accessed, specifically:
[0013] (1.1) Let X and Y be the training and test sets of the current dataset, respectively. Divide the dataset into n tasks, i.e.
[0014] X={x i |i=1,2,…,n},Y={y i |i=1,2,…,n};
[0015] (1.2) During the training process, in the i-th training phase, only the training set x is used. i Train the current network, where 1 ≤ i ≤ n;
[0016] (1.3) During the testing process, in the i-th testing phase, the test set in phases 1, 2, ..., i-1 is tested, that is, y is tested.1 ∪y 2 ∪…∪y i Perform tests, where 1 ≤ i ≤ n; where That is, the classes between tasks are disjoint.
[0017] The method for training the teacher network model in step (2) is as follows:
[0018] First, remove the classification layer; when adding a new task each time, add the corresponding number of classification heads to the teacher network model according to the number of classifications required by the current task; save the corresponding teacher network model parameters after each training of the current task.
[0019] The fine-tuning method for training the student network model in step (3) is as follows:
[0020] For the first task, select the SGD optimizer, calculate the cross-entropy loss and optimize it;
[0021] In the incremental stage, let the current stage be i, where 1 < i ≤ n; the cross-entropy loss of the current task is:
[0022]
[0023] where y i is the true label, is the predicted value, and n is the number of classes;
[0024] Let the classification heads of the old tasks be head1, head2, …, head i-1 , where 1 < i ≤ n; then the logits value output of the current image on the classification heads of the previous old tasks is: Let the logits value output by the student network model fit the logits value output by the teacher network model to make the two output values converge, measure the convergence degree with the knowledge distillation loss function, and optimize this loss function. The knowledge distillation loss function of the current task is:
[0025]
[0026] where is the logits value output by the current task image on the classification heads of the old tasks of the student network model, is the logits value output by the current task image on the corresponding task classification heads of the teacher network model;
[0027] The total loss of the current task is denoted as:
[0028] loss = ce_loss + λ * kd_loss
[0029] where λ is a hyperparameter used to balance the loss ratio.
[0030] The method of completing the knowledge transfer of the teacher network model by the knowledge distillation method in step (3) is as follows:
[0031] Load the trained teacher network model in step (2). After defining the network model, load the corresponding parameters of the teacher network model in step (2);
[0032] During the training process of the student network model, in the incremental stage i (1 < i ≤ n), the training set of the current task is x i , and after sending x i into the student network model, the logits values output on the classification heads head1, head2,..., head i-1 of the old tasks are Concatenate all the logits values into a tensor, denoted as
[0033] Send x i into the trained teacher network model. The logits values output on the classification head of the corresponding task for x i are: Concatenate all the logits values into a tensor, denoted as Process the logits values as follows to increase the weight of the smaller logits values in the network output:
[0034]
[0035] where:
[0036]
[0037] n is the number of labels, log is the logarithmic operation, and T is the temperature coefficient;
[0038] Optimize the student network by combining the cross-entropy loss to complete the training of the student network model. The total loss of the current task is:
[0039]
[0040] Beneficial effects: (1) This knowledge distillation-based incremental learning algorithm can be completed within the framework of fine-tuning, without the need to save samples or expand the model, which greatly reduces memory overhead. (2) There are diverse training methods for the teacher network. The simplest fine-tuning method can be selected. Since the teacher model does not need to be saved in the end, the superior algorithms such as LWF and LWM can be used to train the teacher network when there are sufficient computing resources. After training, the model is deleted to release memory. (3) The number of parameters of the selected student network Mobilenet_v2 is only one-tenth of that of the teacher network. The student network can be deployed on mobile devices. Experimental results show that the student network can be trained to achieve a prediction effect comparable to that of the teacher network while greatly reducing the number of parameters. In the end, the network model is compressed, the amount of computation is reduced and computing resources are saved. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the model of the present invention;
[0042] Figure 2 This is a flowchart of the algorithm of the present invention;
[0043] Figure 3 This is a schematic diagram of the process of the present invention; Detailed Implementation
[0044] The technical solution of the present invention will be further described below with reference to the accompanying drawings.
[0045] Figure 1 This is a schematic diagram of the model of the present invention. Figure 3 This is a flowchart illustrating the process of this invention, using TinyImageNet as an example. TinyImageNet has 200 classes, with 500 training images, 50 test images, and 50 validation images for each class. Since the official documentation does not provide labels for the test images, the 10,000 images in the validation set are used as the test set. The 100,000 images in the training set are divided in a 9:1 ratio, meaning 90,000 images are used for training and 10,000 images are used for validation.
[0046] After preparing the dataset, it is divided into 10 tasks (task1, task2, ..., task10) according to the common long task sequence. Each task contains 20 classes, and each class contains 450 training images, 50 validation images, and 50 test images. The classes of each task are non-overlapping, and the task IDs are inaccessible during testing (following the class incremental learning settings).
[0047] Define the teacher network ResNet34 and remove the classification layer. Currently, there are 20 classes for each task, so a 20-class classification head is added to the model each time a new task is added. After each current task is trained, the corresponding network model parameters are saved.
[0048] During the training process of the student network, for the first task, adopt the common image classification training method, select the SGD optimizer, calculate the cross-entropy loss and optimize it.
[0049] Figure 2 This is the algorithm flowchart of the present invention. In the incremental stage, let the current stage be i, where 1 < i ≤ 10. The current task is still trained according to the standard image classification training method, and the cross-entropy loss of the current task is:
[0050]
[0051] where y i is the true label, is the predicted value.
[0052] Let the classification heads of the old tasks be head1, head2,..., head i-1 , where 1 < i ≤ 10. Then the logits value output of the current image on the classification heads of the previous old tasks is: Here, the knowledge distillation method is used to complete the knowledge transfer from the teacher network model to the student network. Let the logits value output by the student network fit the logits value output by the teacher network. That is, make the two output values as close as possible, measure the closeness with the knowledge distillation loss function, and optimize this loss. Then the knowledge distillation loss of the current task is:
[0053]
[0054] where is the logits value output by the current task image on the classification heads of the old tasks of the student network, is the logits value output by the current task image on the corresponding task classification head of the teacher network.
[0055] Then the total loss of the current task is denoted as:
[0056] loss = ce_loss + λ * kd_loss
[0057] Select λ = 10 as the hyperparameter for balancing the loss ratio, that is, fit the output distributions between the two networks as much as possible.
[0058] Load the teacher network model. After defining the network model ResNet34, load the network model parameters saved after each training of the current task.
[0059] During the training process of the student network, in the incremental phase i (1 < i ≤ 10), the training set of the current task is x i , after sending x i into the network model, the logits values output on the classification heads head1, head2, …, head i-1 of the old tasks are Concatenate all the logits values into a tensor, denoted as Send x i into the trained teacher network model, and the logits values output on the corresponding task's classification head are: Concatenate all the logits values into a tensor, denoted as To better utilize the soft label knowledge in the teacher network, the logits values are further processed here:
[0060]
[0061] where:
[0062]
[0063] l is the number of labels, log is the logarithmic operation, T is the temperature coefficient, which is used to control the weight of smaller logits values.
[0064] Select T = 4 as the temperature coefficient, which increases the weight of smaller logits values, thus strengthening the fitting effect.
[0065] Then the loss of the current task is denoted as:
[0066]
[0067] Optimize this loss to complete the training of the student network.
[0068] After the training of the student network is completed, delete the teacher model, release the memory, and test the prediction performance of the student network on the test set, and measure the performance with the classification average precision.
[0069] For the tiny-imagenet dataset, since the original size of the image is 64 * 64, first resize the image to 224 * 224, the batch size is 256, the learning rate is set to 0.01, and the learning rate is adjusted using the patience mechanism, where patience is set to 15 and the learning rate factor is 3. The number of epochs for each task is 200, the weight decay coefficient is 0.0002, and the random seed is 1993.
[0070] This invention selected six representative international algorithms: finetuning, lwf, ewc, lwm, RWalk, and PathInt, and compared the proposed algorithm with these six algorithms. Using average classification accuracy as the evaluation metric, the proposed algorithm shows a significant improvement in classification accuracy compared to the baseline and other advanced algorithms, even surpassing most methods in teacher networks. The experimental results on tiny-imagenet are shown in Table 1.
[0071] Table 1. Comparison of average classification accuracy on Tiny-ImageNet using different algorithms.
[0072] finetuning lwf ewc lwm RWalk PathInt ours mobilenet_v2 9.0％ 18.7％ 12.1％ 13.9％ 10.8％ 10.8％ 28.4％ resnet34 17.5％ 32.6％ 11.2％ 21.3％ 9.3％ 10.6％
[0073] The algorithm of this invention was also subjected to extensive experiments on the CIFAR100 dataset. The hyperparameter settings were basically consistent with those in the experiments on the tiny-imagenet dataset, with the batch size adjusted to 128. Six algorithms—finetuning, lwf, ewc, lwm, RWalk, and PathInt—were selected for comparison. The experimental results on CIFAR100 are shown in Table 2.
[0074] Table 2 Comparison of average classification accuracy on CIFAR100 under different algorithms
[0075] finetuning lwf ewc lwm RWalk PathInt ours mobilenet_v2 9.2％ 17.1％ 11.6％ 13.4％ 9.7％ 11.0％ 32.4％ resnet34 15.1％ 39.2％ 13.1％ 30.1％ 13.2％ 12.1％ resnet50 14.7％ 32.1％ 12.9％ 28.7％ 12.6％ 13.1％
[0076] Table 3 compares the parameter counts of various network models. The algorithm in this invention uses MobileNet_v2 as the student network and ResNet34 as the teacher network, with the parameter count being only about one-tenth that of the teacher network. Using a larger network model could achieve larger-scale network compression.
[0077] Table 3. Parameter quantities for different network models
[0078]
[0079]
[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A knowledge distillation-based incremental learning method, characterized in that, Includes the following steps: (1) Segment the image dataset into long task sequences; (2) Deploy and train the teacher network model, and save the corresponding network model parameters after each training of the current task; (3) Deploy the student network, train the student network model through fine-tuning, load the teacher network model trained in step (2), and use the knowledge distillation method to fit the output of the student network model to the output of the teacher network model, thereby realizing the knowledge transfer of the teacher network model and completing the training of the student network. (4) Remove the teacher network model loaded in step (3), free up memory, and test the prediction effect of the student network on the image test set; The fine-tuning method for training the student network model in step (3) is as follows: For the first task, the SGD optimizer was selected to calculate and optimize the cross-entropy loss; In the incremental phase, let the current phase be i, where The cross-entropy loss for the current task is: in It's a real label. The predicted value is n, where n is the number of categories. Let the classification heads of the old tasks be respectively ,in The output of the logits value of the current image in the classification header of previous tasks is: The logits values output by the student network model are fitted to the logits values output by the teacher network model to make the two output values converge. The degree of convergence is measured by the knowledge distillation loss function, which is then optimized. The knowledge distillation loss function for the current task is as follows: in It is the logits value output by the old task classification header of the student network model for the current task image. It is the logits value output by the corresponding task classification header of the teacher network model for the current task image; The total loss of the current task is recorded as follows: in This is a hyperparameter used to balance the loss ratio; The method for knowledge transfer in the teacher network model through knowledge distillation in step (3) is as follows: Load the teacher network model trained in step (2), and after defining the network model, load the corresponding parameters of the teacher network model in step (2); During the training of the student network model, in incremental phase i, the training set for the current task is... ,Will After being fed into the student network model, in the classification head of the old task The output value of logits is Concatenate all logits values into a single tensor, denoted as . ; Will It is fed into the trained teacher network model. The logits value output in the category header of the corresponding task is: Concatenate all logits values into a single tensor, denoted as . The following steps are taken to increase the weight of smaller logits values in the network output: in: , Where is the number of tags, log is the logarithm, and T is the temperature coefficient; The student network is optimized using joint cross-entropy loss to complete the training of the student network model. The total loss for the current task is: 。 2. The incremental learning method based on knowledge distillation according to claim 1, characterized in that, The method for splitting the dataset in step (1) is to divide each dataset into n tasks, with no overlap between the data of the tasks, and in the incremental phase, the data of the old tasks cannot be accessed, specifically: (1.1) Let X and Y be the training and test sets of the current dataset, respectively. Divide the dataset into n tasks, i.e. , ； (1.2) During the training process, in the i-th training phase, only the training set is used. Training the current network, where ; (1.3) During the testing process, in the i-th testing phase, the test set in phases 1, 2, ..., i-1 is tested, that is, the test set in phases 1, 2, ..., i-1 is tested. Tests were conducted, including ;in That is, the classes of the various tasks do not overlap.
3. The incremental learning method based on knowledge distillation according to claim 1, characterized in that, The method for training the teacher network model in step (2) is as follows: First, remove the classification layer; each time a new task is added, add the corresponding number of classification heads to the teacher network model according to the number of classifications required for the current task; after each training of the current task, save the corresponding teacher network model parameters.