An incremental learning method based on knowledge distillation and parameter isolation

Abstract

The application discloses a kind of based on knowledge distillation and parameter isolation incremental learning method, it is related to artificial intelligence technical field, including using the parameter initialization teacher model feature extractor of student model feature extractor, utilize multiple pooling branch module auxiliary training teacher model, in the mask learning stage, the teacher model obtained in fine-tuning stage is used to guide the learning of binary mask by the way of knowledge distillation, when the model is reasoned at the end of training, discard teacher model, according to the task ID information provided to find the corresponding mask and classifier and act on student model to reason prediction.The application uses binary mask to isolate the parameters of different tasks, greatly reduces the space consumption caused by the parameter growth of model in incremental task, guides mask update through teacher model on new task, enhances the "plasticity" of model, and the method of multiple pooling branch auxiliary fine-tuning model can improve the performance of model on different tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an incremental learning method based on knowledge distillation and parameter isolation. BACKGROUND

[0002] Parameter isolation method is an important research direction in the field of incremental learning, its main idea is that different tasks are learned by different network parameters, and the mutual coverage and interference between different tasks are minimized to avoid deviating from the optimal solution of historical tasks. Progressive neural networks (PNN) and expert gate (EG) are relatively common methods, but EG and PNN have the same problem, only all networks do not participate in training, and the calculation cost is relatively small. This method of dynamically increasing network parameters cannot be used in long sequence incremental tasks or low storage space, so many works are committed to realizing parameter isolation with less parameter growth.

[0003] The basic idea of the mask-based parameter isolation method is to select different substructures of the network for different tasks by using binary masks, so that only one set of binary masks is grown with the growth of the task, and the parameter growth is greatly reduced. Piggyback uses a 1-bit binary mask for each parameter, and the mask is also learned. Almost all parameters of the network remain unchanged after the initial task training, only the parameters of the classifier will change, so only the binary mask needs to be trained during the training process of the entire incremental task sequence, and no other mechanism is needed to select nodes or parameters. The training process is greatly simplified, and the 1-bit mask value also greatly reduces the space consumption caused by parameter growth. Since the network parameters are always fixed, the learning of different tasks is independent of each other, which brings the benefit that the learning effect of the network is independent of the order of the tasks.

[0004] Since Piggyback fixes all parameters and only uses one-hot encoding as supervision information, the biggest problem it brings is that the binary mask learned by the network on the new task is not optimal, that is, the "plasticity" on the new task is insufficient, so the present application proposes a fine-tuning model guided mask-based task incremental method (abbreviated as FTGM). SUMMARY

[0005] (1) Technical problems solved

[0006] To overcome the shortcomings of the prior art, the application provides an incremental learning method based on knowledge distillation and parameter isolation, which contains two stages when learning a new task, a multi-pooling branch module is proposed in the first fine-tuning stage to assist the teacher model in fine-tuning to achieve higher recognition accuracy on the new task, and a parameter-level learnable binary mask is added to the student network in the second mask learning stage, the teacher model obtained in the first stage is used to guide the update of the mask through the knowledge distillation technology, and each task can be trained to obtain a group of binary masks by using the FTGM method, and when the model performs reasoning, the corresponding mask can be found according to the given task identification information to identify the input data, thereby solving the problems in the background technology.

[0007] (ii) Technical solutions

[0008] To achieve the above object, the application is implemented by the following technical solutions: an incremental learning method based on knowledge distillation and parameter isolation, the basis used by the method includes an incremental data set, an incremental classification model and an incremental task target, the method includes a fine-tuning stage and a mask learning stage when training, the teacher model and the student model have the same structure, and the feature extractor of the student model is pre-trained and fixed in the training process, the parameters of the teacher model feature extractor are initialized by using the parameters of the student model feature extractor in the fine-tuning stage, the teacher model is constructed by fine-tuning through a multi-pooling branch (Diverse Pooling Branches, DPB) module, in the mask learning stage, the teacher model obtained in the fine-tuning stage is used to guide the learning of the binary mask through the knowledge distillation method, when the model performs reasoning at the end of training, the teacher model is discarded, the corresponding mask and classifier are found according to the provided task ID information to act on the reasoning prediction of the student model, wherein the multi-pooling branch (Diverse Pooling Branches, DPB) module assists the teacher model training through online knowledge distillation, different branches share the entire network except the last pooling layer and the fully connected layer of the network, and different branches use different pooling branches to reduce branch homogenization, the global average pooling (Global Average Pooling, GAP) branch is taken as the first branch, the classifier of the branch contains one fully connected layer, and other branches include but are not limited to general average pooling, spatial pyramid pooling and bilinear pooling, and the classifiers of the other branches are composed of two fully connected layers.

[0009] Further, the incremental data is: let the training data set of the entire incremental learning task be A total of T tasks are included, wherein is the training data set corresponding to the tth task, i is the sample image serial number, a total of N t samples are included, C t classes are included, (xt i y t i ) is the i-th sample image and the corresponding class label, y t i ∈ {0, 1,..., C t-1}, is the test data set corresponding to the t-th task, and there are N' total samples, (x t t i is the i-th test sample image and the corresponding task ID, and T task sequences arrive in sequence during the training phase.

[0010] Further, the incremental classification model is: let the target classification model be M, which can only be trained on one data set at a time, the target model trained on the t-th task is M t , and the model M = M T is obtained after all tasks are trained.

[0011] Further, the task identification information provided by the incremental task target during model inference is formalized to improve the recognition accuracy of M on the data set , where 1, 2,..., is the task ID information specified by humans.

[0012] Further, the entire model during the fine-tuning phase when training the t-th task is where f s t is the feature extractor shared by each branch, n = 4 is the number of branches, j is the branch number, is the pooling layer of the j-th branch, θ j t is the classifier of the j-th branch.

[0013] Further, when knowledge distillation guides mask learning, for the t-th task, the student model M t is where is the parameter of the student model feature extractor f, m t = [m t l ] L l＝1 is the corresponding binary mask, θ t is the classifier, and m t l = η (m t,r l , ∈) is the parameter and mask of the l-th layer, m t,r l ​is m t The corresponding real-valued mask, η, is a function of the binary mask, m, and a threshold, ∈, and the output of the l-th layer is o t 0 is the input of the network.

[0014] Further, when the knowledge distillation guides the mask learning, the binary mask m The cross-entropy loss is used to measure the difference between the output of the mask network and the real label:

[0015]

[0016] In the formula: is the cross-entropy loss, c is the class label, is the probability that the student model judges as class c, δ c，y is defined by the following formula:

[0017]

[0018] The teacher model (F1 t ) guides the student model to learn the binary mask through the Kullback Leibler (KL) divergence:

[0019]

[0020] In the formula: is the divergence between the teacher network and the student network output, τ is the temperature coefficient, is the probability that the teacher model judges as class c, is the probability that the student model judges as class c, F1 t is the teacher model.

[0021] Further, the overall loss function of the mask learning stage is:

[0022]

[0023] In the formula: is the overall loss of the mask learning stage.

[0024] Further, the mask function of the training stage and the inference stage and the mechanism of the network are: in the training process, the binary mask m t is maintained, and the binary mask is obtained through the following threshold function: t,r

[0025]

[0026] In the formula: z is the real-valued mask, and ∈ is the threshold.

[0027] Binary mask and feature extractor parameters​ The Hadamard product is calculated, the single weight value of the network is kept active or shielded, the gradient of the overall loss function is iteratively updated to the real value mask, after the training is completed, only the binary mask is retained, and the real value mask is discarded, in the inference stage, the mask corresponding to the task is found according to the task ID, and the mask is applied to the network to obtain the mask network corresponding to the task, and inference is performed on the mask network.

[0028] Further, the real value mask is updated using the gradient of the binary mask, and the formula for updating the real value mask is:

[0029]

[0030] (III) Beneficial effects

[0031] The application provides an incremental learning method based on knowledge distillation and parameter isolation, which has the following beneficial effects:

[0032] The application uses binary masks to isolate the parameters of different tasks, greatly reducing the space consumption caused by the parameter growth of the model in the incremental task, and the mask is updated by guiding the teacher model on the new task, which enhances the plasticity of the model, and the proposed multi-pooling branch auxiliary fine-tuning model method can stably improve the performance of the model on different tasks. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The application provides an incremental learning method based on knowledge distillation and parameter isolation, which has the following beneficial effects: DETAILED DESCRIPTION

[0034] The technical solutions in the application will be described in detail below with reference to the drawings in the application. Obviously, the described solutions are only a part of the application, not the whole. Based on the solutions in the application, all other solutions obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0035] As shown in the drawings, Figure 1 The application proposes a fine-tuning model guided binary mask based task incremental method, which contains two stages when learning a new task: in the first fine-tuning stage, a multi-pooling branch module is proposed to assist the teacher model to fine-tune to achieve higher recognition accuracy on the new task, and in the second mask learning stage, a parameter level learnable binary mask is added to the student network, and the teacher model obtained in the first stage is used to guide the update of the mask through the knowledge distillation technology, a group of binary masks is trained for each task using the FTGM method, and when the model is inferred, the corresponding mask is found according to the given task identification information to identify the input data.

[0036] In the parameter isolation method, the approach of progressive neural networks (PNN) is to fix the model parameters corresponding to the historical tasks. When training on a new task, a new trainable network is added, and the output of the network corresponding to the previous task is used as the input of the current trainable model to establish the horizontal connection between the networks of different tasks. This way, one task corresponds to one network, and all networks participate in the forward propagation process, and the calculation cost and storage cost grow very quickly. The expert gate (EG) method also corresponds to a new network for each task. Unlike PNN, EG selects the most similar task corresponding network to initialize the new network of the new task by calculating the similarity between the current task and the historical task, and selects different training strategies according to the similarity: when the similarity is high (greater than a certain threshold), the learning without forgetting (LwF) method is used to train the new network; when the similarity is low (less than the threshold), fine-tuning is directly performed on the network. However, the EG method has the same problem as PNN, except that not all networks participate in training, the calculation cost is relatively small. This method of dynamically increasing network parameters cannot be used in long sequential incremental tasks or low storage space, so many works are committed to achieving parameter isolation with less parameter growth.

[0037] The basic idea of the mask-based parameter isolation method is to select different substructures of the network for different tasks by using binary masks, so that each task only grows a set of binary masks, and the parameter growth is greatly reduced. PathNet uses a genetic algorithm to select some paths that are important to historical tasks, fixes the parameters on the paths, and reinitializes other parameters for training on new tasks (the fixed parameters can be reused but not updated). This process is repeated every time a new task is trained. This method does not dynamically grow the network, but relies on a computationally intensive path search algorithm. PathNet can be considered a mask-based parameter isolation method, except that its mask is implicit. Unlike PathNet, PackNet uses pruning to select paths similar to PathNet. Specifically, the network is trained on the first task, and then as the number of tasks increases, a pruning rate is set to select parameters important to historical tasks. These parameters are fixed when training new tasks. As the number of tasks increases, the pruning process is repeated. For each parameter, PackNet needs a log2(N) bit mask to maintain which task uses the parameter, where N is the number of tasks. The HAT (Hard attention to the task) method believes that there are some similar or dissimilar features between different tasks, so an attention mechanism is added to each layer of the network. Each neuron node corresponds to an "attention". After training the current task, the attention of each node is calculated with the attention of the node in the historical task to retain the knowledge learned before. In essence, HAT is also a mask-based parameter isolation method. Unlike PackNet, the mask (attention vector) of HAT does not correspond to the parameter, but to the node. Therefore, the HAT model will be lighter. The mask of HAT is learned, not based on a genetic algorithm or some rules, so there is no need to specify a pruning rate. However, the mask of HAT is not binary, but a number between 0 and 1.

[0038] Piggyback uses a 1-bit binary mask for each parameter, which is also learned. Almost all parameters of the network are fixed after the initial task is trained, and only the parameters of the classifier will change. Therefore, only the binary mask needs to be trained during the entire incremental task sequence training process, and no other mechanism is needed to select nodes or parameters. The training process is greatly simplified, and the 1-bit mask value greatly reduces the space consumption caused by parameter growth. Since the network parameters are always fixed, the learning of different tasks is independent of each other, which has the advantage that the learning effect of the network is independent of the order of the tasks.

[0039] Since Piggyback fixes all the parameters and only uses one-hot encoding as supervision information, the biggest problem it brings is that the binary mask learned by the network on the new task is not optimal, that is, the "plasticity" on the new task is not enough.

[0040] Therefore, the present application is used to solve the task increment problem of sample-free replay, and the target network (model) needs to learn on different data sets arriving in sequence, and cannot retain historical data to meet the needs of data privacy, security and low storage consumption. The final model needs to maintain high recognition accuracy on all data sets. In order to clearly explain the problem and method solved by the present application, first of all, the problem is formally expressed as follows:

[0041] Incremental data set: let the training data set of the entire incremental learning task be A total of T tasks, wherein is the training data set corresponding to the tth task, and a total of N t samples, containing C t classes, (x t i ,y t i ) is the i th sample image and the corresponding class label, y t i ∈{0,1,...,C t-1}, is the test data set corresponding to the tth task, and a total of N' samples, (x t t i , t) is the i th test sample image and the corresponding task ID, and in the training stage, T tasks arrive in sequence.

[0042] Incremental classification model: let the target classification model be M, which can only be trained on one data set at a time, and let the target model trained on the tth task be M t , and the model M=M T is obtained after all task training.

[0043] Incremental task target: the purpose of the incremental classification task is to improve the correct rate of the model in recognizing the learned data. The present application faces task increment learning, and task identification information (task ID) needs to be provided when the model is inferred. The task target can be formally expressed as improving the recognition accuracy of the target model M on the data set , wherein 1,2,..., is the task ID information specified by the artificial.

[0044] ​Solution: The application proposes an incremental learning method based on knowledge distillation and parameter isolation, proposes a fine-tuning model guided mask-based task incremental method, which is a two-stage method containing two stages of learning during training, namely the fine-tuning stage and the mask learning stage, and the framework of the whole method is as shown in Figure 1 The solid "→" in the figure represents the forward propagation process, the dashed "→" represents the distillation process, and the different colors of the arrows represent different steps.

[0045] Since the teacher model and the student model have the same structure, and the feature extractor of the student model is pre-trained and fixed during training, in the fine-tuning stage: step 1, the parameters of the teacher model feature extractor are initialized using the parameters of the student model feature extractor; step 2, the teacher model is trained with the help of the Diverse Pooling Branches (DPB) module proposed by the application; in the mask learning stage: step 3, the teacher model obtained in the fine-tuning stage guides the learning of the binary mask through knowledge distillation, and when the model is used for inference at the end of training, the teacher model is discarded, the corresponding mask and classifier are found according to the provided task ID information, and the student model is used for inference and prediction, the two stages during training are described in detail below.

[0046] (1) Fine-tune the teacher model through the DPB module

[0047] Since the incremental learning faces complex task scenarios, in the incremental task composed of 5 data sets, the data domain difference between different tasks is large, and the difficulty of the task is also large, so this stage aims to provide a better teacher model for the target model when learning different tasks, and the method of constructing the teacher model needs to be stable in different task scenarios. The simplest and most direct method is to directly fine-tune the pre-trained model on different task data sets, the problem of this method is that the pre-trained model is pre-trained on ImageNet, and the data domain of the data set contained in the target task is quite different from the general data set. The performance of the model obtained by directly fine-tuning is poor, if you choose to construct a teacher model in the target task on part of the data set with good performance, since the data domain difference between the data sets of the target task is also large, the teacher obtained by this training method cannot stably obtain better performance on all task data sets, if you choose different SOTA methods to construct a teacher model on different tasks, this increases the complexity of the method and the training difficulty.

[0048] Therefore, the present application proposes to use the online knowledge distillation method to assist in fine-tuning the pre-trained model to obtain the final required teacher model, and the existing research and use of the online knowledge distillation method mainly focuses on general image recognition tasks, and the performance of these methods will decrease when processing other tasks such as fine-grained image recognition tasks, and the present application proposes a new multi-pooling branch (DPB) module, which uses the module to assist the training of the target teacher model through online knowledge distillation. Unlike other multi-branch methods, different branches share the entire network except the last pooling layer and the fully connected layer of the network, and different pooling branches are used in different branches to alleviate the branch homogenization problem.

[0049] As shown in Figure 1 , the present application represents the entire model in the fine-tuning stage when training the t-th task as where f s t represents the feature extractor shared by each branch, n = 4 is the number of branches, represents the pooling layer of the j-th branch, θ j t represents the classifier of the j-th branch. For simplicity, the present application takes the Global Average Pooling (GAP) branch as the first branch, and the classifier of this branch contains 1 fully connected layer, and other branches include but are not limited to general average pooling, spatial pyramid pooling and bilinear pooling, and the corresponding classifiers are composed of two fully connected layers.

[0050] (2) Knowledge distillation guided mask learning

[0051] For the t-th task, the student model M t is represented as where represents the parameters of the student model feature extractor f, m t = [m t l ] L l = 1 is the corresponding binary mask, θ t is the classifier, and m t l = η (m t ,r l , ∈) is the parameters and mask of the l-th layer, m t,r l is the real value mask corresponding to m t , and η is a function of binaryzation with threshold ∈, then the output of the l-th layer can be represented as o t 0 is the input of the network.

[0052] right This invention uses cross-entropy loss to measure the difference between the output of the masked network and the true label:

[0053]

[0054] in Meanwhile, the teacher model (F1) t Guiding student models to learn binary masks using KL divergence:

[0055]

[0056] The overall loss function during the mask learning phase is:

[0057]

[0058] The role of masks in the training and inference phases and their relation to the network mechanism: During training, this invention uses a binary mask m. t Maintain a set of real-valued masks m t,r The binary mask is obtained using the following threshold function:

[0059]

[0060] Binary mask and feature extractor parameters The Hadamard product (element-wise multiplication) is calculated to keep individual weights of the network active or masked. The real-valued mask is iteratively updated by backpropagating the gradient of the overall loss function. After training, this invention only retains the binary mask and discards the real-valued mask. During the inference phase, the mask corresponding to the task is found according to the task ID and applied to the network to obtain the mask network corresponding to the task. Inference is then performed on the mask network.

[0061] Since the threshold function is a step function, not a continuous function, backpropagation cannot directly differentiate the real-valued mask. Therefore, the gradient with respect to the binary mask is used to update the real-valued mask, i.e.:

[0062]

[0063] This update method has been proven feasible in some network binarization projects.

[0064] Experimental results show that the network achieves an average recognition accuracy of 86.38% across all tasks on five datasets for incremental recognition, reaching state-of-the-art (SOTA) accuracy on some datasets.

[0065] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0066] Finally, the above merely provides the preferred embodiments of the present application, but is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An incremental learning method based on knowledge distillation and parameter isolation, the method using an incremental dataset, an incremental classification model, and an incremental task objective, characterized in that... This method includes a fine-tuning phase and a mask learning phase during training. The teacher and student models have the same structure, and the student model's feature extractor is pre-trained with fixed parameters during training. In the fine-tuning phase, the parameters of the teacher model's feature extractor are initialized using the student model's feature extractor parameters. The teacher model is constructed through a Diverse Pooling Branches (DPB) module. In the mask learning phase, the teacher model obtained in the fine-tuning phase is used to guide the learning of binary masks through knowledge distillation. At the end of training, when the model performs inference, the teacher model is discarded, and the corresponding mask and classifier are found based on the provided task ID information and applied to the student model's inference prediction. The Diverse Pooling Branches (DPB) module assists the teacher model training through online knowledge distillation. Different branches share the entire network except for the final pooling and fully connected layers, and different branches use different pooling branches to reduce branch homogenization. Global average pooling is used. The pooling (GAP) branch is the first branch, and the classifier of this branch contains one fully connected layer. Other branches include, but are not limited to, general average pooling, spatial pyramid pooling, and bilinear pooling. The classifiers of other branches are all composed of two fully connected layers. Incremental data: Let the training dataset for the entire incremental learning task be denoted as . It contains a total of T tasks, of which This is the training dataset corresponding to the t-th task, where i is the sample image index, and there are a total of N. t A sample containing C t Categories It is the i-th sample image and its corresponding category label. This is the test dataset corresponding to the t-th task, with a total of N′. t One sample, It is the i-th test sample image and its corresponding task ID. During the training phase, T tasks arrive sequentially. The incremental classification model is as follows: Let M be the target classification model. This model can only be trained on one dataset at a time. The training is performed on the t-th task, and the target model is M. t After all tasks are trained, the model M = M is obtained. T ; When performing model inference, the incremental task objective provides a formal representation of task identification information to improve the performance of M on the dataset. The recognition accuracy is given by 1, 2, ..., where 1, 2, ... are manually specified task IDs.

2. The incremental learning method based on knowledge distillation and parameter isolation according to claim 1, characterized in that, The entire model will be fine-tuned during the training of the t-th task. in This is a shared feature extractor for all branches, where n = 4 is the number of branches and j is the branch number. For the pooling layer of the j-th branch, θ j t Let be the classifier for the j-th branch.

3. The incremental learning method based on knowledge distillation and parameter isolation according to claim 2, characterized in that, During knowledge distillation-guided mask learning, for the t-th task, the student model M... t For M t =(W,m t ,θ t ),in m are the parameters of the student model feature extractor f. t =[m t l ] L l＝1 It is the corresponding binary mask, θ t It is a classifier. and m t l =η(m) t,r l ,∈) are the parameters and mask of the l-th layer, m t,r l It is m t The corresponding real-valued mask, η, is a function that binarizes the threshold ∈, and the output of the l-th layer is... o t 0 represents the network input.

4. The incremental learning method based on knowledge distillation and parameter isolation according to claim 3, characterized in that, When guiding mask learning through knowledge distillation, for Cross-entropy loss is used to measure the difference between the output of the masked network and the true label: In the formula: Let c be the cross-entropy loss, and c be the class label. δ represents the probability that a student is classified as class c in the model. c,y Defined by the following formula: Teacher Model (F1) t Guiding the student model to learn the binary mask using KullbackLeibler (KL) divergence: In the formula: Let τ be the divergence between the outputs of the teacher network and the student network, and τ be the temperature coefficient. The probability that the teacher model classifies the subject as category c. The probability of a student being classified as class c in the model, F1 t For the teacher model.

5. The incremental learning method based on knowledge distillation and parameter isolation according to claim 4, characterized in that, The overall loss function for the mask learning phase is: In the formula: This represents the total loss during the mask learning phase.

6. The incremental learning method based on knowledge distillation and parameter isolation according to claim 5, characterized in that, The masking mechanism in the training and inference phases works as follows: During training, a binary mask m is used. t Maintain a set of real-valued masks m t,r The binary mask is obtained using the following threshold function: In the formula: z is a real-valued mask, and ε is the threshold; Binary mask and feature extractor parameters The Hadamard product is calculated to keep individual weights of the network active or masked. The real-valued mask is iteratively updated by backpropagating the gradient of the overall loss function. After training, only the binary mask is retained and the real-valued mask is discarded. During the inference phase, the mask corresponding to the task is found according to the task ID and applied to the network to obtain the mask network corresponding to the task. Inference is then performed on the mask network.

7. The incremental learning method based on knowledge distillation and parameter isolation according to claim 6, characterized in that, Backpropagation uses the gradient of the binary mask to update the real-valued mask. The formula for updating the real-valued mask is:

Images