A class-incremental learning method based on knowledge bridging and category anchoring

Abstract

A class-incremental learning method based on knowledge bridging and class anchoring, which alleviates the catastrophic forgetting problem and improves the overall performance of the model class-incremental learning through knowledge bridging and class anchoring respectively; knowledge bridging aims to establish the semantic correlation between old classes and new classes, and applies the learned knowledge to learn new information by using feature-level distillation; class anchoring focuses on learning class-specific feature centers that are crucial for distinguishing all classes. Through the combination of the class-incremental learning method based on knowledge bridging and class anchoring proposed in the application, the original model can achieve better performance in the incremental learning task and effectively alleviate the catastrophic forgetting phenomenon.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of machine learning models, and relates to a class-incremental learning method for knowledge bridging and class anchoring. BACKGROUND

[0002] In recent years, machine learning models have made remarkable achievements in various fields such as natural language processing and computer vision. In some cases, deep learning models have even outperformed humans on specific tasks. However, for deep neural networks (DNNs), further generalization of their capabilities and training models on new tasks often leads to catastrophic forgetting. That is, models tend to forget previously learned knowledge. To alleviate this problem, incremental learning has been proposed and is increasingly gaining attention.

[0003] Class-incremental learning is one of the topics of incremental learning. The goal is to enable image classification models to learn new class knowledge in a sequential manner, ensuring that the model maximizes the ability of all tasks while minimizing the forgetting of previous tasks. To alleviate catastrophic forgetting, researchers have proposed different methods. These methods can be divided into two categories: network expansion-based methods and replay-based methods. Network expansion-based methods adapt the model to multiple tasks by dynamically updating the network structure. These methods usually require expansion modules or store additional parameters to adapt to new tasks. For example, DER expands the feature extractor to adapt to sequential tasks and uses a channel-level masking strategy to remove redundant parameters from the expansion module to prevent model bloating. FOSTER uses a two-stage learning strategy, using a double branch to fit the residuals of new and old classes in the feature enhancement stage to improve the class-incremental learning performance of the model, and compressing the double branch results into a single branch in the feature compression stage to alleviate the bloating of the model.

[0004] In contrast, replay-based methods put old task image samples into a cache area, and by selectively storing and replaying previous class samples, they achieve the effect of alleviating catastrophic forgetting. For example, iCaRL uses knowledge distillation combined with cache data to replay and remember the knowledge of old task class instances. However, the cache area and the number of new task class instances are vastly different, leading to prediction bias. LUCIR introduces a cosine normalization constraint based on this problem, which alleviates prediction bias by keeping the distribution of instances in the feature space similar to the previous model, further improving the performance of the model. BiC analyzes the difference in prediction weights of the image classification model on new classes and old classes, finds the imbalance of the weights, and further alleviates the prediction bias problem of the model through weight normalization. PODNet further excavates the old class knowledge contained in the previous image classification model, and through the method of distilling the consistency of the intermediate layer output of the current image classification model and the previous image classification model, the inference ability of the current image classification model is further improved.

[0005] Although the recent playback-based methods have achieved success. But there are still two unsolved problems. First, the model updates the original knowledge distillation or the playback class-incremental learning way only focuses on the old knowledge when playing back the old class, which leads to the inevitable conflict between remembering the old knowledge and learning the new knowledge. Second, in order to solve the problem of performance decline caused by the imbalance of CIL new and old class data, previous methods will correct the classification bias and improve the model's class discrimination through the addition of modules, data compression, model compression and other ways, which will cause a significant increase in storage and computing overhead. SUMMARY

[0006] In order to overcome the shortcomings of the prior art, the present application provides a class-incremental learning method based on knowledge bridging and class anchoring, which more conveniently and universally handles catastrophic forgetting, enables the image classification model to better balance between different tasks, and preserves more old task information in class-incremental learning. The present application realizes the comprehensive application of knowledge bridging and class anchoring, bringing a more powerful tool to the field of incremental learning.

[0007] The technical scheme adopted by the present application to solve its technical problems is:

[0008] A class-incremental learning method based on knowledge bridging and class anchoring, comprising the following steps:

[0009] Step one: training the image classification model and the data cache, the process is as follows:

[0010] S11, input: image data of new tasks and cache area of previous tasks, the cache area contains image data of previous tasks, and the cache area is initially empty;

[0011] S12, processing: input the new task image data into the image classification model for training, after training, a part of the new task image data will be added to the cache area and mixed with the image data of the previous task;

[0012] S13, output: the trained image classification model and the updated image data cache;

[0013] Step two: knowledge bridging and class anchoring, the process is as follows:

[0014] S21, mix and input the updated image data cache in step one with the new task image data;

[0015] S22, the image classification model will process the new task image data and the old task image data in the cache area at the same time, and the image classification model will calculate the classification loss of the new task data and the distillation loss of the old task data respectively, in order to adapt to the new task and consolidate the old task knowledge;

[0016] S23, Distillation Process: The image classification model uses knowledge bridging and class anchoring techniques to calculate distillation loss and classification loss, respectively, including extracting class-level association information and class feature centers from old and new task classes, and applying this knowledge to the overall training process of the current image classification model.

[0017] S24, Comprehensive Loss: The image classification model considers classification loss and distillation loss to optimize overall performance, which helps control catastrophic forgetting and balance plasticity and stability.

[0018] S25, After the calculation of distillation loss and classification loss, the image classification model has adapted to the new task and taken into account the knowledge of the old task.

[0019] Further, in S22, the goal of knowledge bridging is to use the old class features that the model has learned to learn new classes. When learning new tasks, the image classification model learns the association features of old and new classes and learns and remembers new classes through association features to ensure that old class features are preserved when the model learns new class parameter updates and do not have the problem of catastrophic forgetting. First, through the previous image classification model M t-1 Extract features of new task image data x This feature is a hard target, representing the representation result of using old class features to represent new classes. Then, through the current image classification model M t Extract features of new task image data x Next, the feature output After dividing by the temperature parameter (Temperature), a softmax transformation is performed to obtain a softened probability distribution (soft target). Finally, the cross-entropy of the soft target and the hard target is weighted to encourage the new class feature distribution of the current image classification model to align with the feature distribution of the previous image classification model, thereby achieving knowledge bridging between new and old classes and alleviating the problem of catastrophic forgetting.

[0020] Preferably, the knowledge bridging loss is defined as follows:

[0021]

[0022]

[0023] where C t represents the class of the current task, C old represents the class of the previous task, q C (x) is the softmax transformation of the soft target, is the softmax transformation of the hard target.

[0024] Further, in the S23, the category anchor retains the category information from the class feature center, and the image classification model will strive to remember the center feature of each class of images in each task to ensure that the information most representative of the image category is retained; this helps to improve the overall performance of the image classification model in the class incremental learning task. In order to save the feature center of the old class, after each task is completed, the current image classification model is calculated for all images of the same class of the new task class and weighted average As the feature center of this class, when the new task comes, the image data of the old task in the cache area and the image data of the new task are input to the image classification model M t of the current task, for the image data of the old task, take the saved old class feature center; for the image data of the new task, calculate the weighted average of the features of the same class of images as the class feature center of the class, respectively do KL loss, by requiring the feature value of the current classification model to be close to the class feature center to retain the most representative information of the old class and the new class.

[0025] Preferably, the loss definition of the category anchor is as follows:

[0026]

[0027] Where, n k represents the number of new categories k under the current batchsize. The CA function measures the difference between the predicted distribution and the class feature center distribution. By minimizing this loss, the current image classification model can remember the feature center of the entire class and thus improve the overall class incremental learning performance of the model. Our loss function can finally be defined as:

[0028] L totat = L method + αL KB + βL CA

[0029] Where, α and β are hyperparameters that measure the importance of knowledge bridging and category anchoring, respectively, and L method is the training loss function of the original method.

[0030] The technical concept of the present application is that in order to avoid the catastrophic forgetting problem as much as possible when learning new categories, the present application proposes knowledge bridging and category anchoring, the goal of knowledge bridging is to learn the semantic association features of old and new categories, and learn the new category through the association relationship between the old category features and the new category, thereby relatively smoothly consolidating the old category knowledge and alleviating the catastrophic forgetting problem of the model when training a new task; through category anchoring, the model learns the category feature center which is essential for distinguishing all categories, thereby improving the overall class incremental learning performance of the model. The existing method can achieve better performance in the class incremental learning task by combining knowledge bridging and category anchoring, and effectively alleviates the catastrophic forgetting problem.

[0031] The beneficial effects of the present application mainly manifest in:

[0032] (1) Introduce knowledge bridging into class incremental learning, compared with the prior art, the technology of the present method has significant advantages in dealing with the catastrophic forgetting problem. The traditional incremental learning method based on replay usually only processes the feature distillation of the old class alone, while the present method aims to automatically learn the associated features between the new class and the old class during the class incremental learning process, and learn the new class and consolidate the knowledge of the old class through the associated features, greatly alleviating the catastrophic forgetting problem of the model in continuous incremental learning.

[0033] (2) Introduce category anchoring into the model training process of the class incremental learning method to improve the incremental learning performance. Since the method of the present application can better preserve the feature center of the class as a whole, the overall performance in the task is better, and according to the category feature center, the image classification model no longer excessively deviates from the new task, which improves the classification accuracy of the new and old classes and achieves the effect of improving the incremental learning performance.

[0034] (3) Provide more flexible use, our method can adapt to existing models, improve the performance of the original method under the premise of ensuring the original class incremental method unchanged, and realize the effect of plug and play. Extensive experiments on multiple comparative experiments and data sets show that the method of the present application can enhance the incremental learning performance of the existing model without increasing additional costs. Ablation studies further verify the effectiveness of each component in our method. In general, the class incremental learning method using knowledge bridging and category anchoring is successful. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The class incremental learning experimental results on the CIFAR100 dataset, the performance is measured by cumulative accuracy.

[0036] Figure 2For the class-incremental learning experimental results on the ImageNet-Subset dataset, the performance is measured by the cumulative accuracy, wherein (a) represents ImageNet-SubSet, 2+2; (b) represents ImageNet-SubSet, 5+5; (c) represents ImageNet-SubSet, 10+10; (d) represents ImageNet-SubSet, 50+2; (e) represents ImageNet-SubSet, 50+5; (f) represents ImageNet-SubSet, 50+10.

[0037] Figure 3 For the catastrophic forgetting solution example diagram, wherein (a) represents a schematic diagram of catastrophic forgetting encountered by the model in class-incremental learning, and (b) represents a schematic diagram of the effect of knowledge bridging and class anchoring.

[0038] Figure 4 For the knowledge bridging function schematic diagram, wherein (a) represents a knowledge bridging loss function schematic diagram, and (b) represents a feature correlation schematic diagram between new and old classes.

[0039] Figure 5 For the concept of class anchoring schematic diagram.

[0040] Figure 6 For the implementation flowchart of the method of the present application. DETAILED DESCRIPTION

[0041] The present application will be further described below with reference to the accompanying drawings.

[0042] REFERENCE Figures 1-6 A class-incremental learning method based on knowledge bridging and class anchoring, comprising the following steps:

[0043] Step one: training an image classification model and a data cache, the process is as follows:

[0044] S11, input: image data of a new task and a cache area of a previous task, the cache area contains image data of the previous task, and the cache area is initially empty;

[0045] S12, processing: input the new task image data into the image classification model for training, and after the training is completed, a part of the new task image data will be added to the cache area and mixed with the image data of the previous task;

[0046] S13, output: the trained image classification model and the updated image data cache;

[0047] Step two: knowledge bridging and class anchoring, the process is as follows:

[0048] S21, mix the updated image data buffer in step one with new task image data;

[0049] S22, the image classification model will process both new task image data and old task image data in the buffer, and will calculate classification loss for new task data and distillation loss for old task data respectively, so as to adapt to new task and consolidate old task knowledge;

[0050] S23, distillation process: the image classification model uses knowledge bridging and class anchoring techniques to calculate distillation loss and classification loss respectively, including extracting class-level association information and class feature center knowledge from new and old task classes, and applying these knowledge to the overall training process of the current image classification model.

[0051] S24, comprehensive loss: the image classification model will consider classification loss and distillation loss to optimize overall performance, which helps to control catastrophic forgetting and balance plasticity and stability.

[0052] S25, the image classification model calculated by distillation loss and classification loss has adapted to the new task and considered the knowledge of the old task.

[0053] In S22, the goal of knowledge bridging is to use the old class features that the model has learned to learn new classes. When learning new tasks, the image classification model learns the association features of old and new classes, and learns and remembers new classes through association features to ensure that old classes retain their features when the model learns new class parameter updates, and do not have the problem of catastrophic forgetting. First, the previous image classification model M t-1 Extract the features of new task image data x This feature is a hard target, which represents the representation result of using old class features to represent new classes. Then, the current image classification model M t Extract the features of new task image data x Then, the feature output After dividing by the temperature parameter (Temperature), do softmax transformation to get the softened probability distribution (soft target). Finally, take the weighted average of the cross-entropy of the soft target and the hard target, encourage the new class feature distribution of the current image classification model to align with the feature distribution of the previous image classification model, so as to realize the knowledge bridging of new and old classes and alleviate the problem of catastrophic forgetting.

[0054] The definition of knowledge bridging loss is as follows:

[0055]

[0056]

[0057] where C t represents the class of the current task, C old represents the class of the previous task, q C (x) is the softmax transformation of soft targets, is the softmax transformation of hard targets.

[0058] In the S23, the class anchor retains the class information from the class feature center, and the image classification model will strive to remember the center feature of each class of images in each task to ensure that the information most representative of the image class is retained; this helps to improve the overall performance of the image classification model in the class incremental learning task. In order to save the feature center of the old class, after each task is completed, the features of all images of the same class under the new task class of the current image classification model are calculated and weightedly averaged as the feature center of the class, when the new task comes, the image data of the old task in the cache area and the image data of the new task are input to the image classification model M t of the current task, for the image data of the old task, take the saved feature center of the old class; for the image data of the new task, calculate the feature of the same class of images of the class as the class feature center of the class, respectively do KL loss, by requiring the feature value of the current classification model to be close to the class feature center to retain the most representative information of the old class and the new class.

[0059] The loss definition of the class anchor is as follows:

[0060]

[0061] where n k represents the number of new classes k under the current batchsize. The CA function measures the difference between the predicted distribution and the class feature center distribution. By minimizing this loss, the current image classification model can remember the feature center of the entire class and thus improve the overall class incremental learning performance of the model. Our loss function can finally be defined as:

[0062] L total = L method + aL KB + bL CA

[0063] where a and b are hyperparameters that measure the importance of knowledge bridging and class anchoring, respectively, and L method is the training loss function of the original method.

[0064] Two widely used datasets, CIFAR100 and ImageNet-Subset, are used in this example. The CIFAR100 dataset contains 100 classes, with 500 test images and 100 training images per class. On the other hand, ImageNet-Subset is a subset of the ImageNet dataset, containing 60,000 images divided into 100 classes, each consisting of more than 13,000 images of size 224x224.

[0065] Our method is evaluated using two protocols. First, we follow the F0 protocol proposed by iCaRL, which divides the data of all 100 classes into 2, 5, and 10 incremental tasks. Second, we apply the F50 protocol proposed by LUCIR, which initially trains on half of the total classes (e.g., 50 classes). The remaining data is then split into tasks with 2, 5, and 10 classes.

[0066] The method of the invention is compared with several well-known class-incremental learning methods, including iCaRL (CVPR'17), LUCIR (CVPR'19), BiC (CVPR'19), PODNet (ECCV'20), DER (CVPR'21), and FOSTER (ECCV'22).

[0067] Standard metrics are used to evaluate class-incremental learning: mean accuracy, calculated as the average accuracy across all incremental tasks, and cumulative accuracy, defined as the accuracy of the model after completing all incremental tasks.

[0068] To ensure fairness, the same settings as all previous experiments [5, 6, 13, 7, 8, 9] are strictly followed, using the same learning rate, optimizer, buffer sample selection, distillation method, etc.

[0069] Figure 1 and Figure 2 Cumulative accuracy for various experimental settings on CIFAR100 and ImageNet-Subset is shown. Meanwhile, Table 1 and Table 2 detail the average accuracy for different experimental configurations.

[0070]

[0071] Table 1

[0072]

[0073] Table 2

[0074] The proposed KBCA integration into these methods significantly improves the accuracy to varying degrees. Specifically, when the method of the present application is applied to iCaRL, BiC, LUCIR, PODNet, DER and FOSTER on CIFAR100, the respective baseline average accuracy is increased by 0.37%, 0.77%, 0.36%, 2.08%, 2.13%, 1.61% respectively, while on ImageNet-Subset, they are increased by 0.40%, 0.50%, 0.48%, 0.25% respectively. These results show that the present method can be used for existing methods, which can improve the accuracy of existing models to varying degrees, and knowledge bridging and class anchoring can significantly improve the performance of the model.

[0075] The embodiments of the present application are merely illustrative of the implementation forms of the present application concept, and are only used for the purpose of description. The protection scope of the present application should not be regarded as being limited to the specific forms presented in the present embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the present application concept.

Claims

1. A class incremental learning method based on knowledge bridging and category anchoring, characterized in that, The method includes the following steps: Step 1: Train the image classification model and data cache, as follows: S11. Input: Image data of the new task and the cache of the previous task, wherein the cache contains image data of the previous task and the cache is initially empty; S12. Processing: Input the new task image data into the image classification model for training. After training, a portion of the new task image data will be added to the cache and mixed with the image data of the previous task. S13. Output: The trained image classification model and the updated image data buffer; Step Two: Knowledge Bridging and Category Anchoring, the process is as follows: S21. Input the updated image data buffer from step one together with the new task image data. S22. The image classification model will process new task image data and old task image data in the cache at the same time. The image classification model will calculate the classification loss of new task data and the distillation loss of old task data respectively, so as to adapt to the new task and consolidate the knowledge of the old task. S23. Distillation process: The image classification model uses knowledge bridging and category anchoring techniques to calculate the distillation loss and classification loss respectively. This includes extracting class-level association information and knowledge of class feature centers from new and old task categories, and applying this knowledge to the overall training process of the current image classification model. S24. Comprehensive Loss: Image classification models take into account both classification loss and distillation loss to optimize overall performance. This helps to control catastrophic forgetting and balance plasticity and stability. S25. The image classification model, after calculating distillation loss and classification loss, has adapted to the new task and taken into account the knowledge of the old task. In step S22, the goal of knowledge bridging is to learn new categories using features of old categories that the model has already learned. When learning a new task, the image classification model learns the association features between old and new categories, and learns and remembers the new category through these association features. This ensures that the features of the old categories are preserved when the model learns new category parameters and updates, preventing catastrophic forgetting. First, through the previous image classification model... Extracting features from image data x for a new task This feature, as a hard objective, represents the representation result of the new class using the old class features, and then, through the current image classification model... Extracting features from image data x for a new task Next, the feature output is... After dividing by the temperature parameter, a softmax transformation is performed to obtain the probability distribution of softening. Finally, a weighted average of the cross-entropy corresponding to the soft and hard targets is calculated to encourage the new class feature distribution of the current image classification model to align with the feature distribution of the previous image classification model, thereby achieving knowledge bridging between the old and new classes and alleviating the catastrophic forgetting problem.

2. The class incremental learning method based on knowledge bridging and category anchoring as described in claim 1, characterized in that, The definition of knowledge bridging loss is as follows: ； ； in, Indicates the category of the current task. Indicates the category of the previous task. It is the softmax transform of a soft target. It is a softmax transform for hard targets.

3. A class incremental learning method based on knowledge bridging and category anchoring as described in claim 1 or 2, characterized in that, In step S23, category anchoring retains category information from class feature centers. The image classification model strives to remember the central features of images of each category in each task to ensure that the information most representative of the image category is preserved. This helps improve the overall performance of the image classification model in class incremental learning tasks. To preserve the feature centers of old categories, after each task, the features of all images in the same category of the new task category for the current image classification model are calculated. and calculate the weighted average. As the feature center for this class, when a new task arrives, the image data from the old task in the cache and the image data from the new task are input together into the image classification model for the current task. In this process, for image data of old tasks, the old category feature centers are saved; for image data of new tasks, the weighted average of the features of the same type of images of that type is calculated as the category feature center of that type, and KL loss is applied to each type. By requiring the feature values ​​of the current classification model to be close to the category feature centers, the most representative information of the old and new types of classes is preserved.

4. The class incremental learning method based on knowledge bridging and category anchoring as described in claim 3, characterized in that, The loss for category anchoring is defined as follows: ； in, Indicates the new category under the current batch size. The number of features is used. The CA function measures the difference between the predicted distribution and the distribution of class feature centers. By minimizing this loss, the current image classification model can remember the feature centers of the entire class, thereby improving the overall class incremental learning performance of the model. The loss function is defined as: ； in, and These are hyperparameters that measure the importance of knowledge bridging and category anchoring. It is the training loss function of the original method.

Images