A similar prototype-based playback-based incremental learning method

Abstract

The application provides a similar prototype playback-based incremental learning method, which solves the catastrophic forgetting problem in incremental learning and uses old knowledge to assist in learning new tasks.The method comprises S1.model training, S2.prototype saving, and S3.target identification.The application has the beneficial effects that the similar class prototype playback, the similarity constraint regularization and the feature extractor normalization are comprehensively used, the catastrophic forgetting is effectively coped with, the model classification effect is improved, the new class learning is assisted, and excellent performance is achieved in actual target classification tasks.The application is very valuable for processing actual multi-class image classification problems.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning, and specifically relates to a class incremental learning method based on similar prototype playback. Background Technology

[0002] Image classification is a crucial task in computer vision, aiming to categorize input images into different classes or labels. Incremental class classification methods are often applied to real-world object classification tasks, such as image classification applications on smartphones. In such applications, the initial model can classify common categories like cats, dogs, and cars. However, as user demands and data grow, these applications need to continuously adapt to new categories, such as different breeds of dogs or car models. Incremental class classification methods enable these applications to continuously learn and adapt to new categories, thus providing more accurate and comprehensive object classification services that meet the diverse needs of real-world applications.

[0003] However, most current incremental methods face the following two challenges in image classification:

[0004] (1) The problem of catastrophic forgetting

[0005] Catastrophic forgetting is an inherent problem in continuous learning, characterized by the forgetting of previously learned knowledge, and it is also an unavoidable issue in backpropagation. This is because when a model learns a new class, it inevitably changes the network weights, causing the network to forget what it has learned before.

[0006] (2) Forward transfer

[0007] Previous incremental learning methods typically focused on preventing the forgetting of old categories when learning new ones, with little consideration for effectively utilizing previously learned knowledge of old categories to learn new ones. In real-world applications, knowledge of new and old categories is often correlated; therefore, forward transfer reflects the structure of real-world knowledge, making the model more suitable for handling real-world tasks. A good incremental learning method should be able to address the forgetting problem while allowing the model to retain and accumulate knowledge of old tasks. This knowledge from old tasks can help learn new tasks better, thereby improving overall performance.

[0008] In summary, to address the above challenges, a class incremental learning method based on similar prototype playback is proposed. Summary of the Invention

[0009] In view of the above problems, the purpose of this invention is to provide a class incremental learning method based on similar prototype playback.

[0010] An incremental learning method based on similar prototype playback includes the following steps:

[0011] S1. Model Training: Obtain the dataset at time t, including new class images and new class image labels, and train it on the standard image classification ResNet-18 neural network. The training methods include similar class prototype playback, similarity constraint regularization method and feature extractor normalization.

[0012] The aforementioned similar class prototype replay means that when learning a new class, the model learns knowledge from the old classes that is similar to the new class by replaying the prototypes of the old classes that are beneficial to the new class.

[0013] The regularization method for the aforementioned similarity constraint is to introduce regularization constraints into the training of the model to ensure that the model does not affect the performance of the old class when learning a new class; similarity is the measure of similarity between the new class and the old class.

[0014] The above feature extractor normalization is achieved by matching the new data features extracted by the current model with the features of the previous model, thereby forcing the adjustment of the feature extractor's behavior.

[0015] S2. Prototype saving: Save the model trained on the dataset at time t after completing the task, and save the prototypes of different categories in the dataset at time t. Merge the prototypes saved at time t with the prototypes saved before time t. The prototype is an abstract representation of the category, which is a vector in a high-dimensional feature space, representing the common features of the category. The common features are that the images in the category have similar features, which are representative features of this category.

[0016] S3. Perform target recognition: Input the target image into the trained model, and the model generates the final prediction result.

[0017] The ResNet-18 neural network described above consists of a feature extractor and a classifier. The feature extractor transforms the input image into a high-dimensional feature vector, which contains the abstract features of the image. The classifier extracts the high-dimensional feature vector from the feature extractor and maps it to the probability distribution of different categories to obtain the final prediction result.

[0018] The above-mentioned similarity class prototype replay includes similarity comparison and prototype selection replay:

[0019] Similarity comparison: Extract the high-dimensional feature vector of the new class sample, compare the feature vector of the new class sample with the old class prototype to determine similarity, and the degree of similarity is expressed as:

[0020]

[0021] Where x represents the new class of images, Sim(·,·) represents the similarity between the two, cos(·,·) represents the cosine function, and K... cLet φ(x) represent the prototype of category c, φ(x) represent the high-dimensional feature vector of the image, and ‖·‖ represent the magnitude of the vector;

[0022] Prototype Replay Selection: Prototype replay with similarity greater than the similarity threshold ε is selected. The prototype replaying at time t is enhanced as follows:

[0023]

[0024] Here, θ~N(0,1) represents Gaussian noise, which has the same dimensions as the saved prototype. This represents the enhanced feature of category c at time t. The parameter δ is used to dynamically adjust the weights.

[0025] When t = 1, the parameter δ satisfies:

[0026]

[0027] When t>1, the parameter δ satisfies:

[0028]

[0029] Where N1 represents the number of categories learned by the model at t=1, N old N new Let ∑ represent the number of the old class and the number of the new class at time t, respectively. t,k Let be the covariance matrix of the k-class features at time t, Tr denote the trace of the matrix, D is the dimension of the feature space, and sqrt(·) denotes the square function. Then, the enhanced class knowledge is selectively fed into the classifier, and backpropagation is performed according to the following loss function formula.

[0030]

[0031] Among them, L protoAug Y represents the value of the replay loss function, w represents the number of prototypes selected for replay, L(·,·) is the cross-entropy loss, G(·) is the result of the classifier mapping features to different class probability distributions using the Softmax normalization function, and Y... i This is the tag corresponding to the old class prototype.

[0032] The regularization method for the aforementioned similarity constraint is based on the similarity between the feature vectors of the new class image and the old class prototype. The high-dimensional feature vectors of the new class image and the old class prototype are normalized and used as probability distributions, and backpropagation is performed according to the following loss function formula:

[0033]

[0034]

[0035] Among them, Lkl P(x) represents the value of the regularization loss function. i Let Q(x) be the probability distribution of the new class. j Let λ represent the probability distribution of the old class, m and n represent the number of new and old classes respectively, and λ represent the similarity constraint weights. The formula is shown below:

[0036]

[0037] Here, arccos(·,·) represents the inverse cosine function.

[0038] During the learning of new categories, the feature extractor is continuously updated, which can lead to a mismatch between the stored old knowledge and the feature extractor. To address this issue, knowledge distillation is used to normalize the feature extractor. This is achieved by matching the new class image features extracted by the model at time t with the new class image features extracted by the model at time t-1, thereby forcibly adjusting the behavior of the feature extractor. The matching formula is as follows:

[0039] L kd =||F t (x)-F t-1 (x)‖,

[0040] Where F t (·) represents the feature extractor of the current model, F t-1 (·) indicates the feature extractor of the previous stage model.

[0041] The above prototype storage process involves extracting features from images of each known category. The mean of the high-dimensional feature vector for each category represents the category prototype. The calculated prototypes are stored in a prototype container for use in subsequent incremental learning. The specific operation for prototype calculation is as follows:

[0042]

[0043] Among them, K d Let be the prototype of category d, h represent the number of images of category d, and φ(x) be the prototype of category d. i This represents the high-dimensional feature vector of each sample.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. From the perspective of knowledge reuse, a similar category feature-based knowledge replay method is proposed. This method not only addresses the forgetting problem but also enables the knowledge learned from previous tasks to aid in learning new tasks.

[0046] 2. From the perspective of addressing catastrophic forgetting, a similarity-adjusted regularization method is proposed. This method allows the model to dynamically adjust weights based on the similarity between new category samples and old category knowledge during continuous learning, while balancing the learning of new and old categories through regularization to solve the catastrophic forgetting problem.

[0047] 3. By combining replay and regularization methods, we maximize the forward transition of the model while minimizing catastrophic forgetting. Extensive experiments on benchmark datasets demonstrate that our method offers significant performance improvements in incremental scenarios. Attached Figure Description

[0048] Figure 1 This is the overall flowchart.

[0049] Figure 2 It is the input classification image.

[0050] Figure 3 It is a framework for incremental learning methods based on similar prototype playback.

[0051] Figure 4 This is a comparison chart of experimental results. Detailed Implementation

[0052] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0053] An incremental learning method based on similar prototype playback. The overall flowchart is attached. Figure 1 As shown. Specifically, it includes the following steps:

[0054] S1. Model Training: Obtain the dataset at time t, including new class images and new class image labels, and train it on the standard image classification ResNet-18 neural network. The training methods include similar class prototype playback, similarity constraint regularization method and feature extractor normalization.

[0055] The feature extractor transforms the input image data into a high-dimensional feature vector, which contains abstract features of the image. The classifier extracts the high-dimensional feature vector from the feature extractor, maps it to the probability distribution of different categories, and obtains the final prediction result. (See attached image) Figure 2 The image shown is the input image, with a resolution of 32×32 pixels and the image label is "apple".

[0056] Incremental learning of classes includes similarity class prototype replay, regularization methods for similarity constraints, and feature extractor normalization. The overall framework is as follows: Figure 3 As shown.

[0057] Similarity comparison: Extract the high-dimensional feature vector of the new class sample, compare the feature vector of the new class sample with the old class prototype to determine similarity, and the degree of similarity is expressed as:

[0058]

[0059] Where x represents the new class of images, Sim(·,·) represents the similarity between the two, cos(·,·) represents the cosine function, and K... c Let φ(x) represent the prototype of category c, φ(x) represent the high-dimensional feature vector of the image, and ‖·‖ represent the magnitude of the vector;

[0060] Prototype Replay Selection: Prototype replay with similarity greater than the similarity threshold ε is selected. The prototype replaying at time t is enhanced as follows:

[0061]

[0062] Here, θ~N(0,1) represents Gaussian noise, which has the same dimensions as the saved prototype. This represents the enhanced feature of category c at time t. The parameter δ is used to dynamically adjust the weights, and when t=1, the parameter δ satisfies:

[0063]

[0064] When t>1, the parameter δ satisfies:

[0065]

[0066] Where N1 represents the number of categories learned by the model at t=1, N old N new Let ∑ represent the number of the old class and the number of the new class at time t, respectively. t,k Let be the covariance matrix of the k-class features at time t, Tr denote the trace of the matrix, D is the dimension of the feature space, and sqrt(·) denotes the square function. Then, the enhanced class knowledge is selectively fed into the classifier, and backpropagation is performed according to the following loss function formula:

[0067]

[0068] Among them, L protoAug Y represents the value of the replay loss function, w represents the number of prototypes selected for replay, L(·,·) is the cross-entropy loss, G(·) is the result of the classifier mapping features to different class probability distributions using the Softmax normalization function, and Y... i This is the tag corresponding to the old class prototype.

[0069] The similarity-constrained regularization method uses the similarity between the feature vectors of the new class image and the old class prototype to constrain the regularization method. The high-dimensional feature vectors of the new class image and the old class prototype are normalized and used as probability distributions. Backpropagation is then performed according to the following loss function formula:

[0070]

[0071] Among them, L kl P(x) represents the value of the regularization loss function. i Let Q(x) be the probability distribution of the new class. j Let λ represent the probability distribution of the old class, m and n represent the number of new and old classes respectively, and λ represent the similarity constraint weights. The formula is shown below:

[0072]

[0073] Here, arccos(·,·) represents the inverse cosine function.

[0074] During the learning of new categories, the feature extractor is continuously updated, which can lead to a mismatch between the stored old knowledge and the feature extractor. To address this issue, knowledge distillation is used to normalize the feature extractor. This is achieved by matching the new data features extracted by the current model with the features from previous models, thereby forcing adjustments to the feature extractor's behavior.

[0075] L kd =||F t (x)-F t-1 (x)‖,

[0076] Where F t (·) represents the feature extractor of the current model, F t-1 (·) indicates the feature extractor of the previous stage model.

[0077] S2. Prototype Saving: Save the prototype of the data category at time t and merge it with the prototype saved before time t for subsequent use. The prototype is an abstract representation of the category, which is a point in a high-dimensional feature space that represents the common features of the category. The common features are that the samples in the category have similar features, which are representative characteristics of this category.

[0078] Prototype storage involves extracting features from images of each known category. The mean of the high-dimensional feature vector for each category represents the category prototype. The calculated prototypes are stored in a prototype container for use in subsequent incremental learning. The specific operations for prototype calculation are as follows:

[0079]

[0080] Among them, Kd Let be the prototype of category d, h represent the number of images of category d, and φ(x) be the prototype of category d. i This represents the high-dimensional feature vector of each sample.

[0081] S3. Perform target recognition: Input the target image into the trained model, and the model generates the final prediction result.

[0082] The following simple application example illustrates the application of the proposed method for image classification:

[0083] The Tiny ImageNet multi-class image classification dataset was selected. This dataset includes various animals, plants, vehicles, and everyday objects, with a rich variety of image categories, which can better simulate complex multi-class classification tasks in the real world. Each category has 500 training images and 50 validation images. The experiment was set up as follows: the first learning task contained 100 categories, and the subsequent 20 tasks each contained 5 categories, requiring the model to learn these multi-class classification tasks step by step. In the experiment, the present invention was compared with EWC[1], LwF[2], LwF-MC[2], MUC[3], IL2A[4], PASS[5], iCaRL[6], EEIL[7] and LUCIR[8]. The experimental results are attached. Figure 4 As shown. From Figure 4 It can be observed that as the number of tasks increases, when the number of categories reaches 135, the proposed method shows a significant advantage over other methods. This advantage not only gradually increases but also continues until the end of the experimental task, further demonstrating the superior performance of this method in the incremental image classification problem.

[0084] This application example illustrates that the algorithm provided by this invention can handle large-scale multi-class image classification problems in real life, and it is of great significance in solving such problems.

[0085] References:

[0086] [1]J.Kirkpatrick,Razvan Pascanu,Neil C.Rabinowitz,J.Veness,G.Desjardins,Andrei A.Rusu,K.Milan,John Quan,Tiago Ramalho,Agnieszka Grabska-Barwinska,Demis Hassabis,CClopath,D.Kumaran,and Raia Hadsell.Overcomingcatastrophic forgetting in neural networks.

[0087] Proceedings of the National Academy of Sciences, pages 3521–3526,2017.

[0088] [2]Zhizhong Li and Derek Hoiem.IEEETrans.Pattern Anal.Mach.Intell.,pages 2935–2947,2018.

[0089] [3]Yu Liu,Sarah Parisot,Gregory G.Slabaugh,Xu Jia,Ales Leonardis,andTinne Tuytelaars.More classifiers,less forgetting:A generic multi-classifier paradigm for incremental learning.In ECCV,pages 699–716,2020.

[0090] [4]Zhu,Fei,et al."Class-incremental learning via dual augmentation."Advances in Neural Information Processing Systems 34(2021):14306-14318.

[0091] [5]Zhu,Fei,et al."Prototype augmentation and self-supervision forincremental learning."Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition.2021.

[0092] [6]Sylvestre-Alvise Rebuffi,A.Kolesnikov,Georg Sperl,and ChristophH.Lampert.icarl:Incremental classifier and representation learning.In CVPR,pages 5533–5542,2017

[0093] [7]Francisco M Castro,Manuel J Marín-Jiménez,Nicolás Guil,CordeliaSchmid,and Karteek Alahari.End-to-end incremental learning.In ECCV,pages 233–248,2018.

[0094] [8]Saihui Hou,Xinyu Pan,Chen Change Loy,Zilei Wang,and D.Lin.Learninga unified classifier incrementally via rebalancing.In CVPR,pages 831–839,2019.

Claims

1. A class incremental learning method based on similar prototype playback, characterized in that, Includes the following steps: S1. Model Training: Obtaining The dataset at each time step includes images of new classes and their labels. It is trained on the standard image classification ResNet-18 neural network. The training methods include similar class prototype playback, regularization of similarity constraints, and normalization of feature extractors. The similarity class prototype replay refers to the process of learning knowledge beneficial to the new class by replaying the prototypes of old classes similar to the new class during the learning of a new class. The similarity class prototype replay includes similarity comparison and prototype selection for replay. Similarity comparison: Extract the high-dimensional feature vector of the new class image, and determine similarity by comparing the feature vector of the new class image with the old class prototype. The degree of similarity is expressed as: ， Where x represents the new class of image, Indicates the similarity between the two. Represents the cosine function. Indicate category The prototype, Represents the high-dimensional feature vector of an image. Represents the magnitude of a vector; Select Prototype Replay: Select prototype replays with a similarity greater than the similarity threshold. The prototype for time-replay has been enhanced to: ， in, This represents Gaussian noise and has the same dimensions as the saved prototype. Indicate category exist Time-enhanced features, parameters Used for dynamically adjusting weights; when At that time, parameters satisfy: ， when At that time, parameters satisfy: ， in, express The number of categories learned by the model at that time. , They represent The number of old and new classes at any given moment. It is the first time The covariance matrix of class features, where Tr represents the trace of the matrix. It is the dimension of the feature space, sqrt( () represents the square function. The enhanced class knowledge obtained is then selectively fed into the classifier, and backpropagation is performed according to the following loss function formula: ， in, This represents the value of the replay loss function, where w represents the number of prototypes selected for replay. For cross-entropy loss, For use in classifiers The normalization function maps features to the results of different class probability distributions. The tag corresponding to the old class prototype; The regularization method for the similarity constraint is to introduce regularization constraints into the training of the model to ensure that the model does not affect the performance of the old class when learning a new class; similarity is the measure of similarity between the new class and the old class. The feature extractor normalization is achieved by matching the new data features extracted by the current model with the features of the previous model, thereby forcibly adjusting the behavior of the feature extractor. S2. Prototype Saving: Complete the task. Save the model trained on the time-based dataset and save it. The different categories of prototypes in the time-based dataset will Prototypes saved at all times and The prototypes saved before the time step are merged. The prototype is an abstract representation of the category. It is a vector in a high-dimensional feature space that represents the common features of the category. The common features are that the images in the category have similar features, which are representative characteristics of this category. S3. Perform target recognition: Input the target image into the trained model, and the model generates the final prediction result.

2. The class incremental learning method based on similar prototype playback according to claim 1, characterized in that, The ResNet-18 neural network consists of a feature extractor and a classifier. The feature extractor converts the input image into a high-dimensional feature vector, which contains the abstract features of the image. The classifier extracts the high-dimensional feature vector from the feature extractor and maps it to the probability distribution of different categories to obtain the final prediction result.

3. The class incremental learning method based on similar prototype playback according to claim 1, characterized in that, The regularization method for the similarity constraint is based on the similarity between the feature vectors of the new class image and the old class prototype. The high-dimensional feature vectors of the new class image and the old class prototype are normalized and used as probability distributions. Backpropagation is performed according to the following loss function formula: ， in, This represents the value of the regularization loss function. It is the probability distribution of the new class. It is the probability distribution of the old class. and These represent the number of new and old classes, respectively. The similarity constraint weights are represented by the following formula: ， in, This represents the inverse cosine function.

4. The class incremental learning method based on similar prototype playback according to claim 1, characterized in that, The feature extractor normalization employs knowledge distillation to normalize the feature extractor. This is achieved by matching the new class of image features extracted by the model at time t with the new class of image features extracted by the model at time t-1, thereby forcibly adjusting the behavior of the feature extractor. The matching formula is as follows: ， in, This represents the value of the loss function for feature matching. This represents the feature extractor of the model at time t. This represents the feature extractor of the model at time t-1.

5. The class incremental learning method based on similar prototype playback according to claim 1, characterized in that, The prototype storage process involves extracting features from images of each known category. The mean of the high-dimensional feature vector for each category represents the category prototype. The calculated prototypes are stored in a prototype container for use in subsequent incremental learning. The specific operation for prototype calculation is as follows: ， in, Indicate category The prototype, h represents the category The number of images, This represents the high-dimensional feature vector of each image.

Images