Class increment image classification method based on prototype learning

Abstract

The invention belongs to the technical field of image classification, and relates to a prototype learning-based class increment image classification method, which comprises the following steps of: obtaining image data, and constructing a first data set and a second data set; the first data set and the second data set respectively comprise different sample pairs; constructing an initial model, and training the initial model by adopting the first data set to obtain a teacher model; obtaining sample features of the first data set according to a teacher model; obtaining prototype learning loss according to sample features of the first data set; obtaining the total loss of the model according to the prototype learning loss; based on the total loss of the model, training the teacher model by using the second data set to obtain a student model; and classifying the class increment images by adopting a student model. By adopting the method, the problem of disastrous forgetting can be effectively relieved, and meanwhile, the stability and generalization ability of the model are improved.

Description

Technical Field

[0001] The present application relates to the technical field of image classification, and in particular to a class-incremental image classification method based on prototype learning. Background Art

[0002] In recent years, deep learning has achieved performance close to or even surpassed that of humans in various tasks. However, for learning new knowledge, the current paradigm mostly adopts the method of adding new data to old data to retrain the model, which is significantly different from human learning. Humans can remember old knowledge and continuously learn and accumulate new knowledge on this basis. This leads to incremental learning, which is the ability of the model to learn new data inputs instead of retraining the model with new and old data after learning old knowledge. This is very similar to the way humans learn new knowledge. However, this creates a new problem, catastrophic forgetting. Catastrophic forgetting refers to the serious degradation of the performance of the incremental learning model on old data after learning new data. Therefore, it is necessary to explore a solution that enables the model to learn new knowledge well while reducing the catastrophic forgetting of old knowledge, which gives rise to quasi-incremental learning.

[0003] Class incremental learning is a major application direction of incremental learning, which mainly applies incremental learning to classification problems. Class incremental learning is to incrementally learn new categories based on a trained classification model so that the new model can correctly classify the input data.

[0004] In the prior art, methods of incremental learning can be roughly divided into four types: regularization-based methods, example-based methods, bias correction-based methods, and knowledge distillation methods.

[0005] (1) When solving the catastrophic forgetting problem in incremental learning, the regularization method is to add a regularization term to the loss function of the new model, combine the importance of each parameter in the loss function of the original model, and penalize large changes in important parameters, thereby avoiding large changes in weight parameters that are closely related to the classification ability of the old model.

[0006] (2) The example-based method imitates the process of "human beings frequently review old knowledge in order to avoid forgetting old knowledge during the process of learning knowledge". In the process of learning new data, a small subset of the old data set is added to the new data set for training, thereby maintaining the memory of old knowledge. This can alleviate the problem of catastrophic forgetting to a certain extent.

[0007] (3) The bias correction method is to identify the difference between the data distribution when learning new data and the data distribution of the old knowledge learned, and make corresponding adjustments. For example, when the model learns new category data, it often assigns higher weights to the newly introduced categories compared to the old categories. Bias correction reduces the problem of catastrophic forgetting by identifying this difference and adjusting the uneven weights.

[0008] (4) The knowledge distillation method regards the new and old models as the student model and the teacher model respectively. The teacher model can learn to classify the old categories, while the student model can learn new categories under the guidance of the teacher model's classification method of the old categories. The training is mainly guided by cross entropy loss and distillation loss. The cross entropy loss function is used to guide the student model to learn the new categories. The distillation loss strictly aligns the output of the student model with that of the teacher model, guiding the student model to learn the teacher model's learning of the old categories while learning the new categories.

[0009] However, the above method has the following disadvantages.

[0010] (1) Regularization-based methods mainly focus on parameter changes rather than retaining old task knowledge, which has been proven to be insufficient to effectively alleviate the problem of catastrophic forgetting.

[0011] (2) Example-based methods need to store a small subset of old category data sets, which requires a certain amount of storage space. When the input data of the model is large or the number of incremental learning times is too large, the storage space requirement will increase dramatically. In addition, due to confidentiality requirements for some financial, medical and other data, access to historical data is often not allowed. Therefore, in this case, example-based methods cannot be implemented normally.

[0012] (3) Bias correction-based methods correct the bias that may be introduced by learning new tasks by adjusting the model; however, if the distribution difference between the old and new tasks is large, this method may not be sufficient to maintain the performance on the old tasks.

[0013] (4) The knowledge distillation method can consolidate the learned knowledge through the old model. It has been widely used in incremental learning. It can help the model overcome the catastrophic forgetting problem of old categories through point-to-point distillation loss. However, in example-free incremental learning, the new model does not have access to the old task data. Even if there are prototype samples of the old category, there is a serious imbalance between the old and new categories in the training data set. Therefore, the new model will be biased towards the new category, resulting in significant differences in the predictions of the old and new models for the old categories. In addition, the cross entropy loss in the knowledge distillation training process tends to constrain the model to better learn the new category, that is, to assign higher weights to the new category, while the distillation loss constrains the output probabilities of the student model and the teacher model to be strictly point-to-point the same. However, the teacher model outputs a higher probability for the old category data. There is a certain contradiction between the two loss functions, which leads to the model tending to optimize the simple cross entropy loss rather than the complex distillation loss during training. Therefore, although traditional knowledge distillation has a certain effect in solving catastrophic forgetting, it is not very outstanding. Summary of the invention

[0014] Based on this, it is necessary to provide a class-incremental image classification method based on prototype learning to address the above technical problems, which can effectively alleviate the catastrophic forgetting problem and improve the stability and generalization ability of the model.

[0015] The class-incremental image classification method based on prototype learning includes: Acquire image data and construct a first data set and a second data set; the first data set and the second data set respectively contain different sample pairs; Constructing an initial model, and using the first data set to train the initial model to obtain a teacher model; According to the teacher model, the sample features of the first data set are obtained; according to the sample features of the first data set, the prototype learning loss is obtained; according to the prototype learning loss, the total loss of the model is obtained; based on the total loss of the model, the teacher model is trained using the second data set to obtain the student model; The student model is used to classify class increment images.

[0016] In one embodiment, the method further includes: after obtaining the prototype learning loss, using the Pearson coefficient to measure the advantage of the correlation, calculating the inter-class relationship loss and the intra-class relationship loss, and obtaining the distillation loss; According to the prototype learning loss, distillation loss and cross entropy loss, the total loss of the model is obtained.

[0017] In one embodiment, the Pearson coefficient is used to measure the advantage of correlation, and the inter-class relationship loss and the intra-class relationship loss are calculated to obtain the distillation loss, including: Calculate the Pearson coefficient and get the Pearson distance; According to the Pearson distance, calculate the inter-class relationship loss and the intra-class relationship loss; According to the inter-class relationship loss and the intra-class relationship loss, the distillation loss is obtained.

[0018] In one embodiment, according to the sample characteristics of the first data set, the prototype learning loss is obtained, including: According to the sample features of the first data set, the feature mean is calculated to obtain the original prototype set; Add noise to the original prototype set to obtain the extended prototype set; The extended prototype set is input into the classifier to obtain the prototype learning loss.

[0019] In one embodiment, based on the sample features of the first data set, the feature mean is calculated to obtain the original prototype set:

[0020] In the formula, For the original prototype set, For the images, The first category of the dataset The number of all images, For extraction The function corresponding to the feature, is an input image of a certain category, for different rotation angles. In one embodiment, noise is added to the original prototype set to obtain an extended prototype set:

[0021] In the formula, To expand the prototype set, ~N(0,1) is Gaussian noise distribution, is the average variance of the class features.

[0022] In one embodiment, the extended prototype set is input into the classifier to obtain the prototype learning loss, including:

[0023] In the formula, is the prototype learning loss, For the categories, is the number of old categories, For and Do cross entropy loss, For The category result after the classifier, is the correct category result in the label.

[0024] In one embodiment, the Pearson coefficient is calculated and the Pearson distance is obtained, including:

[0025]

[0026] In the formula, for and The Pearson coefficient, is the predicted value of the teacher model for the image, is the predicted value of the student model for the image, is the covariance, is the variance, For the categories, is the number of old categories, Predicting images as categories for the teacher model The probability of for The mean of Predicting images as categories for the student model The probability of for The mean of for and The Pearson distance. In one embodiment, according to the Pearson distance, the inter-class relationship loss and the intra-class relationship loss are calculated, including:

[0027]

[0028] In the formula, is the inter-class relationship loss, is the intra-class relationship loss, is the total number of samples, is the number of old categories, is the feature matrix f No. OK, is the feature matrix List, for and The Pearson distance, is the teacher model for the feature matrix The prediction results of the line, The student model is the feature matrix The prediction results of the line for and The Pearson distance, is the teacher model for the feature matrix The prediction results of the column, The student model is the feature matrix The predicted results of the column.

[0029] In one embodiment, the distillation loss is obtained according to the inter-class relationship loss and the intra-class relationship loss, including:

[0030] In the formula, is the distillation loss, is the weight of the inter-class relationship loss, is the inter-class relationship loss, is the weight of the intra-class relationship loss, is the intra-class relationship loss.

[0031] The above-mentioned class-incremental image classification method based on prototype learning designs a prototype-based method and a correlation-based distillation loss method, which solves the problem of forgetting old knowledge due to the inability to save a large number of examples in the training model due to limited storage and the inability to access examples due to security and privacy considerations. Compared with mainstream class-incremental learning, it can better solve the problem of catastrophic forgetting of knowledge in the absence of examples, and at the same time has higher classification accuracy, better model stability and generalization performance. It is particularly suitable for image classification scenarios with example-free class-incremental learning. It is simple but extremely effective, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an application scenario diagram of a class incremental image classification method based on prototype learning in one embodiment; Figure 2 Schematic diagram of a process of a class incremental image classification method based on prototype learning in one embodiment; Figure 3 Schematic diagram of inter-class loss Inter and intra-class loss Intra in one embodiment; Figure 4 A diagram showing the effect of rotation in one embodiment; Figure 5 is a graph showing the accuracy of different methods in an incremental step of S=5 in one embodiment; Figure 6 is a graph showing the accuracy of different methods in an incremental step of S=10 in one embodiment; Figure 7 is a graph showing the accuracy of different methods at an incremental step of S=20 in one embodiment; Figure 8 A heat map of PASS on CIFAR-100 and a corresponding step accuracy map in one embodiment; Fig. 9 A heat map of SSRE on CIFAR-100 and a corresponding step accuracy map in one embodiment; Fig.10 A heat map of the method of the present application on CIFAR-100 and a corresponding step accuracy map in an embodiment; Fig.11 This is a confusion matrix diagram of Finetune at 20 incremental steps in one embodiment; Fig.12 is a confusion matrix diagram of PASS in 20 incremental steps in one embodiment; Fig.13 This is a confusion matrix diagram of the method of the present application in 20 incremental steps in one embodiment. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0034] In addition, the descriptions of "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "multiple groups" means at least two groups, such as two groups, three groups, etc., unless otherwise clearly and specifically defined.

[0035] In this application, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0036] In addition, the technical solutions between the various embodiments of the present application can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0037] The class-incremental image classification method based on prototype learning provided in this application can be applied to Figure 1 In the application scenario diagram shown, the terminal 102 communicates with the server 104 through the network, and the terminal 102 may include but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 may be various portal websites, servers corresponding to the backend of the work system, etc.

[0038] This application provides a class-incremental image classification method based on prototype learning, such as Figure 2 The flowchart shown in FIG. 1 shows a method for applying the method to Figure 1 The following is an example of a terminal in the example, including: Step 202 , acquiring image data, and constructing a first data set and a second data set; the first data set and the second data set respectively contain different sample pairs.

[0039] In this step, the first data set and the second data set have different samples and corresponding sample labels. A sample and the corresponding sample label constitute a sample pair, and the sample label is used to represent the category of the sample. The first data set contains old categories, and the second data set contains new categories. In other words, the categories contained in the two are different, that is, the first data set and the second data set respectively contain sample pairs of different categories.

[0040] The first dataset is used as the initial dataset and is subsequently used to train the teacher model for image classification; the second dataset is used as the newly added category dataset and is subsequently used to train the student model's recognition and classification capabilities for new categories.

[0041] Step 204: construct an initial model, and use the first data set to train the initial model to obtain a teacher model.

[0042] In this step, how to construct the initial model, how to train the initial model, and how to obtain the teacher model are all existing technologies and will not be elaborated here.

[0043] Step 206, according to the teacher model, obtain the sample characteristics of the first data set; according to the sample characteristics of the first data set, obtain the prototype learning loss; according to the prototype learning loss, obtain the total model loss; based on the total model loss, use the second data set to train the teacher model to obtain the student model.

[0044] Specifically: According to the teacher model, the sample features of the first data set are obtained; according to the sample features of the first data set, the feature mean is calculated to obtain the original prototype set; noise is added to the original prototype set to obtain the extended prototype set: the extended prototype set is input into the classifier to obtain the prototype learning loss; according to the prototype learning loss and the cross entropy loss, the total model loss is obtained; based on the total model loss, the second data set is used to train the teacher model to obtain the student model.

[0045] More specifically: According to the teacher model, sample features of the first data set are obtained (how to obtain the features is known in the prior art); According to the sample features of the first data set, the feature mean is calculated to obtain the original prototype set:

[0046] In the formula, For the original prototype set, For the images, The first category of the dataset The number of all images, For extraction The function corresponding to the feature, is an input image of a certain category, For different rotation angles, the data set is rotated according to different rotation angles to strengthen the labels; Noise is added to the original prototype set to obtain more prototypes by expanding the prototype of each old class, and the extended prototype set is obtained:

[0047] In the formula, To expand the prototype set, ~N(0,1) is Gaussian noise distribution, is the average variance of the class feature; Input the expanded prototype set into the classifier and obtain the prototype learning loss:

[0048] In the formula, is the prototype learning loss, For the categories, is the number of old categories, For and Do cross entropy loss, For The category result after the classifier, The correct category result in the label; According to the prototype learning loss and cross entropy loss, the total loss of the model is obtained; Based on the total loss of the model, the teacher model is trained using the second data set to obtain the student model.

[0049] In this step, a prototype-based method is designed, and the prototype learning loss is used to constrain model training so that the model can learn and maintain the prototype of the first data set; during the training process, the student model obtained through the guidance of the teacher model and the loss constraints of each part of the total loss can maintain the recognition ability of the old categories while being able to recognize and classify new categories.

[0050] Step 208, using the student model to classify the class increment image.

[0051] In this step, a new model is used for classification to overcome the problem of catastrophic forgetting.

[0052] In another embodiment, step 206 also includes: after obtaining the prototype learning loss, using the Pearson coefficient to measure the advantage of correlation, calculating the inter-class relationship loss and the intra-class relationship loss, and obtaining the distillation loss; according to the prototype learning loss, the distillation loss and the cross entropy loss, the total model loss is obtained.

[0053] That is, step 206 is: according to the teacher model, obtain the sample characteristics of the first data set; according to the sample characteristics of the first data set, obtain the prototype learning loss; use the Pearson coefficient to measure the advantage of correlation, calculate the inter-class relationship loss and the intra-class relationship loss, and obtain the distillation loss; according to the prototype learning loss, the distillation loss and the cross entropy loss, obtain the total model loss; based on the total model loss, use the second data set to train the teacher model to obtain the student model.

[0054] Specifically: According to the teacher model, the sample features of the first data set are obtained; according to the sample features of the first data set, the feature mean is calculated to obtain the original prototype set; noise is added to the original prototype set to obtain the extended prototype set: the extended prototype set is input into the classifier to obtain the prototype learning loss; the Pearson coefficient is calculated to obtain the Pearson distance; according to the Pearson distance, the inter-class relationship loss and the intra-class relationship loss are calculated; according to the inter-class relationship loss and the intra-class relationship loss, the distillation loss is obtained; according to the prototype learning loss, the distillation loss and the cross entropy loss, the total model loss is obtained; based on the total model loss, the second data set is used to train the teacher model to obtain the student model.

[0055] More specifically: According to the teacher model, the sample features of the first data set are obtained; According to the sample features of the first data set, the feature mean is calculated to obtain the original prototype set:

[0056] In the formula, For the original prototype set, For the images, The first category of the dataset The number of all images in the first dataset is c The number of training samples for each class, For extraction The function corresponding to the feature, is an input image of a certain category, For different rotation angles; Add noise to the original prototype set to get the extended prototype set:

[0057] In the formula, To expand the prototype set, ~N(0,1) is Gaussian noise distribution, is the average variance of the class feature; Input the expanded prototype set into the classifier and obtain the prototype learning loss:

[0058] In the formula, is the prototype learning loss, For the categories, is the number of old categories, For and Do cross entropy loss, For The category result after the classifier, The correct category result in the label; Calculate the Pearson coefficient and get the Pearson distance:

[0059]

[0060] In the formula, for and The Pearson coefficient ranges from -1 to +1, 0 means no correlation, -1 and +1 mean a completely linear relationship, a positive correlation means that if one increases, the other also increases, and a negative correlation means that if one increases, the other decreases; is the teacher model's prediction of the image (specifically, the teacher model identifies an image as a vector of probabilities of different categories), is the predicted value of the student model for the image, is the covariance, which determines the linear correlation of the two variables. The two variables here refer to the prediction probabilities of each category obtained by the teacher model and the student model for the same input image. is the variance, For the categories, is the number of old categories, Predicting images as categories for the teacher model The probability of for The mean of Predicting images as categories for the student model The probability of for The mean of for and The Pearson distance measures the linear correlation between two variables rather than the match of absolute values, which means that it focuses on the relative relationship between the outputs rather than their specific values. The smaller the Pearson distance value, the greater the correlation between the two predictions. According to the Pearson distance, the inter-class relationship loss and the intra-class relationship loss are calculated:

[0061]

[0062] In the formula, is the inter-class relationship loss, is the intra-class relationship loss, is the total number of samples, is the number of old categories (specifically the total number of categories in the first dataset), is the feature matrix f No. OK, is the feature matrix Column, feature matrix f is obtained from the teacher model, the feature matrix f Depend on K The prediction results are composed of feature matrix f The size is K × O , K × O express K OK O List, for and The Pearson distance, is the teacher model for the feature matrix The prediction results of the line The student model is the feature matrix The prediction results of the line for and The Pearson distance, is the teacher model for the feature matrix The prediction results of the column, The student model is the feature matrix The prediction results of the column; According to the inter-class relationship loss and intra-class relationship loss, the distillation loss is obtained:

[0063] In the formula, is the distillation loss, is the weight of the inter-class relationship loss, is the inter-class relationship loss, is the weight of intra-class relationship loss, is the intra-class relationship loss; According to the prototype learning loss, distillation loss and cross entropy loss, the total loss of the model is obtained:

[0064] in,

[0065]

[0066] In the formula, is the total loss of the model, is the prototype learning loss, is the distillation loss, is the cross entropy loss, is the inter-class relationship loss, is the intra-class relationship loss, is the weight of the prototype learning loss, is the weight of the inter-class relationship loss, is the weight of the intra-class relationship loss; for x and y The cross entropy loss is For new category dataset, m is the number of new samples, Represent the input image and label respectively, For the categories, is the number of old categories, is the number of new categories, is an indicator function, which is 1 if y is equal to c. is the output probability of the cth category, and the logarithm is taken to facilitate comparison in logarithmic space; Based on the total loss of the model, the teacher model is trained using the second data set to obtain the student model.

[0067] In this embodiment, the features learned by the teacher model and the features learned by the student model are used to calculate the inter-class relationship loss and the intra-class relationship loss.

[0068] like Figure 3 As shown in the figure, the value of a row in the prediction matrix represents the input of a sample data, and the model predicts it. Since the model does not directly give the prediction result, but gives a probability distribution of different categories for the input data, the class corresponding to the value with the largest probability is taken as the prediction result. Therefore, by comparing the probability distribution of the new and old models for the same input in different categories, the new model (student model) is allowed to learn the order of the probability of the prediction value of the old model (teacher model) for an input in different categories. This is a relationship between different classes, corresponding to the inter loss part.

[0069] In the prediction matrix, the columns represent the predicted probability values ​​of the corresponding categories for the input of different sample data in a batch. By letting the new model learn the probability of the old model predicting a certain category in multiple input sample data, the new model can learn the characteristics of a class, that is, the intra-class relationship, corresponding to the intra loss.

[0070] In traditional point-to-point distillation, strict numerical matching may cause the model to face contradictions during the optimization process. Specifically, when the model optimizes the cross entropy loss (for new classes) and the traditional distillation loss (for old classes) at the same time, it may cause the student model to be affected if the output probability of the teacher model for the old class is very high, and it may be biased towards the old class, resulting in poor learning effect on the new class. This application adopts a correlation-based distillation loss. When calculating the inter-class relationship loss and the intra-class relationship loss, it only focuses on the relative order of probabilities to ensure that the order of the class probabilities predicted by the new model is the same as that of the old model, that is, it only loosely matches the correlation of the output probabilities without forcing an exact match, which can reduce the sensitivity to noise and alleviate this contradiction; especially when dealing with complex feature spaces, some features may be disturbed due to the imbalance of training data. By focusing on correlation rather than absolute values, the model can better adapt to such changes, thereby improving stability, which is particularly important for incremental learning of example-free classes, because the model must learn new data without old data. In addition, based on the correlation-based distillation loss, the present application adopts a correlation-based feature distillation loss. Instead of focusing only on the distillation method of the prediction results, the feature distillation method is adopted. By learning the feature processing method of the teacher model, the student model can better generalize to unseen data because it learns the internal representation of the data rather than just the output; under the guidance of the teacher model, the student model may avoid learning noise or unnecessary data features, thereby reducing the risk of overfitting; learning directly from the high-quality features of the teacher model can improve learning efficiency and make the student model training process more efficient, especially when resources are limited or processing complex and hierarchical features. Feature distillation does not ignore the rich information expressed by the internal features of the model, and can more comprehensively convey the knowledge in the teacher model, including the ability to extract and utilize features, which is particularly important for learning complex problems.

[0071] Cross entropy loss is used to measure the difference between the probability distribution predicted by the model and the probability distribution of the true label. It is used to learn the classification of newly added category data in incremental class learning. The goal of the new model is to be able to classify O old categories and N new categories. In example-free incremental class learning, the model can only access new category data sets. The cross entropy loss function takes the logarithm of the output probability of all correctly predicted classes for all new category data inputs and takes the negative value. In this way, the more accurate the model's prediction of new category data is, the smaller the loss function is, which can constrain the model to better learn to predict new category data.

[0072] The above-mentioned class-incremental image classification method based on prototype learning designs a prototype-based method and a correlation-based distillation loss method, which solves the problem of forgetting old knowledge due to the inability to save a large number of examples in the training model due to limited storage and the inability to access examples due to security and privacy considerations. Compared with mainstream class-incremental learning, it can better solve the problem of catastrophic forgetting of knowledge in the absence of examples, and at the same time has higher classification accuracy, better model stability and generalization performance. It is particularly suitable for image classification scenarios with example-free class-incremental learning. It is simple but extremely effective, and has broad application prospects.

[0073] Specifically: In order to solve the problem that old sample data cannot be accessed or stored during example-free incremental learning, the present application designs a prototype-based method. When performing deviation correction, the prototype of the old samples in the training process is saved, that is, the mean or center point obtained by the statistical analysis of the sample features of each category. Compared with saving examples, which requires saving multiple samples of each category of the original sample each time, the method of the present application only stores one prototype regardless of the number of samples, greatly reducing the pressure on storage. At the same time, the method of the present application stores statistical information of each category rather than specific samples, which can ensure the privacy and security of the data. In addition, after saving the prototype, Gaussian noise will be added to the prototype, thereby enhancing the diversity of the prototype and improving the stability and generalization ability of the model.

[0074] This application improves the distillation loss of traditional knowledge distillation, designs a distillation loss method based on correlation and a feature distillation loss method based on correlation, changes the strict point-to-point learning of the output probabilities of the teacher model (old model) and the student model (new model of incremental learning), focuses on the intra-class correlation and inter-class correlation of the prediction probabilities output by the new and old models, and constrains the correlation of the output results of the two (that is, constrains the correlation of the weight parameters learned by the teacher model and the student model for the old category data), simplifies the process of optimizing the distillation loss based on correlation during model training, so that the new model can classify more input data into the old category compared to the traditional knowledge distillation method, greatly improves the recognition accuracy, and effectively alleviates the catastrophic forgetting (CF) problem in sample-free class incremental learning (EFCIL).

[0075] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0076] In a specific embodiment, experiments are conducted on the CIAR-100 dataset (100 categories) with two incremental steps and 50 classes in each step. The experiment sets up three paradigm-free incremental learning methods, setting (a): Finetune only uses cross entropy loss, setting (b): Vanilla KD uses both cross entropy loss and distillation loss, setting (c): Correlation-based KD uses both cross entropy loss and correlation-based distillation loss (the method of the present application). The number of samples counted in the incremental learning process is n, which are divided into two types, samples classified into new categories (n, n) and samples classified into old categories (n, o). After incremental learning, the average accuracy acc of sample recognition after the first incremental learning step of the model is calculated. old , as shown in Table 1 below, the comparative experimental results are as follows.

[0077] Table 1: Comparative experimental results 1

[0078] As can be seen from Table 1, setting (a) basically classifies all samples into the new category, resulting in a serious bias in learning new samples, resulting in a recognition accuracy of 0 for samples in the old category, and a serious catastrophic forgetting problem. Setting (b) The model will take the old category into consideration to a certain extent, but the recognition of the new and old categories is still unfair, resulting in an improvement in the final recognition accuracy, but not much, indicating that the distillation loss has a certain effect in solving the catastrophic forgetting problem. Setting (c) The model classifies more samples into the old category, and the sample recognition accuracy is also greatly improved.

[0079] In another specific embodiment, in order to evaluate the effectiveness of the present application, experiments are conducted on two commonly used datasets CIFAR-100 and Tiny-ImageNet in the field of example-free incremental learning.

[0080] Selection and processing of data sets: CIFAR-100 consists of 100 32×32 pixel color images, including 50,000 images for training (500 images for each class) and 10,000 images for evaluation (100 images for each class). Tiny-ImageNet has 200 classes, each with 500 training images, 50 validation images, and 50 test images, which can be used to provide more stages and incremental classes to compare the sensitivity of various methods. In order to enhance the generalization ability of the model and reduce the feature bias of the model, the dataset labels are enhanced. The specific implementation scheme is: rotate the samples by 90 degrees, 180 degrees, and 270 degrees, respectively, as shown in Figure 4 The rotated processing effect is shown in the figure. After label enhancement, the model needs to classify four times the number of original categories, but during the training process, only the initial categories are concerned, so the expanded categories are not considered.

[0081] Implementation details: Implemented using pytorch and PYCIL on NVIDIA 3090, using ResNet-18 as the model architecture on CIFAR-100 and Tiny-ImageNet. In CIFAR-100, the model is trained with a batch size of 256 using SGD and momentum for optimization. In Tiny-ImageNet, the batch size is set to 64. According to the common settings in EFCIL, a random seed of 1993 is used to randomly shuffle the order of training categories. In order to control the scale of the loss value, the correlation-based FD loss is scaled to the same scale as the original KD loss. In addition, the hyperparameters λ, α, and β are all set to 10.

[0082] Evaluation indicators of scheme effectiveness: Two indicators are used to evaluate all methods. The overall accuracy performance is calculated by the average incremental accuracy To evaluate, ,in represents the test accuracy of all seen categories at incremental step i, The higher it is, the better the method is. Another evaluation metric is the accuracy of the final step , which measures the accuracy of the model after all steps of training are completed, reveals the performance gap between the class incremental learning (CIL) method and joint training.

[0083] In order to more objectively demonstrate the effect of the present invention, the method of the present application is compared with multiple mainstream methods, including regularization-based sample-free incremental learning methods: LwF, LwF-MC and EWC; sample-based CIL methods: EEIL, LUCIR and iCaRL-NCM; other state-of-the-art prototype-based EFCIL methods: PASS, IL2A, SSRE and MDPCR. The specific scheme is: calculate the average incremental accuracy A(%) of various CIL methods on CIFAR-100 and Tiny-ImageNet. Different incremental step S settings (5, 10 and 20 steps) were performed on CIFAR-100 and Tiny-ImageNet respectively. In CIFAR-100, for the first task, half of the categories (50 categories) were trained under the 5-step and 10-step settings, and 40 categories were trained under the 20-step setting. In Tiny-ImageNet, the model is trained on half of the categories (100 categories) in the first task, and trained on equal categories in the incremental step. The comparative experiment results are shown in Table 2 below.

[0084] Table 2: Comparative experimental results 2

[0085] As can be seen from Table 2, under different datasets and all incremental settings, the method of this application consistently outperforms other state-of-the-art methods significantly, which confirms that the method of this application can effectively solve the catastrophic forgetting (CF) problem in EFCIL without saving old samples. Taking the 5-step setting of CIFAR-100 as an example, the method of this application significantly outperforms the regularization-based method LwF-MC, the sample-based method LUCIR, and the state-of-the-art EFCIL method SSRE by 26.90%, 7.64%, and 2.89%, respectively.

[0086] like Figures 5 to 7 The following is the accuracy curve of different methods at different incremental steps (S=5 means that all categories of learning are completed in 5 steps, S=10 means that all categories of learning are completed in 10 steps, and S=20 means that all categories of learning are completed in 20 steps). Analyze the trends of different methods. The data with ↑ indicates the final step accuracy of the best method of this application and other methods. It can be seen that the method of this application is superior to other methods in almost all stages and achieves a better balance between stability and plasticity.

[0087] like Figures 8 to 10The heat map of PASS and SSRE and the method of the present application on CIFAR-100 and the corresponding step accuracy map are shown, where the category accuracy of the model in each incremental step is marked in the heat map, and the step accuracy is marked on the right. It can be seen that, first, the step accuracy of the method of the present application is always higher than that of PASS and SSRE; secondly, from the data in the small box, it can be seen that the performance gap between other methods and the method of the present application increases with the increase of incremental steps. For example, the performance gap between SSRE and the method of the present application increases from 1.53% to 3.49%. In addition, taking the category accuracy of step 2 as an example (data in the large box), the method of the present application shows better model plasticity than SSRE; in addition, after all steps are trained, the forgetting rates of the method of the present application and PASS in step 2 are 12.9% and 19.9% ​​respectively, which proves the excellent performance of the method of the present application in overcoming catastrophic forgetting (CF). This proves that the method of the present application has largely solved the dilemma between stability and plasticity.

[0088] like Figures 11 to 13 The confusion matrix diagrams of Finetune, PASS and the method of the present application at 20 incremental steps are shown, where the diagonal entries represent correct predictions and the non-diagonal entries represent misclassifications. It can be seen that Finetune obviously produces the worst results. Finetune always classifies samples into new categories, and exhibits catastrophic forgetting (CF) and significant prediction bias for the last step due to unrestricted training. PASS has fewer misclassified samples, but many samples of old categories are still misclassified into new categories. In contrast, the method of the present application misclassifies fewer samples of old categories and performs better in the classification of all categories. The results show that the method of the present application can eliminate most of the prediction biases and achieve better overall performance than its competitors.

[0089] In summary, this application accurately measures the linear correlation between two matrices, provides a loose optimization target, uses label enhancement to improve feature generalization, and proposes a correlation-based feature distillation method to eliminate feature bias. In addition, prototypes are preserved and enhanced to address classifier bias. Extensive experiments have shown that the method of this application is significantly better than regularization-based and prototype-based methods under various settings and indicators, and overcomes the problem of catastrophic forgetting of knowledge that has plagued the industry for a long time in solving incremental learning without examples, showing excellent performance.

[0090] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.

[0091] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A class-incremental image classification method based on prototype learning, characterized in that: include: Acquire image data and construct a first data set and a second data set; the first data set and the second data set respectively contain different sample pairs; Constructing an initial model, and using the first data set to train the initial model to obtain a teacher model; According to the teacher model, the sample features of the first data set are obtained; according to the sample features of the first data set, the prototype learning loss is obtained; according to the prototype learning loss, the total loss of the model is obtained; based on the total loss of the model, the teacher model is trained using the second data set to obtain the student model; The student model is used to classify class increment images.

2. The class-incremental image classification method based on prototype learning according to claim 1, characterized in that: Also includes: After obtaining the prototype learning loss, the Pearson coefficient is used to measure the advantage of correlation, and the inter-class relationship loss and intra-class relationship loss are calculated to obtain the distillation loss. According to the prototype learning loss, distillation loss and cross entropy loss, the total loss of the model is obtained.

3. The class-incremental image classification method based on prototype learning according to claim 2, characterized in that: The Pearson coefficient is used to measure the advantage of correlation, and the inter-class relationship loss and intra-class relationship loss are calculated to obtain the distillation loss, including: Calculate the Pearson coefficient and get the Pearson distance; According to the Pearson distance, calculate the inter-class relationship loss and the intra-class relationship loss; According to the inter-class relationship loss and the intra-class relationship loss, the distillation loss is obtained.

4. The class-incremental image classification method based on prototype learning according to any one of claims 1 to 3, characterized in that: According to the sample characteristics of the first data set, the prototype learning loss is obtained, including: According to the sample features of the first data set, the feature mean is calculated to obtain the original prototype set; Add noise to the original prototype set to obtain the extended prototype set; The extended prototype set is input into the classifier to obtain the prototype learning loss.

5. The class-incremental image classification method based on prototype learning according to claim 4, characterized in that: According to the sample features of the first data set, the feature mean is calculated to obtain the original prototype set: In the formula, For the original prototype set, For the images, The first category in the dataset The number of all images, For extraction The function corresponding to the feature, is an input image of a certain category, for different rotation angles.

6. The class-incremental image classification method based on prototype learning according to claim 5, characterized in that: Add noise to the original prototype set to get the extended prototype set: In the formula, To extend the prototype set, ~N(0,1) is Gaussian noise distribution, is the average variance of the class features.

7. The class-incremental image classification method based on prototype learning according to claim 6, characterized in that: Input the extended prototype set into the classifier to obtain the prototype learning loss, including: In the formula, is the prototype learning loss, For the categories, is the number of old categories, For and Do cross entropy loss, For After the classification result of the classifier, is the correct category result in the label.

8. The class-incremental image classification method based on prototype learning according to claim 3, characterized in that: Calculate the Pearson coefficient and get the Pearson distance, including: In the formula, for and The Pearson coefficient, is the predicted value of the teacher model for the image, is the predicted value of the student model for the image, is the covariance, is the variance, For the categories, is the number of old categories, Predicting images as categories for the teacher model The probability of for The mean of Predicting images as categories for the student model The probability of for The mean of for and The Pearson distance.

9. The class-incremental image classification method based on prototype learning according to claim 8, characterized in that: According to the Pearson distance, the inter-class relationship loss and the intra-class relationship loss are calculated, including: In the formula, is the inter-class relationship loss, is the intra-class relationship loss, is the total number of samples, is the number of old categories, is the feature matrix f No. OK, is the feature matrix List, for and The Pearson distance, is the teacher model for the feature matrix The prediction results of the line, The student model is the feature matrix The prediction results of the line, for and The Pearson distance, is the teacher model for the feature matrix The prediction results of the column, The student model is the feature matrix The predicted results of the column.

10. The class-incremental image classification method based on prototype learning according to claim 9, characterized in that: According to the inter-class relationship loss and intra-class relationship loss, the distillation loss is obtained, including: In the formula, is the distillation loss, is the weight of the inter-class relationship loss, is the inter-class relationship loss, is the weight of intra-class relationship loss, is the intra-class relationship loss.

Images