Image classification method for small sample class incremental learning based on prototype features

Abstract

The invention discloses an image classification method for small sample class incremental learning based on prototype features. The method comprises the following steps of: 1, training an end-to-end network based on a ResNet framework by using image data # imgabs0 # of a base class task; 2, for the end-to-end network, discarding a fully connected classifier of the end-to-end network, adopting a distance-based nearest neighbor classifier, and freezing parameters of a feature extractor; 3, inputting the image features into a prototype correction module, and optimizing the prototype correction module through a cross entropy loss function and a prototype comparison loss function; 4, continuously training a prototype correction module by using the image data, and generating a new category of prototype features with higher discrimination by the prototype correction module; and 5, classifying the distance between the image features and the prototype features by using a nearest neighbor classifier. According to the method, the distribution of the classes in the feature space is accurately described, so that the image classification precision in a small sample class incremental learning scene is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image classification, and particularly relates to an image classification method for few-shot class-incremental learning based on prototype features. Background Technique

[0002] With the rapid development of deep learning, models have achieved significant performance improvements in many tasks. However, traditional deep learning methods usually rely on large-scale labeled data for training, and in practical applications, obtaining a large amount of labeled data is often expensive and time-consuming. Especially in a dynamically changing environment, the model needs to continuously learn new categories, and the number of samples of new categories is usually very limited. In this context, few-shot class-incremental learning (FSCIL) has emerged. It requires the model to be able to effectively learn new categories while using only a small number of samples, and at the same time avoid catastrophic forgetting of the already learned categories.

[0003] In the existing FSCIL methods, there are still some challenges to be solved:

[0004] Challenge 1: The deviation of prototype features caused by noise. In FSCIL, prototype features are widely used for class representation and classification. Traditional prototype features are usually obtained by calculating the average value of all sample features within a class. This method is simple and intuitive, but in the few-shot scenario, due to the limited number of samples, the average prototype feature is easily affected by noise and sample bias, resulting in a decrease in the accuracy of class representation.

[0005] Challenge 2: The problem of catastrophic forgetting. When the model learns new categories, it is easy to forget the categories that have already been learned. In incremental learning, the model needs to effectively retain the knowledge of previous categories while learning the features of new categories. How to balance the learning of new categories and the knowledge of old categories is a key issue.

[0006] Challenge 3: The overfitting problem. Due to the scarcity of new category samples, the model is often prone to overfitting to new categories during the learning process, resulting in a decline in generalization ability and difficulty in adapting to new tasks.

[0007] For example, in the "Few-shot Image Class Incremental Learning Method and System Based on Data Augmentation" with the publication number CN117975203A, the invention first uses the first type of dataset to train a training image classification model to obtain a base image classification network model. Then, the next sub-dataset is used to train the class-incremental learning network model after the previous round of training. Repeat the execution until the model converges to obtain the classification label of the image to be classified.

[0008] This invention uses data augmentation strategies to improve the performance of the model in classification tasks, but does not focus on solving the catastrophic forgetting problem in FSCIL, resulting in poor performance in incremental tasks. Summary of the Invention

[0009] To overcome the deficiencies of the above-mentioned existing technologies, the purpose of this invention is to provide an image classification method for few-shot class incremental learning based on prototype features. The prototype correction module of this method can generate more representative prototype features, accurately depicting the distribution of categories in the feature space, thereby significantly improving the image classification accuracy in the few-shot class incremental learning scenario.

[0010] To achieve the above purpose, the technical solution adopted by this invention is:

[0011] An image classification method for few-shot class incremental learning based on prototype features, comprising the following steps;

[0012] Step 1: Use the image data of the base class task (Task 0) Train an end-to-end network based on the ResNet architecture. The end-to-end network consists of a feature extractor and a fully connected classifier. The feature extractor is used to extract high-dimensional feature representations from input images, and the fully connected classifier is used to perform class prediction based on the high-dimensional feature representations;

[0013] Step 2: Discard the fully connected classifier of the end-to-end network, adopt a distance-based nearest neighbor classifier, and freeze the parameters of the feature extractor; in subsequent incremental tasks, its parameters remain fixed to prevent the model from forgetting the knowledge of old tasks;

[0014] Step 3: Use the feature extractor with frozen parameters to extract the image features of the image data ; Input the image features into the prototype correction module, train the prototype correction module based on the episodic paradigm in the meta-learning stage, and optimize the prototype correction module through the cross-entropy loss function and the prototype contrast loss function;

[0015] Step 4: After step 3 is trained on the base class task, continue to train the prototype correction module in incremental tasks. In incremental task i in the incremental learning stage, use the image data Continue to train the prototype correction module, and additionally use the inter-class separability loss function and the knowledge distillation loss function to optimize the prototype correction module to prevent catastrophic forgetting. After the above training and optimization, the prototype correction module can generate more discriminative prototype features, enabling the prototype features of new categories to accurately represent the data distribution of the current task and maintaining sufficient distinguishability from the prototype features of historical categories, thereby achieving higher classification accuracy and stronger robustness in the few-shot class incremental learning scenario;

[0016] Step 5: Use the nearest neighbor classifier to classify based on the distance between the image features and the prototype features. Whichever prototype feature has the closest distance is the category it belongs to.

[0017] In the said Step 1, the image dataset is divided into the image data of a basic task and the image data of multiple incremental tasks to simulate the setting of few-shot class incremental learning; first, select a part of the categories and their samples from the dataset as the basic task for the initial learning stage of the model;

[0018] After the basic task, the remaining categories in the dataset are actively divided into multiple small parts, with each part being an incremental task. Each incremental task only contains samples of a small number of new categories, simulating the scenario where the model needs to gradually learn new categories in actual applications.

[0019] In the said Step 2, given the query sample x i , the formula of the distance-based nearest neighbor classifier is as follows:

[0020]

[0021] where f i is the high-dimensional feature obtained by the sample x i after passing through the feature extractor , P j represents the prototype feature of category j. By calculating the distance between the query sample and the prototype feature of each category, the query sample is assigned to the prototype category with the closest distance. The calculation method of the prototype feature is as follows:

[0022]

[0023] where K represents the number of samples in a class support set, represents the feature extractor, S i represents the data set of class i. The distance-based nearest neighbor classifier determines the category of the sample by calculating the distance between the query sample and the prototype feature, that is, the query sample belongs to the category of the prototype feature with the closest distance. Compared with the traditional classifier, the distance-based nearest neighbor classifier does not need to retrain the entire model when facing new categories, so it has high adaptability and flexibility.

[0024] The process of the prototype correction module in the said Step 3 during training on the base class task is as follows:

[0025] Step 3.1: Construct a support set S and a query set Q base in the same form as the incremental task data from the base class data base ;

[0026] Step 3.2: Calculate the support set S base The prototype features of each class in i . Taking class i as an example, the calculation steps of the prototype feature P i are described. The linear transformation of the initial prototype feature P i ′ obtained by the average method is used as the query Q, and the initial prototype feature and the support set image features are concatenated to obtain the feature h

[0027]

[0028] Next, the linear transformation of h i is used as the key K and the value V. Finally, the residual connection is adopted to obtain the final prototype feature P i of class i:

[0029] P i = P i ′ + Attention(Q, K, V)

[0030]

[0031] where Q = W Q P i , K = W K h i , V = W V h i , W Q , W K , W V are three learnable weight matrices. Attention(·) represents the self-attention mechanism. P i represents the prototype feature of class i throughout the text.

[0032] Step 3.3: Optimize the prototype correction module for the prototype feature P i obtained in Step 3.2 through the cross-entropy loss function. Given the query sample (x i , y i ), its classification loss is calculated through the cross-entropy loss function, and the formula is as follows:

[0033]

[0034] where f i is the feature of the sample x i , that is, P j represents the prototype feature of class j, |C k | represents the number of all classes in the current task k. I(y i = j) is the indicator function, which is 1 when y i = j and 0 otherwise.

[0035] Step 3.4: Calculate the support set S base The sub-prototype features of each class in i . Take class i as an example to illustrate the calculation steps of the sub-prototype feature SP . In an incremental task with the data form of N-way K-shot, each time exclude 1 sample (i.e.,

[0036] SP i = SP i,1 , SP i,2 , …, SP i,K i = 1, 2, …, N

[0037] N represents the number of samples of each class in the incremental task;

[0038] Step 3.5: After calculating all the sub-prototype features in Step 3.4, regard the sub-prototype features from the same class as positive sample pairs, and the sub-prototype features from different classes as negative sample pairs. Given a sub-prototype feature SP i,a (belonging to class i), the prototype contrast loss is as follows:

[0039]

[0040] where SP = SP 1 ∪SP 2 ∪…∪SP N represents all the sub-prototype features in an N-way K-shot incremental task.

[0041] Step 3.6: After obtaining the cross-entropy loss function in Step 3.3 and the prototype contrast loss in Step 3.5 , the total loss function of the prototype correction module in the basic task stage is as follows, λ ce and λ con represent hyperparameters:

[0042]

[0043] Use to perform stochastic gradient descent on the prototype correction module to update its parameters.

[0044] Step 3.7: Repeat Steps 3.1 - 3.6 until the prototype correction module converges.

[0045] The process of training the prototype correction module in step 4 on incremental tasks is as follows:

[0046] Step 4.1: Calculate the prototype features of each class in the base task for calculating the relevant losses in subsequent steps 4.3 and 4.4. Since the number of samples in each class of the base task is sufficient, the prototype features of each class in the base task are still represented by the average method;

[0047] Step 4.2: Calculate the cross-entropy loss in the incremental task through steps 3.2 - 3.5 and the prototype contrast loss

[0048] Step 4.3: As the incremental tasks increase continuously, the prototype features of the new tasks may overlap with the class prototype features of the previous tasks in the feature space, making it difficult to distinguish between classes. Use the prototype features of the old tasks obtained in step 4.1. To suppress the problem of prototype feature overlap, an inter-class separability loss function is introduced in the incremental stage This loss function encourages the class prototypes of the current task to be as far away as possible from the class prototypes of the previous tasks, thereby enhancing the inter-class separability.

[0049] The loss function in step 4.3 is defined as follows:

[0050]

[0051] where, ΔC k represents the class of the current task k, |ΔC k | represents the number of all class labels of task k, and C k-1 represents all the classes up to task k - 1;

[0052] Step 4.4: To alleviate the problem of forgetting old knowledge by the prototype correction module during the fine-tuning process and preserve the knowledge of the old tasks, the idea of knowledge distillation is introduced. The model of the previous task is regarded as the teacher model, and the model of the new task is regarded as the student model. The knowledge distillation loss is calculated as follows:

[0053]

[0054] where, |C k-1 | represents the total number of classes in task k - 1, and τ is the temperature parameter. and represent the modified logits (i.e., unnormalized scores) of the student and teacher models respectively;

[0055] Step 4.5: Through steps 4.2 - 4.4, the loss function for prototype correction in the incremental stage is as follows, λ ce, λ con , λ S and λ KD represent hyperparameters.

[0056]

[0057] Use to perform stochastic gradient descent on the prototype correction module to update its parameters;

[0058] Step 4.6: Repeat steps 4.2 - 4.5 until the prototype correction module converges on the current incremental task;

[0059] Step 4.7: Save the prototype features obtained by the current incremental task through the prototype correction module for the calculation of the inter - class discriminative loss function in subsequent tasks and as the classification basis for the distance - based nearest - neighbor classifier;

[0060] Step 4.8: Proceed to the next incremental task and repeat steps 4.2 - 4.7 until all incremental tasks are completed. By gradually completing the training of all incremental tasks, the model achieves efficient new - class learning and stable historical - class retention in the few - shot class incremental learning scenario, and finally reaches a comprehensively optimized classification performance.

[0061] An image classification system for few - shot class incremental learning based on prototype features, including a feature extractor and a prototype correction module: The feature extractor maps the original data to the feature space to generate high - dimensional feature vectors;

[0062] The prototype correction module is responsible for generating and optimizing class prototypes in the feature space to ensure inter - class discrimination and intra - class consistency, thus supporting efficient few - shot class incremental learning.

[0063] After the feature extractor is trained on the base - class task, its parameters remain fixed and provide a general feature representation ability in subsequent incremental tasks;

[0064] The prototype correction module enhances the prototype features of each class during the incremental process through contrastive learning and self - attention mechanism, eliminating the problem that the averaging - based calculation method is vulnerable to noise interference.

[0065] An image classification device for few - shot class incremental learning based on prototype features, including:

[0066] A memory: used to store a computer program for implementing the image classification method for few - shot class incremental learning based on prototype features;

[0067] A processor: used to implement the image classification method for few - shot class incremental learning based on prototype features when executing the computer program.

[0068] A computer-readable storage medium, comprising:

[0069] The computer-readable storage medium stores a computer program, which when executed by a processor can implement an image classification method for few-shot class incremental learning based on prototype features.

[0070] Advantages of the present invention:

[0071] The prototype contrast loss function, inter-class separability loss function, and knowledge distillation loss function used in the prototype correction module of the present invention effectively improve the performance of the few-shot class incremental learning model. The prototype contrast loss function encourages the prototype features of the same-class samples to be as close as possible in the feature space, while the prototype features of different-class samples are as far away as possible, effectively enhancing the representation ability of the prototype features. The inter-class separation loss function enables the model to better maintain the boundaries between classes, thereby reducing the interference between old and new classes during the incremental learning process. The knowledge distillation loss function constrains the new model to maintain the learning ability of the old-class features during training, thereby ensuring that the model does not lose the discriminative ability for old classes during the incremental learning process. The combination of the three loss functions effectively improves the performance of the few-shot class incremental learning model and effectively alleviates the forgetting problem. Description of the Drawings

[0072] Figure 1 It is the task scenario targeted by the present invention, that is, the setting of few-shot class incremental learning.

[0073] Figure 2 It is the overall framework of the present invention.

[0074] Figure 3 It is the design architecture of the prototype correction module adopted in the present invention.

[0075] Figure 4 It is for the present invention on the miniImageNet dataset, for and A line graph of ablation experiments on three loss functions. Detailed Embodiments

[0076] The present invention will be further described in detail below with reference to the drawings.

[0077] Referring to Figure 1 , the purpose of few-shot class incremental learning is to learn new knowledge from a small number of samples without forgetting the knowledge that has already been learned.

[0078] Specifically, in a task t, represents the training set of this task. represents the training sets of all tasks, and n represents the number of all tasks. It should be noted in particular that when Indicates the base task. The corresponding class label is C t , and there is no overlap in the class labels between different tasks, that is and the model needs to have the function of "one-shot learning". Few-shot class incremental learning includes several tasks that arrive in batches. Once the model enters the next task, the training data of all previous tasks can no longer be used. At task t, only can be used for model training, and the data of all previous tasks cannot be used, but the corresponding test set comes from the class labels. So the model cannot forget the knowledge learned before after learning on a new task.

[0079] In the setting of few-shot class incremental learning, it is divided into a base task (also called the base session) and multiple incremental tasks (also called incremental sessions). In the base task, a large number of base class samples are provided for the initial learning of the model. In the incremental tasks, the sample size is extremely small, meeting the definition of few-shot learning. Usually, there are only 1 to 5 samples per class. The data form of the incremental task is N-way K-shot, that is, there are N classes in an incremental task, and each class has only K samples.

[0080] Usually, a dataset is divided into a base task and multiple incremental tasks to simulate the setting of few-shot class incremental learning. For example, in the general dataset miniImageNet, there are a total of 100 classes, among which 60 classes serve as the base task, and the remaining 40 classes are divided into 8 incremental tasks. The data form of each incremental task is 5-way 5-shot. That is, there are only 5 classes in each incremental task, and each class has only 5 samples. And each class in the base class task has 500 samples available for training. Therefore, generally, the base class data is used to train a feature extractor with stronger representation ability.

[0081] See Figure 2 , an image classification method for few-shot class incremental learning based on prototype features, mainly including three stages: the pre-training stage, the meta-learning stage, and the incremental learning stage. In the pre-training stage, the feature extractor is trained on the base class dataset with a large amount of data to learn the general feature representation ability, laying a foundation for subsequent incremental tasks and improving the generalization ability of the model.

[0082] In the meta-learning stage, by simulating few-shot learning tasks, the prototype correction module is trained to quickly adapt to new classes. Enable the model to have the ability to quickly adapt to new classes and be able to generate discriminative prototype features with a small number of samples.

[0083] The incremental learning stage is the process by which the model continuously learns new classes during actual application. In this stage, while retaining knowledge of old classes, the model needs to learn new knowledge using a small number of new class samples. This enables the model to continuously learn new classes without retraining and adapt to dynamically changing task requirements.

[0084] The following Steps 1-2 introduce the specific process of the pre-training stage, Step 3 introduces the specific process of the meta-learning stage, and Step 4 introduces the specific process of the incremental learning stage.

[0085] Step 1: In the pre-training stage, use the base class dataset to train an end-to-end network based on the ResNet architecture. This invention conducts experiments on three general datasets: miniImageNet, CUB200, and CIFAR100. On the CIFAR100 dataset, the feature extractor uses the ResNet20 architecture; on the miniImageNet and CUB200 datasets, the ResNet18 architecture is used. For the CUB200 dataset, the number of training epochs is set to 400, and the batch size is set to 128; for the CUB200 and miniImageNet datasets, the number of training epochs is set to 200, and the batch size is set to 256. The training of all datasets uses the Stochastic Gradient Descent optimizer with momentum, where the initial learning rate for the CUB200 dataset is set to 0.05; while the initial learning rate for the miniImageNet and CUB200 datasets is set to 0.1. The momentum for all three is set to 0.9, and the weight decay coefficient is set to 5e-4. In addition, the cosine annealing strategy is adopted to adjust the learning rate.

[0086] Step 2: For the end-to-end network obtained in Step 1, discard its fully connected classifier and adopt a distance-based nearest neighbor classifier. And freeze the parameters of the feature extractor, and keep its parameters fixed in subsequent incremental tasks to prevent the model from forgetting the knowledge of old tasks;

[0087] Given the query sample x i , the formula for the distance-based nearest neighbor classifier is as follows:

[0088]

[0089] where f i is the high-dimensional feature obtained by the sample x i after passing through the feature extractor , P j represents the prototype feature of class j. By calculating the distance between the query sample and the prototype feature of each class, the query sample is assigned to the prototype class with the closest distance. The calculation method of the prototype feature is as follows:

[0090]

[0091] Among them, K represents the number of samples in a class support set, represents the feature extractor, S i represents the data set of class i. The distance-based nearest neighbor classifier determines the class of a sample by calculating the distance between the query sample and the prototype feature. Compared with traditional classifiers, the distance-based nearest neighbor classifier does not need to retrain the entire model when facing new classes, so it has high adaptability and flexibility.

[0092] Step 3: In the meta-learning stage, use the feature extractor obtained in Step 2 to calculate the image features of the base class image data and train the prototype correction module based on the scenario paradigm, and optimize the prototype correction module through the cross-entropy loss function and the prototype contrast loss function; Figure 3 Introduced the architecture of the prototype correction module for each scenario;

[0093] Step 3.1: Construct a support set S in the same form as the incremental task data from the base class data base and a query set Q base . For example, for the CIFAR100 dataset, the support set S base has a dataset form of 5-way 5-shot, and the query set Q base has a data form of 5-way 20-shot.

[0094] Step 3.2: Calculate the prototype features of each class in the support set S base obtained in Step 3.1. Here, take class i as an example to illustrate the calculation steps of the prototype feature P i . Take the linear transformation of the initial prototype feature P i ′ obtained by the average method as the query Q, and splice the initial prototype feature with the support set image features to obtain the feature h i :

[0095]

[0096] Next, take the linear transformation of h i as the key K and the value V. Finally, use the residual connection to obtain the final prototype feature P i of class i:

[0097] P i = P i ′ + Attention(Q, K, V)

[0098]

[0099] where Q = W QP i , K = W K h i , V = W V h i , W Q , W K , W V are three learnable weight matrices. Attention(·) represents the self-attention mechanism.

[0100] Step 3.3: Optimize the prototype correction module for the prototype features obtained in Step 3.2 through the cross-entropy loss function. Given a query sample (x i , y i ), its classification loss is calculated through the cross-entropy loss function, and the formula is as follows:

[0101]

[0102] where f i is the feature of sample x i , that is P j represents the prototype feature of class j, |C k | represents the number of all classes in the current task k. I(y i = j) is an indicator function, which is 1 when y i = j, and 0 otherwise.

[0103] Step 3.4: Calculate the sub-prototype features of each class in the support set S base . Here, taking class i as an example, the calculation steps of the sub-prototype feature SP i are described. In the incremental task of CIFAR100, each time 1 sample is sequentially excluded from 5 support set samples of class i (that is combination ways), forming 5 subsets each containing 4 samples. For each sample subset, through the calculation in Step 3.2, the corresponding sub-prototype features are obtained. Finally, class i will obtain K sub-prototype features, denoted as:

[0104] SP i = SP i,1 , SP i,2 , …, SP i,K i = 1, 2, …, 5

[0105] Step 3.5: After calculating all the sub-prototype features in Step 3.4, regard the sub-prototype features from the same class as positive sample pairs, and the sub-prototype features from different classes as negative sample pairs. Given a sub-prototype feature SP i,a (belonging to class i), the prototype contrast loss is as follows:

[0106]

[0107] where SP = SP 1 ∪ SP 2 ∪ … ∪ SP N represents all sub - prototype features in the incremental task.

[0108] Step 3.6: The cross - entropy loss function obtained in Step 3.3 and the prototype contrast loss obtained in Step 3.5 After that, the total loss function of the prototype feature optimization module in the base task stage is as follows, λ ce and λ con represent hyperparameters:

[0109]

[0110] Use to perform stochastic gradient descent on the prototype correction module to update its parameters. Here, both λ ce and λ con are set to 1.

[0111] Step 3.7: Steps 3.1 - 3.6 count as one iteration, and repeat the iteration 50000 times. Use the stochastic gradient descent optimizer to train the prototype correction module, and set the initial learning rate to 1e - 3. To further adjust the learning rate, use step decay during training, and apply the decay factor 0.8 after every 1000 iterations.

[0112] Step 4: After Step 3 is completed in the base - class task, continue to train the prototype correction module in the incremental task. In the incremental task i of the incremental learning stage, use the data to continue training the prototype correction module, and additionally use the inter - class separability loss function and the knowledge distillation loss function to optimize the prototype correction module to prevent catastrophic forgetting;

[0113] Step 4.1: Calculate the prototype features of each class in the base task for the calculation of relevant losses in subsequent Steps 4.3 and 4.4. In the CIFAR100 dataset, there are 60 classes in the base task, and a total of 60 classes' prototype features need to be calculated. Since the number of samples in each class of the base task is sufficient, the prototype features of each class in the base task are still represented in an average manner.

[0114] Step 4.2: Calculate the cross - entropy loss and the prototype contrast loss

[0115] Step 4.3: As the incremental tasks keep increasing, the prototype features of new tasks may overlap with the category prototype features of previous tasks in the feature space, making it difficult to distinguish between categories. Using the prototype features of the old tasks obtained in Step 4.1, in order to suppress the problem of prototype feature overlap, an inter-class separability loss function is introduced in the incremental stage. This loss function encourages the category prototypes of the current task to be as far away as possible from the category prototypes of the previous tasks, thereby enhancing the inter-class separability. Specifically, this loss function is defined as follows:

[0116]

[0117] where ΔC k represents the category of the current task k, |ΔC k | represents the number of all category labels of task k, and C k-1 represents all the categories up to task k - 1.

[0118] Step 4.4: In order to alleviate the problem of forgetting old knowledge by the prototype correction module during the fine-tuning process and preserve the knowledge of old tasks, the idea of knowledge distillation is introduced in the present invention. The model of the previous task is regarded as the teacher model, and the model of the new task is regarded as the student model. The knowledge distillation loss is calculated as follows:

[0119]

[0120] where |C k-1 | represents the total number of categories in task k - 1, and τ is the temperature parameter. and respectively represent the modified logits (i.e., unnormalized scores) of the student and teacher models.

[0121] Step 4.5: Through the above Steps 4.2 - 4.4, the loss function for prototype correction in the incremental stage λ ce 、λ con 、λ S and λ KD represent hyperparameters.

[0122]

[0123] Here, λ ce 、λ con 、λ S and λ KD are respectively set to 1, 1, 0.2, and 0.5. Use to perform stochastic gradient descent on the prototype correction module to update its parameters.

[0124] Step 4.6: Steps 4.2 - 4.5 constitute one iteration in the incremental task. A total of 50 iterations are performed, and the learning rate is set to 1e-4.

[0125] Step 4.7: Save the prototype features obtained by the current incremental task through the prototype correction module, which are used for the calculation of the inter-class discriminative loss function in subsequent tasks and as the classification basis for the distance-based nearest neighbor classifier.

[0126] Step 4.8: Proceed to the next incremental task and repeat Steps 4.2 - 4.7 until all incremental tasks are completed for training. For example, in the CIFAR100 dataset, there are 8 incremental tasks, and Steps 4.2 - 4.7 need to be repeated 8 times.

[0127] To evaluate the performance of the proposed few-shot class incremental learning method based on prototype features, the method of this application (Ours) is compared with various methods. The comparison methods adopted can be divided into three categories: The first category is the baseline method FineTune, which directly fine-tunes the model using incremental task data without adopting any additional anti-forgetting measures and serves as a reference benchmark for the non-anti-forgetting strategy; the second category is traditional incremental learning methods, including iCaRL and EEIL. Among them, iCaRL effectively alleviates the catastrophic forgetting problem through a strategy combining sample replay and knowledge distillation. And EEIL further introduces a prototype-based incremental learning method on the basis of iCaRL, significantly improving the model performance; the third category is FSCIL-related methods, including TOPIC, CEC, MetaFSCIL, FeSSSS, LIMIT, FACT, TEEN. The experiment reports the accuracy on each task, and the experimental results on the miniImageNet, CUB200, and CIFAR100 datasets are shown in Table 1, Table 2, and Table 3 respectively.

[0128] As can be seen from Table 1, on the miniImageNet dataset, the accuracy of the present invention in the initial task is 73.64%, which is not much different from the TEEN method (73.53%), the FACT method (72.56%), and other few-shot class incremental learning methods. This result indicates that all the compared few-shot class incremental learning methods are based on the same backbone network, and the performance advantage of the method of the present invention stems from the design of its anti-forgetting mechanism rather than relying on a stronger feature extractor. In the subsequent 8 incremental tasks, a relatively high accuracy is always maintained, significantly superior to other methods, and no significant forgetting problem occurs. Especially in the last task, a relatively high accuracy level is still maintained.

[0129] Table 1: Comparison results of accuracy (%) on each task of the miniImgeNet dataset

[0130]

[0131] Table 2 shows the performance of the method of this application on the CUB200 dataset. From the experimental results, it can be directly seen that the experimental effect is better than other methods.

[0132] Table 2: Comparison results of accuracies (%) on various tasks of the CUB200 dataset

[0133]

[0134] Table 3 shows the experiments of the method of this application on the CIFAR100 dataset. By synthesizing the three datasets of miniImageNet, CUB200, and CIFAR100, the superiority of the method of this application can be seen.

[0135] Table 3: Comparison results of accuracies (%) on various tasks of the CIFAR100 dataset

[0136]

[0137] See Figure 4 , in order to explore the influence of different loss functions on the final performance, this application conducted experiments on the miniImageNet dataset. As the incremental tasks progress, all loss functions combined show the best classification performance. In addition, deleting any one loss function will lead to a decrease in accuracy to varying degrees.

[0138] In the present invention, in order to overcome the deviation of the prototype features caused by noise, a prototype feature correction module introducing contrastive learning and self-attention mechanism is introduced to replace the traditional average prototype feature calculation method. This module enhances the similarity between samples of the same category through contrastive learning, widens the distance between samples of different categories, and at the same time uses the self-attention mechanism to capture the global dependence relationship between samples, thereby generating more robust and accurate prototype features.

[0139] In order to avoid catastrophic forgetting, first, the parameters of the feature extractor are frozen in the incremental stage to ensure that all incremental tasks and base-class tasks share the same feature extractor parameters, thereby retaining the knowledge of previous categories. In addition, the old prototype features of each generated class are replayed to further enhance the model's memory ability for old categories.

[0140] In order to alleviate the overfitting problem, meta-learning is adopted for training. Under the meta-learning framework, the model can quickly adapt through the training of multiple tasks and avoid over-reliance on limited new category samples. In this way, the model can maintain good generalization ability when learning new categories, thereby effectively alleviating the overfitting phenomenon.

[0141] A small-sample class incremental learning image classification device based on prototype features, comprising: a memory for storing a computer program for implementing the image classification method of small-sample class incremental learning based on prototype features; a processor for implementing the image classification method of small-sample class incremental learning based on prototype features when executing the computer program.

[0142] A computer-readable storage medium, comprising: the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement an image classification method of small-sample class incremental learning based on prototype features.

Claims

1. An image classification method based on small sample class incremental learning of prototype features, characterized in that: The steps include: Step 1: Use the image data of the base task Training an end-to-end network based on a ResNet architecture, the end-to-end network comprising a feature extractor and a fully connected classifier, wherein the feature extractor is used to extract a high-dimensional feature representation from an input image, and the fully connected classifier is used to perform category prediction based on the high-dimensional feature representation; Step 2: discard the fully connected classifier of the end-to-end network, adopt a distance-based nearest neighbor classifier, and freeze the parameters of the feature extractor; Step 3: Extract image data using the feature extractor with frozen parameters The image features are input into the prototype correction module, and the prototype correction module is optimized by the cross entropy loss function and the prototype contrast loss function; Step 4: In the incremental learning phase, in incremental task i, use the image data Continue to train the prototype correction module, and additionally use the inter-class separable loss function and the knowledge distillation loss function to optimize the prototype correction module. The prototype correction module generates more discriminative prototype features of new categories. Step 5: Use the nearest neighbor classifier to classify the distance between image features and prototype features.

2. The image classification method based on small sample class incremental learning of prototype features according to claim 1, characterized in that: In step 1, the image dataset is divided into image data of a base class task and image data of multiple incremental tasks to simulate the setting of incremental learning of small sample classes; First, a part of categories and their samples are selected from the data set as base class tasks for the initial learning stage of the model; After the base class task, the remaining categories in the dataset are actively divided into multiple small parts, each of which serves as an incremental task. Each incremental task contains only a small number of samples of the new category, simulating the scenario in actual applications where the model needs to gradually learn new categories.

3. The image classification method based on small sample class incremental learning of prototype features according to claim 1, characterized in that: In step 2, given a query sample x i , the distance-based nearest neighbor classifier formula is as follows: Among them, f i is the sample x i After feature extractor The obtained high-dimensional features, P j Represents the prototype feature of category j. By calculating the distance between the query sample and the prototype feature of each category, the query sample is assigned to the prototype category with the closest distance. The prototype feature calculation method is as follows: Where K represents the number of samples in a class support set. represents the feature extractor, S i Represents a data set of class i. The distance-based nearest neighbor classifier determines the category of the sample by calculating the distance between the query sample and the prototype feature. That is, the query sample belongs to the category of the prototype feature to which the query sample is closest.

4. The image classification method based on small sample class incremental learning of prototype features according to claim 1, characterized in that: The process of training the prototype correction module in step 3 on the base class task is as follows: Step 3.1: Image data from the base task Construct a support set S in the same format as the incremental task data base and the query set Q base ; Step 3.2: Calculate the support set S base The prototype features of each class in the , using the average initial prototype features P i ′ is used as the query Q, and the initial prototype feature is concatenated with the support set image feature to obtain the feature h i : Next, h i The linear transformation of is used as the key K and value V, and finally the residual connection is used to obtain the final prototype feature P of class i i : P i =P i ′+Attention(Q,K,V) Where Q = W Q P i , K=W K h i , V=W V h i , W Q , W K , W V are three learnable weight matrices, Attention(·) represents the self-attention mechanism; Step 3.3: For the prototype feature P obtained in step 3.2 i The prototype correction module is optimized through the cross entropy loss function. Given a query sample (x i ,y i ), whose classification loss is calculated by the cross entropy loss function; Step 3.4: Calculate the support set S base Each sub-class prototype feature SP i Denoted as: SP i =SP i,1 ,SP i,2 ,…,SP i,K i=1,2,…,N N represents the number of samples of each class in the incremental task; Step 3.5: Consider the sub-prototype features from the same class as positive sample pairs, and the sub-prototype features from different classes as negative sample pairs. Given a sub-prototype feature SP i,a , the prototype contrast loss is as follows: Among them, SP=SP1∪SP2∪…∪SP N Represents all sub-prototype features in an N-way K-shot incremental task; Step 3.6: According to the cross entropy loss function The prototype contrast loss The total loss function of the prototype feature optimization module in the basic task stage As shown below, λ ce and λ con Represents hyperparameters: use Perform stochastic gradient descent on the prototype correction module to update its parameters; Step 3.7: Repeat steps 3.1-3.6 until the prototype correction module converges.

5. The image classification method based on small sample class incremental learning of prototype features according to claim 4 is characterized in that: The process of training the prototype correction module in step 4 on the incremental task is as follows: Step 4.1: Calculate the prototype features of each class in the base class task for the calculation of the related losses in subsequent steps 4.3 and 4.4; Step 4.2: Calculate the cross entropy loss in the incremental task through steps 3.2-3.5 Compared with the prototype loss Step 4.3: Use step 4.1 to get the prototype features of the old task and introduce the inter-class separable loss function in the incremental stage Step 4.4: Consider the model of the previous task as the teacher model, the model of the new task as the student model, and the knowledge distillation loss The calculation formula is as follows: Among them, |C k-1 | represents the total number of categories in task k-1, τ is the temperature parameter, and Represent the modified logits of the student and teacher models respectively; Step 4.5: Through steps 4.2-4.4, the loss function of the prototype correction in the incremental stage is obtained As shown below, λ ce , con , S and λ KD represents a hyperparameter; use Perform stochastic gradient descent on the prototype correction module to update its parameters; Step 4.6: Repeat steps 4.2-4.5 until the prototype correction module converges on the current incremental task; Step 4.7: Save the prototype features obtained by the prototype correction module of the current incremental task, which are used for the calculation of the inter-class classifiable loss function in subsequent tasks, as well as the classification basis of the distance-based nearest neighbor classifier; Step 4.8: Proceed to the next incremental task and repeat steps 4.2-4.7 until all incremental tasks are trained. By gradually completing the training of all incremental tasks, the model achieves efficient new category learning and stable historical category retention in the small sample class incremental learning scenario, and finally achieves fully optimized classification performance.

6. The image classification method based on small sample class incremental learning of prototype features according to claim 5, characterized in that: The loss function in step 4.3 is defined as follows: Where, ΔC k represents the category of the current task k, |ΔC k | represents the number of category labels for task k, C k-1 Represents all categories of task k-1.

7. A small sample class incremental learning system based on prototype features, characterized in that: It includes a feature extractor and a prototype correction module: the feature extractor maps the original data into a feature space to generate a high-dimensional feature vector; The prototype correction module is responsible for generating and optimizing category prototypes in the feature space to ensure the distinction between categories and the consistency within categories, thereby supporting efficient small-sample class incremental learning.

8. The image classification system based on small sample class incremental learning of prototype features according to claim 7, characterized in that: After the feature extractor is trained on the base class task, its parameters remain fixed, providing a general feature representation capability in the incremental task; The prototype correction module enhances the prototype features of each class in the incremental process through contrastive learning and self-attention mechanism.

9. An image classification device based on small sample class incremental learning of prototype features, characterized in that: include: Memory: used to store a computer program for implementing an image classification method based on small sample class incremental learning of prototype features as described in any one of claims 1 to 6; Processor: used to implement the image classification method of small sample class incremental learning based on prototype features as described in any one of claims 1-6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement the image classification method based on small sample class incremental learning based on prototype features as described in any one of claims 1 to 6.

Images