Incremental learning classification method for small sample image classification

Abstract

The invention provides an incremental learning classification method for small sample image classification, which belongs to the field of computer vision, incremental learning, prompt learning and machine learning, and comprises the following modules: a base class training prompt regularization module, a semantic knowledge distillation module, an incremental class training prompt regularization module and a subspace prototype aggregation module. Compared with the prior art, two regularization strategies are designed by utilizing the generalization ability of the pre-trained visual model, and the generalization ability of the model is kept when the model is finely adjusted by applying the self-regularization constraint in the large sample training in the model basic training stage; in the small sample training in the increment stage, regularization constraint is applied to prompts through generalization ability of training prompts in the basic stage and the model itself so as to relieve the overfitting problem. Meanwhile, the training of a semantic knowledge distillation module auxiliary model is designed by utilizing the label text description of the image and a language model. In addition, subspace aggregation is carried out on a new-class small sample prototype by using a prototype calculated by a base-class large sample, so that the discriminability of the new class is improved.

Description

Technical Field

[0001] The technical field to which the present invention belongs is the field of machine learning, and its application is the field of image classification, and specifically relates to a regularization strategy for prompt learning and a calibration scheme for a prototype network. Background Art

[0002] Image classification is a widely used computer vision task. It can not only automatically process a large amount of visual data, but also provide accurate decision support in many fields. For example, in the medical field, image classification can help doctors quickly detect and diagnose diseases and improve the efficiency of early screening; in autonomous driving, it ensures the safe driving of vehicles by identifying traffic signs, pedestrians and obstacles; in security monitoring and industrial quality inspection, image classification can identify abnormal situations in real time and effectively prevent safety accidents and production defects. Therefore, image classification technology plays a vital role in improving work efficiency, reducing labor costs, and enhancing decision-making accuracy. As data distribution continues to change, deep learning models often need to transfer from a familiar data distribution to another unfamiliar data distribution and learn new knowledge from new data. In some real-world application scenarios, the cost of data collection and labeling is sometimes unacceptable. For example, in training incremental models for face recognition and in intelligent medical decision-making systems that track physiological signals, the model needs to learn new categories from limited data streams, which has led domestic and foreign scholars to study the problem of small sample incremental learning. The current small sample incremental learning methods usually involve a backbone network responsible for feature extraction and a prototype network responsible for classification. The overfitting caused by fine-tuning the backbone network will cause the performance of the previous task to deteriorate severely, and is not conducive to the learning of subsequent tasks. Therefore, many methods freeze the backbone network in the basic training stage to prevent overfitting problems, but this cannot make good use of the large sample knowledge in the basic stage to improve the model performance. In the existing patents, the SAR image target recognition method and device based on small sample incremental learning proposed by Zhao Lingjun et al. constructs a multi-class target confusion training set and a single-class target rotation training set to increase the target categories of training, so that the trained feature extractor can quickly extract and characterize the unknown target features in the real recognition scene, and the method, device and medium for small sample incremental learning of a small number of network parameters manipulation parameters proposed by Ye Qixiang et al. generate the control coefficient α by designing a small number of parameter control submodules. The feature representation capability of the incremental new class is expanded by the control coefficient α, but these patents do not take into account the impact of the imbalance of new and old class data on the prototype network, and fail to make good use of the information of image labels. The parameters of the prototype network come from the feature mean of each category, and the imbalance in the number of samples between the base class and the new class causes the prototype network to tend to predict the new class as the base class that is most similar to it, which reduces the accuracy of the new class. In their NeurIPS paper, Wang Qiwei et al. proposed to aggregate the prototypes of the base class and the new class in the current incremental session based on similarity. This method alleviates the problem of classification bias to a certain extent, but the update process is directly based on the weighted average of the current prototype and the base class prototype, which may cause a large deviation due to the weight distribution, especially when the distribution difference between the base class and the new class is large.The present invention adopts a subspace aggregation prototype method, and together with a two-stage prompt parameter regularization strategy and a semantic knowledge distillation module, forms a small sample incremental model for image classification. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention utilizes the generalization ability of the pre-trained model itself, designs corresponding regularization strategies in the basic training stage and incremental training stage of the model to alleviate the overfitting problem of the model, and designs a subspace prototype aggregation algorithm to calibrate the deviation of the prototype network. Ultimately, the ability of the image classification model to learn new categories in a small number of samples is improved.

[0004] like Figure 1 As shown, an incremental learning classification method for small sample image classification mainly includes: S103 base class training prompt regularization module, S104 semantic knowledge distillation module, S107 incremental class training prompt regularization module, S110 subspace prototype aggregation module.

[0005] like Figure 2 As shown in FIG. 1 , the base class training prompt regularization module alleviates the overfitting of prompt parameters by utilizing the generalization ability of the pre-trained model itself, so that the model can learn task-specific knowledge and task-independent generalized knowledge in a balanced manner at the basic stage.

[0006] like Figure 3 As shown, the incremental training prompt regularization module applies regularization constraints to the newly trained prompt parameters based on the prompt parameters trained in the basic stage and the generalization ability of the model itself to alleviate the overfitting problem of small sample training.

[0007] like Figure 4 As shown, the semantic knowledge distillation module provides semantic information to assist the training of the model based on the language encoder model and the text labels of the image samples.

[0008] The subspace prototype aggregation module uses the rich feature information of the large sample prototype of the base class to supplement the insufficient feature information of the small sample prototype, thereby improving the model's ability to recognize new categories.

[0009] As an embodiment of the present invention, the base class training prompt regularization module first trains a prompt parameter specific to the basic task, then aligns the output with the prompt and the model output without the prompt through KL divergence, and uses the two outputs to jointly optimize the model. It is possible to learn task-specific and task-independent knowledge in a balanced manner.

[0010] As an embodiment of the present invention, the semantic knowledge distillation module can be used to learn the semantic information of the language model and the label text description, thereby enriching the semantic information of the prompt parameter and serving as auxiliary information for model training in the usage scenario where the acquisition of image samples is limited.

[0011] As an embodiment of the present invention, the incremental class training prompt regularization module is an improvement on the base class prompt regularization module, which can better adapt to small sample tasks.

[0012] As an embodiment of the present invention, the subspace prototype aggregation module can assign different aggregation weights through a similarity matrix to enhance the feature representation of the new class prototype, thereby alleviating the problem that the prototype network is biased towards the base class.

[0013] Through the above modules, incremental learning of small sample image data sets can be completed.

[0014] The beneficial effects of the present invention are:

[0015] An incremental learning classification method for small sample image classification, which imposes regularization constraints on model training based on pre-trained visual models and language models. In the base class training stage of the model, this patent can fine-tune the model while maintaining the generalization ability of the pre-trained semantic model, so that it can fully learn the rich image knowledge brought by the large samples of the base class, and will not have a negative impact on the learning of subsequent tasks. In the incremental training stage of the model, this patent can alleviate the overfitting problem of the model for small sample tasks, and thus alleviate the catastrophic forgetting problem in incremental learning. For the prototype network, this patent fully alleviates the deviation problem caused by the large gap in the number of samples between the base class and the new class, and greatly improves the accuracy of new category prediction. Therefore, the present invention ultimately improves the stability and plasticity of the model to a certain extent.

[0016] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0017] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0019] Figure 1The flowchart of the whole module of an incremental learning classification method for small sample image classification in an embodiment of the present invention includes main modules such as S103 base class training prompt regularization module, S104 semantic knowledge distillation module, S107 incremental class training prompt regularization module, S110 subspace prototype aggregation module, etc., wherein S103 is a pre-module of S107 and S110, S104 is in parallel relationship with S103 and S107, and S110 is a post-module of S104 and S107;

[0020] Figure 2 Detailed structural diagram of a base class training prompt regularization module S103 in an incremental learning classification method for small sample image classification in an embodiment of the present invention;

[0021] Figure 3 Detailed structural diagram of an incremental class training prompt regularization module S107 in an incremental learning classification method for small sample image classification in an embodiment of the present invention;

[0022] Figure 4 Detailed structural diagram of the semantic knowledge distillation module S110 in an incremental learning classification method for small sample image classification in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0024] See also Figure 1 The embodiment of the present invention provides an incremental learning classification method for small sample image classification, and its overall implementation process includes: S101 obtains a base class data set. Like most incremental learning tasks, the data will be divided into (D0, D1, D2, ..., D t ) multiple task data sets, D0 represents the base class data set, and the sample data is represented as x is an image sample, y is the corresponding label; S102 performs model basic stage training, which includes S103 basic training prompt regularization module and S104 semantic knowledge distillation module, and S103 and S104 are parallel; S105 updates the prototype network parameters, extracts the image features of each category in D0 by using the model trained in the basic stage, and calculates the mean c k Expressed as the mean of the category It should be noted that after the initial training, the subspace aggregation module S110 does not need to be used for calibration; S108 incremental training, this training stage includes S107 incremental prompt regularization module and S104 semantic knowledge distillation module, S107 and S104 are in parallel relationship; S105 updates the prototype network, and the prototype network update operation after the incremental training stage includes S110 subspace prototype aggregation module;

[0025] The working principle of the above technical solution is as follows: a two-stage prompt regularization strategy is designed in the basic training stage, and a large number of image samples in the basic stage are used to enhance the model's discrimination ability while maintaining generalization ability. During the incremental stage training, the large sample feature prototypes obtained from the basic stage training of the model are fully utilized to enhance small sample training. For example, this patent constructs a similarity matrix through prototype similarity, and based on this similarity matrix, the model output with the basic stage prompt parameters and the current model output without prompts are weighted and regularized. When updating the prototype network, the base class prototype feature information is also aggregated to the current prototype according to the prototype similarity matrix to enrich the feature information of the current prototype.

[0026] The beneficial effects of the above technical solution are: by designing a good basic stage training strategy, the stability and plasticity of the model are enhanced in the basic stage, and the large sample knowledge acquired in the basic stage is used to help the model learn the small sample knowledge in the incremental stage, which ultimately greatly alleviates the catastrophic forgetting problem of the incremental model.

[0027] See also Figure 2 In one embodiment, the base training prompts the regularization module to alleviate the overfitting problem caused by the fine-tuning model by applying a self-regularization.

[0028] The working principle of this technical solution: Considering that good generalization ability helps the model learn downstream tasks, and downstream tasks usually have only a small number of samples. In order to better utilize the large sample data of basic training and help the learning of subsequent tasks, the present invention designs a two-stage prompt regularization strategy in the basic training stage:

[0029]

[0030] Among them, y i represents the i-th image sample, represents the model output prediction probability of the i-th sample, L CE and L KLThey are cross entropy loss and KL divergence loss, and epoch is the number of back propagation rounds in deep learning. This patent further divides the basic training stage into two training stages. In the first stage, when epoch < ρ, the prompt parameters are trained with cross entropy in order to make the prompt parameters carry task-specific knowledge. In the second stage, when epoch ≥ ρ, the model is a loss function consisting of three parts, represents the cross entropy loss calculated using the model output with hints, represents the cross entropy loss calculated on the output of the unprompted model, represents the KL divergence loss of the output of the two models.

[0031] The beneficial effect of the above technical solution is: using the generalization ability of the pre-trained visual model itself and the prompt parameters specific to the basic task to balance the learning of task-specific knowledge and task-independent knowledge.

[0032] See also Figure 3 In one embodiment, the incremental training prompts the regularization module: it can be used to alleviate the overfitting problem of the model in small sample incremental training, thereby reducing the degree of forgetting of the model. In this embodiment, the present invention makes full use of the large sample knowledge in the basic stage to help small sample task learning:

[0033]

[0034] Among them, w b and w c are the prototype matrices of the base class and the current new class respectively, τ is a hyperparameter, and S is the similarity matrix between the base class and the new class. is the model output with basic stage hints, This is the output without any prompts. This patent combines the above two outputs through the coefficient ρ(x) calculated based on the similarity matrix S and aligns them with the current prompt model output to alleviate the overfitting problem. The hint parameter with base class training information in assisted the training of incremental classes, and also took advantage of the silent model output To maintain generalization ability.

[0035] In one embodiment, the subspace prototype aggregation module enriches the feature information of the new class prototype based on the prototype aggregation operation of the base class and the new class similarity matrix. The present invention can effectively use the large sample prototype of the base class to enrich the small sample prototype features. Specific implementation measures: First, the base class prototype Wb is QR decomposed to obtain its Q matrix and R matrix, and then the current prototype Wc is multiplied by the Q matrix. The purpose of this step is to map the current prototype to the subspace of the base class prototype, and then normalize the current prototype in the subspace. The formula is expressed as: w CAt this point, it contains rich information about the large sample prototype of the base class. Then calculate the similarity matrix between the current prototype and the base class prototype Calculate ΔW based on the similarity matrix S c =w′ C S. The principle of this step is that if the current category has a high similarity with the base category, then the global features of the base category can better enrich the prototype features of the current new category, so a higher aggregation ratio should be given. Finally, the final calibration weight is calculated using convex combination. a is a hyperparameter.

[0036] See also Figure 4 The semantic knowledge distillation module can assist the model training by using the language model and the text description of the image label. For the data {xi,yi}, x represents the image sample, y represents its corresponding label information, and a pre-trained language encoder (such as the BERT model) is used to extract the text feature w of the label y. cni ,Use the visual encoder to extract the features of the language prompt prompt-L because and w cni Coming from two different encoders, directly combining the two outputs will have a poor effect. Therefore, the prototype network is needed here to serve as a stable basis to adjust the two features from different feature spaces. The specific method is to calculate the language prompt feature. The predicted probability output by the prototype network Then calculate With the cross entropy loss of the image label, the loss function of the semantic knowledge distillation module is expressed as:

[0037]

[0038] The beneficial effects of the above technical solution are: the present invention provides additional semantic information for model training, which can further enhance the stability of the incremental model. When image samples are difficult to obtain in the incremental stage, these semantic information can serve as supplementary information of the category to assist the training of the model.

[0039] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. An incremental learning classification method for small sample image classification, characterized in that: include: Base class training prompt regularization module, semantic knowledge distillation module, incremental class training prompt regularization module, subspace prototype aggregation module. The operation process of these modules is as follows: First, in the basic training stage of the incremental learning model, the base class training prompt regularization module is used to alleviate the generalization ability of the model. In the incremental training stage, the incremental class training prompt regularization module is used to further alleviate the overfitting problem caused by small sample training. The semantic knowledge distillation module is used in parallel with the base class training prompt regularization module and the quantity class training prompt regularization module; when updating the prototype network in the incremental stage, the subspace prototype aggregation module is used to improve the model's ability to discriminate new classes. The base class training prompts the regularization module. Considering that the model with good generalization performance can better adapt to downstream tasks and transfer knowledge, this module focuses on improving the generalization ability of the model in the basic stage and improving the discriminability of the base class. i represents the i-th image sample, represents the model output prediction probability of the i-th sample, L CE and L KL They are cross entropy loss and KL divergence loss, and epoch is the number of back propagation rounds in deep learning. This patent further divides the basic training stage into two training stages. In the first stage, when epoch < ρ, cross entropy is used. The purpose of training the prompt parameters is to make them carry task-specific knowledge. In the second stage, when epoch ≥ ρ, the generalization ability of the pre-trained visual model and the prompt parameters specific to the basic task are used to learn task-specific knowledge and task-irrelevant knowledge in a balanced manner. The semantic knowledge distillation module is a module that uses text descriptions of image tags and pre-trained language models to assist model training. The structure of this module is as follows: is the feature output of the language prompt, w cni is the label text description embedding vector of the i-th image sample output by the pre-trained language model, y i is the label of the i-th image, is the model prediction probability for the corresponding image. The semantic knowledge distillation module: Considering the difficulty of obtaining image samples in some application scenarios, this patent designs this module to use text description to assist the model in learning image knowledge. and w cni From different feature spaces, directly use KL divergence The alignment effect is poor, this patent adopts The feature differences are further eliminated. The incremental training prompt regularization module is a module that uses the prompt parameters after basic training and the generalization ability of the model itself to jointly impose regularization constraints on small sample training. Assume w b and w c are the prototype matrices of the base class and the current new class respectively, and S is the similarity matrix between the base class and the new class. is the model output with basic stage hints, It is an output without any prompt. This patent combines the above two outputs through the coefficient ρ(x) calculated based on the similarity matrix S and aligns them with the current prompt model output to alleviate the overfitting problem. The subspace prototype aggregation module normalizes the new class prototype in the base class subspace and aggregates it based on the base class-new class similarity matrix to improve the prototype network's ability to distinguish new classes. This module is a non-training module that takes effect after each update of the prototype network. The starting point of the subspace prototype aggregation module is that the sample size in the incremental stage of small sample incremental learning is small, and the category prototype of the incremental class cannot fully represent the characteristics of the class. Therefore, the base class prototype calculated by the large sample can be used to provide some category features for supplementation. For example, when calculating the prototype of the incremental class "dog", this patent will use the category prototypes similar to dogs in the base class, such as "wolf" and "fox", to supplement the characteristics of "dog" based on the similarity matrix.

2. The incremental learning classification method for small sample image classification as claimed in claim 1, characterized in that: Corresponding regularization strategies were designed in the basic training stage and incremental training stage of the model to alleviate the overfitting problem of the model.

3. The incremental learning classification method for small sample image classification as claimed in claim 1, characterized in that: The semantic knowledge distillation module is used to provide semantic information from text descriptions in all stages of model training.

4. The incremental learning classification method for small sample image classification as claimed in claim 1, characterized in that: The subspace regularization module is used to calibrate the prototype network, solve the problem that the prototype network is biased towards the base class, and improve the discriminability of new classes.

5. The regularization strategy of claim 2, characterized in that: In the basic stage, the generalization ability of the pre-trained model is used to gradually learn task-specific knowledge and task-independent knowledge in two steps, thereby improving the discriminability of the base class while maintaining the generalization ability of the model.

6. The regularization strategy of claim 2, characterized in that: In the incremental stage, the basic stage hints and the generalization performance of the model are used to perform weighted regularization based on the similarity matrix between the new and old classes to maintain the generalization ability of the model in small sample training.

7. The subspace regularization module of claim 5, characterized in that: By utilizing the large sample prototype data of the base class, the prototype of the current new class is normalized in the base class subspace to obtain the base class feature information, thus compensating for the negative impact of the incremental class prototype caused by insufficient samples.

Images