Incremental image classification method based on small sample classes based on cue word fine-tuning and feature replay

Abstract

A small sample class incremental image classification method based on cue word fine-tuning and feature replay belongs to the field of small sample class incremental image classification in the field of computer vision. The present invention aims at the problem of poor classification ability of existing small sample class incremental image classification methods based on pure visual models. It includes using the concatenation of learnable cue words and category names as the input of a text encoder, using the matching results of image features and text features, and the real label to calculate the cross entropy loss function to optimize the learnable cue words; using the cue words trained in the previous stage to initialize the cue words in the current stage, and estimating the Gaussian distribution at the feature level of each category according to the features of the training pictures of each category and the synthetic features output by VAE; using the feature Gaussian distribution of the old category for virtual feature sampling in the current stage, calculating the cross entropy loss function, and combining the cross entropy losses of the new and old categories to constrain the optimization direction of the cue words. The present invention is used for incremental image classification.

Description

Technical Field

[0001] The invention relates to a small sample class incremental image classification method based on prompt word fine-tuning and feature replay, and belongs to the field of small sample class incremental image classification in the field of computer vision. Background Art

[0002] The class-incremental image classification problem is a special machine learning problem that focuses on enabling the image classification model to continuously learn new categories without forgetting the classification ability of old categories. Specifically, the model is first trained on an initial category dataset so that the model acquires the image classification ability in the category space. Next, the model needs to continuously learn new categories in a series of stages. In each stage, the model can only be trained on new category data and cannot access the training data of old categories. After the training of this stage is completed, the model is required to have the classification ability of both the new categories of this stage and the old category data of the previous stage.

[0003] However, in real-world applications, it is expensive and impractical to continuously obtain large amounts of data annotations for new categories. For example, in applications such as facial recognition systems and smart photo albums, annotations for new categories of images are often limited. To address this problem, small sample incremental image classification methods have emerged. For these methods, the models are designed to be able to adapt and recognize new categories with very little support from training data. These methods not only improve the model's ability to adapt to new categories, but also demonstrate their application potential and flexibility in data-scarce environments.

[0004] Most of the existing small sample incremental image classification methods are based on pure visual networks (such as convolutional neural networks), which first perform pre-training in a base stage with sufficient training images, and then use knowledge distillation or regularization methods to ensure that the model does not forget the old category knowledge when learning new category knowledge in the incremental stage. This method based on pure visual models is limited by the feature representation ability of the image encoder, and the semantic separability of its new and old category images in the feature space is poor, which affects the classification ability of the model. Summary of the invention

[0005] In view of the problem that the existing small-sample incremental image classification method based on pure visual model has poor semantic separability between new and old category images in feature space, which affects the classification ability of the model, the present invention provides a small-sample incremental image classification method based on prompt word fine-tuning and feature replay.

[0006] The invention provides a small sample class incremental image classification method based on cue word fine-tuning and feature replay, comprising:

[0007] Establish a CLIP classification model, use base class samples to perform base stage training on the CLIP classification model to obtain a base class trained CLIP classification model, then use incremental samples in different small sample incremental sample libraries to perform incremental stage training on the base class trained CLIP classification model, and obtain an incremental stage trained CLIP classification model for incremental image classification;

[0008] The training process of the CLIP classification model after base class training is obtained from the CLIP classification model includes:

[0009] Randomly initialize to obtain the learnable prompt word prefix vector, and concatenate it with the N category vectors of the base class samples to obtain N complete prompt texts. The N category text features of the base stage are obtained through the text encoder.

[0010] Each base class sample is passed through an image encoder to obtain the base class sample image coding features, and the cosine similarity scores of each base class sample image coding feature and the N category text features in the base stage are calculated respectively, and the cross entropy loss function of the current training stage is calculated in combination with the label of each base class sample; during the base stage training process, the cross entropy loss is minimized, and the learnable prompt word prefix vector is optimized using the back propagation algorithm to obtain the optimized prompt word prefix vector in the base stage; the CLIP classification model after base class training is obtained;

[0011] Obtaining N categories of base class sample feature-level Gaussian distributions and corresponding base class Gaussian distribution parameters from all base class sample image encoding features; storing the N base class Gaussian distribution parameters as base class category knowledge;

[0012] The training process of obtaining the CLIP classification model after incremental training from the CLIP classification model after base class training includes:

[0013] The prefix vector of the prompt word after optimization in the base stage is used as the initial prompt word vector of the current incremental stage, and is concatenated with the N1 category vectors composed of the current incremental sample and the base class sample, and then the text features of N1 categories are obtained through the text encoder;

[0014] The cross entropy loss function of the current training stage is calculated based on each incremental sample, the text features of N1 categories and the label of each incremental sample;

[0015] Randomly sample B base class Gaussian distribution parameters from the N base class Gaussian distribution parameters of the base class category knowledge, sample the base class sample image encoding features from them, and calculate the base class stage cross entropy loss function with the corresponding base class sample labels;

[0016] The final cross entropy loss function of the current stage is obtained from the cross entropy loss function of the current training stage and the cross entropy loss function of the base class stage; during the training process of the incremental stage, the final cross entropy loss function of the current stage is minimized, and the initialization prompt word vector of the current stage is optimized using the back propagation algorithm to obtain the optimized prompt word prefix vector of the current incremental stage; and the CLIP classification model after training in the current incremental stage is obtained;

[0017] Then, the incremental sample category features of N2 categories are obtained from the incremental sample image coding features; at the same time, the pre-trained VAE decoder is used to obtain M reconstructed synthetic image features based on the incremental samples from M Gaussian distribution random sampling noise vectors with a mean of 0 and a variance of 1; then, the incremental sample feature-level Gaussian distribution of each category is estimated from each incremental sample image coding feature and M synthetic image features in the incremental sample category features of each category, and N2 incremental sample Gaussian distribution parameters are obtained; the N2 incremental sample Gaussian distribution parameters are continued to be stored in the base category knowledge to obtain the current incremental category knowledge; N2+N=N1;

[0018] When continuing to train the incremental samples in the next small sample incremental sample library, the process of obtaining the CLIP classification model after training in the next incremental stage from the CLIP classification model after training in the current incremental stage corresponds to the same training process as that of obtaining the CLIP classification model after training in the incremental stage from the CLIP classification model after training in the base class.

[0019] According to the small sample class incremental image classification method based on prompt word fine-tuning and feature replay of the present invention, the method for calculating the cross entropy loss function of the current training stage based on each incremental sample, the text features of N1 categories and the label of each incremental sample is:

[0020] Each incremental sample is passed through the image encoder to obtain the incremental sample image coding features, and the cosine similarity scores between each incremental sample image coding feature and the N1 category text features of the current incremental stage are calculated respectively. The cross entropy loss function of the current training stage is calculated in combination with the label of each incremental sample.

[0021] According to the small sample class incremental image classification method based on cue word fine-tuning and feature replay of the present invention, the method for calculating the cross entropy loss function at the base class stage is:

[0022] B base class Gaussian distribution parameters are randomly sampled from the N base class Gaussian distribution parameters of the base class category knowledge, and a base class sample image encoding feature is sampled from the base class sample feature-level Gaussian distribution corresponding to the B base class Gaussian distribution parameters to obtain B pseudo-features. The cosine similarity scores of the B pseudo-features and the N1 category text features in the current incremental stage are calculated respectively, and the cross entropy loss function of the base class stage is calculated in combination with the corresponding base class sample labels.

[0023] According to the small sample class incremental image classification method based on prompt word fine-tuning and feature replay of the present invention, the pre-trained VAE decoder includes a decoding multi-layer perceptron MLP and a decoding partial text encoder, and the pre-training process of the VAE decoder includes:

[0024] The incremental samples of the current incremental stage are passed through the image encoder to obtain the incremental sample image coding features, and then passed through the VAE encoder to obtain the estimated mean and the estimated variance vector By the estimated mean and the estimated variance vector The divergence loss is calculated from the Gaussian distribution composed of the mean and variance of 1. At the same time, a noise vector is randomly sampled from the Gaussian distribution with mean 0 and variance 1, and the estimated mean is used. and the estimated variance vector Perform linear transformation to obtain linear transformation features; input the linear transformation features into the decoding multi-layer perceptron MLP to obtain an estimated bias value; add the estimated bias value to the preset learnable vector to obtain a reconstructed prompt word, concatenate the reconstructed prompt word with the corresponding category vector, and then obtain the reconstructed synthetic image features through the decoding partial text encoder; calculate the L-2 loss from the reconstructed synthetic image features and the incremental sample image coding features;

[0025] The final loss of VAE is obtained by the divergence loss and L-2 loss;

[0026] Minimize the final loss of VAE, use the back-propagation algorithm to optimize the VAE encoder, decoding multi-layer perceptron MLP and preset learnable vectors to obtain the pre-trained VAE decoder.

[0027] According to the small sample class incremental image classification method based on cue word fine-tuning and feature replay of the present invention, the cross entropy loss function of the base stage is:

[0028]

[0029] Where exp(·) is the exponential function, f is the encoding feature of the base class sample image, and g y represents the text feature of the corresponding category of the base class sample image encoding feature f, τ is the temperature coefficient, c represents the category, represents the category set of the sth stage. In the base stage, Contains N categories; is the set of categories from the base stage t to the current incremental stage s; g c is the text feature of category c.

[0030] According to the small sample class incremental image classification method based on prompt word fine-tuning and feature replay of the present invention, in the base stage, the base class Gaussian distribution parameters corresponding to the base class sample feature level Gaussian distribution of each category include the mean vector μ c and the covariance diagonal matrix ∑ c :

[0031]

[0032]

[0033] Where N c is the number of base class samples of category c, f i Encode features for the i-th base class sample image in category c, is the covariance diagonal matrix ∑ c The value of the dth dimension on the diagonal of id f i The d-th dimension encoding feature, μ cd μ c The d-th dimension mean vector of .

[0034] According to the small sample class incremental image classification method based on prompt word fine-tuning and feature replay of the present invention, in the incremental stage, the method for calculating the cross entropy loss function of the current training stage is the same as the method for calculating the cross entropy loss function of the base stage; wherein the total number of categories is adaptively updated;

[0035] In the incremental stage, the base class stage cross entropy loss function is calculated as

[0036]

[0037] In the formula is the pseudo feature corresponding to the b-th base class Gaussian distribution parameter, Pseudo-feature The text features of the corresponding categories.

[0038] According to the small sample class incremental image classification method based on prompt word fine-tuning and feature replay of the present invention, the final cross entropy loss function obtained in the incremental stage is:

[0039]

[0040] In the formula is the calculated cross entropy loss function of the current training stage, λ o is the weight coefficient of the classification model.

[0041] According to the small sample class incremental image classification method based on cue word fine-tuning and feature replay of the present invention, the divergence loss in the pre-training process of the VAE decoder is

[0042]

[0043] Where KL is the KL divergence, represents Gaussian distribution;

[0044] L-2 loss is

[0045]

[0046] In the formula is the feature of the reconstructed synthetic image, and fz is the encoding feature of the incremental sample image.

[0047] According to the small sample class incremental image classification method based on cue word fine-tuning and feature replay of the present invention, the final loss of VAE is

[0048]

[0049] Where λ r is the weight coefficient of the pre-trained model.

[0050] Beneficial effects of the present invention: The method of the present invention uses the visual-language multimodal pre-training model CLIP as the basic network for image classification, and implements the image classification process based on the similarity between the image features and the language features constrained by the category name. The concatenation of the learnable prompt words and the category name is used as the input of the text encoder, and the cross entropy loss function is calculated using the matching results of the image features and the text features, as well as the real labels to optimize the learnable prompt words. The prompt words trained in the previous stage are used to initialize the prompt words in the current stage. According to the features of the training pictures of each category and the synthetic features output by the variational autoencoder (VAE), the Gaussian distribution at the feature level of each category is estimated. In the current stage, the characteristic Gaussian distribution of the old category is used for virtual feature sampling, the cross entropy loss function is calculated using the sampling features of the old category, and the cross entropy loss of the new and old categories is combined to constrain the optimization direction of the prompt words. Through the above training method, the model's ability to resist catastrophic forgetting of the old category is effectively improved, thereby improving the model's classification ability for all category images.

[0051] The method of the present invention adopts a visual-language multimodal pre-training model to construct a class incremental image classifier. Compared with a pure visual model, the image encoder in the visual-language multimodal pre-training model has a stronger feature representation capability, making the images of new categories and old categories more semantically separable in the feature space. At the same time, the language level information of the category (category name) is encoded into the text feature by the text encoder, so that the text feature is constrained by the semantic information of the category name in the feature space, effectively alleviating the problem of catastrophic forgetting of the model for old knowledge. In the method of the present invention, a learnable prompt word is used to replace a fixed prompt word template (for example, "A photo of a[CLS]", [CLS] is the category name), and by optimizing the loss function, the prompt word can be more adapted to the information in each incremental stage, thereby improving the classification ability of the model. In addition, the existing class incremental image classification method based on replay directly stores the original image of the old category, which will bring a large storage overhead. In the method of the present invention, a Gaussian distribution of feature levels is maintained for each category, and pseudo features are randomly sampled from these Gaussian distributions to achieve the replay of old category knowledge. The Gaussian distribution of each category only requires two vectors to represent, thus greatly saving the storage overhead of old knowledge. Therefore, this method saves more storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 y is a flowchart of using Gaussian distribution for feature replay and optimizing prompt words in the incremental training phase of the small sample incremental image classification method based on prompt word fine-tuning and feature replay according to the present invention; (t) represents the sample category label at stage t; x (t) represents an incremental sample;

[0053] Figure 2 is a schematic diagram of the construction process of the feature-level Gaussian distribution in the incremental training phase; cls represents the category mean vector, is the diagonal value of the category covariance diagonal matrix;

[0054] Figure 3 This is a schematic diagram of the pre-training process of the VAE decoder. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0057] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but they are not intended to limit the present invention.

[0058] Specific implementation method 1. Combination Figures 1 to 3 As shown, the present invention provides a small sample class incremental image classification method based on cue word fine-tuning and feature replay, comprising:

[0059] Establish a CLIP classification model, use base class samples to perform base stage training on the CLIP classification model to obtain a base class trained CLIP classification model, then use incremental samples in different small sample incremental sample libraries to perform incremental stage training on the base class trained CLIP classification model, and obtain an incremental stage trained CLIP classification model for incremental image classification;

[0060] The training process of the CLIP classification model after base class training is obtained from the CLIP classification model includes:

[0061] Randomly initialize to obtain the learnable prompt word prefix vector, and concatenate it with the N category vectors of the base class samples to obtain N complete prompt texts. The N category text features of the base stage are obtained through the text encoder.

[0062] Each base class sample is passed through an image encoder to obtain the base class sample image coding features, and the cosine similarity scores of each base class sample image coding feature and the N category text features in the base stage are calculated respectively, and the cross entropy loss function of the current training stage is calculated in combination with the label of each base class sample; during the base stage training process, the cross entropy loss is minimized, and the learnable prompt word prefix vector is optimized using the back propagation algorithm to obtain the optimized prompt word prefix vector in the base stage; the CLIP classification model after base class training is obtained;

[0063] Obtaining N categories of base class sample feature-level Gaussian distributions and corresponding base class Gaussian distribution parameters from all base class sample image encoding features; storing the N base class Gaussian distribution parameters as base class category knowledge;

[0064] The training process of obtaining the CLIP classification model after incremental training from the CLIP classification model after base class training includes:

[0065] The prefix vector of the prompt word after optimization in the base stage is used as the initial prompt word vector of the current incremental stage, and is concatenated with the N1 category vectors composed of the current incremental sample and the base class sample, and then the text features of N1 categories are obtained through the text encoder;

[0066] The cross entropy loss function of the current training stage is calculated based on each incremental sample, the text features of N1 categories and the label of each incremental sample;

[0067] Randomly sample B base class Gaussian distribution parameters from the N base class Gaussian distribution parameters of the base class category knowledge, sample the base class sample image encoding features from them, and calculate the base class stage cross entropy loss function with the corresponding base class sample labels;

[0068] The final cross entropy loss function of the current stage is obtained from the cross entropy loss function of the current training stage and the cross entropy loss function of the base class stage; during the training process of the incremental stage, the final cross entropy loss function of the current stage is minimized, and the initialization prompt word vector of the current stage is optimized using the back propagation algorithm to obtain the optimized prompt word prefix vector of the current incremental stage; and the CLIP classification model after training in the current incremental stage is obtained;

[0069] Then, the incremental sample category features of N2 categories are obtained from the incremental sample image coding features; at the same time, the pre-trained VAE decoder is used to obtain M reconstructed synthetic image features based on the incremental samples from M Gaussian distribution random sampling noise vectors with a mean of 0 and a variance of 1; then, the incremental sample feature-level Gaussian distribution of each category is estimated from each incremental sample image coding feature and M synthetic image features in the incremental sample category features of each category, and N2 incremental sample Gaussian distribution parameters are obtained; the N2 incremental sample Gaussian distribution parameters are continued to be stored in the base category knowledge to obtain the current incremental category knowledge; N2+N=N1;

[0070] When continuing to train the incremental samples in the next small sample incremental sample library, the process of obtaining the CLIP classification model after training in the next incremental stage from the CLIP classification model after training in the current incremental stage corresponds to the same training process as that of obtaining the CLIP classification model after training in the incremental stage from the CLIP classification model after training in the base class.

[0071] After the training process of the previous incremental stage is completed, when the next incremental stage training is carried out, relative to the next incremental sample, all the incremental samples that have completed the training before can be used together with the base class samples as the base class samples and base class categories, and then the next training is carried out. At this time, the trained CLIP classification model of the next incremental stage can be obtained according to the data processing process similar to the training process of obtaining the CLIP classification model after incremental stage training from the CLIP classification model after base class training.

[0072] In this implementation mode, the CLIP classification model is built with pre-trained model weights. The image encoder and text encoder are loaded with pre-trained weights officially released by the CLIP model. The weights of the learnable prompt word prefix vectors are determined by random initialization. In the base stage, the prompt word is optimized. The base class sample feature-level Gaussian distribution is constructed for each category in the base stage. Then the training of the first incremental stage is entered. In each incremental stage, the prompt word optimized in the adjacent previous stage is used as the initialization of the prompt word in this stage.

[0073] After completing the last incremental stage of training, the training process ends. p After training for an iterative cycle, the weights of each group of student networks are averaged to obtain the model weight after parameter averaging.

[0074] The image encoder can be represented as E Img (x), the text encoder can be represented as E Txt (p), the learnable hint word prefix vector obtained by random initialization can be expressed as [V]1[V]2…[V] L The complete prompt text obtained by concatenating the learnable prompt word prefix vector and the category vector can be expressed as p(ν)=[V]1[V]2…[V] L [CLS], where [CLS] represents the vectorized form of the category name; then the text feature of each category The base class sample is input into the image encoder, and the obtained base class sample image encoding feature f = E Img (x).

[0075] Further, combined with Figure 1 As shown in the figure, the method for calculating the cross entropy loss function of the current training stage based on each incremental sample, the text features of N1 categories and the label of each incremental sample is:

[0076] Each incremental sample is passed through the image encoder to obtain the incremental sample image coding features, and the cosine similarity scores between each incremental sample image coding feature and the N1 category text features of the current incremental stage are calculated respectively. The cross entropy loss function of the current training stage is calculated in combination with the label of each incremental sample.

[0077] Recombination Figure 1 As shown, the method for calculating the cross entropy loss function at the base class stage is:

[0078] B base class Gaussian distribution parameters are randomly sampled from the N base class Gaussian distribution parameters of the base class category knowledge, and a base class sample image encoding feature is sampled from the base class sample feature-level Gaussian distribution corresponding to the B base class Gaussian distribution parameters to obtain B pseudo-features. The cosine similarity scores of the B pseudo-features and the N1 category text features in the current incremental stage are calculated respectively, and the cross entropy loss function of the base class stage is calculated in combination with the corresponding base class sample labels.

[0079] In this implementation, pseudo features are used in combination with incremental samples in the current incremental phase to optimize prompt words.

[0080] Then, the incremental samples of the current incremental stage are used to pre-train the VAE decoder.

[0081] Going a step further, combined Figure 3 As shown, the pre-trained VAE decoder includes a decoding multi-layer perceptron MLP and a decoding partial text encoder, and the pre-training process of the VAE decoder includes:

[0082] The incremental samples of the current incremental stage are passed through the image encoder to obtain the incremental sample image coding features, and then passed through the VAE encoder to obtain the estimated mean and the estimated variance vector By the estimated mean and the estimated variance vector The divergence loss is calculated from the Gaussian distribution composed of the mean and variance of 1. At the same time, a noise vector is randomly sampled from the Gaussian distribution with mean 0 and variance 1, and the estimated mean is used. and the estimated variance vector Perform linear transformation to obtain the features after linear transformation Input the linearly transformed features into the decoding multi-layer perceptron MLP to obtain the estimated bias value; add the estimated bias value to the preset learnable vector to obtain the reconstructed prompt word The reconstructed prompt word is concatenated with the corresponding category vector, and then the reconstructed synthetic image features are obtained by decoding the partial text encoder. The L-2 loss is calculated by reconstructing the synthetic image features and the incremental sample image coding features; the VAE encoder is a multi-layer perceptron MLP;

[0083] The final loss of VAE is obtained by the divergence loss and L-2 loss;

[0084] Minimize the final loss of VAE, use the back-propagation algorithm to optimize the VAE encoder, decoding multi-layer perceptron MLP and preset learnable vectors to obtain the pre-trained VAE decoder.

[0085] The VAE decoder synthesizes several reconstructed synthetic image features for each category in the current incremental stage, and then uses the incremental sample image encoding features and the reconstructed synthetic image features of the incremental samples to construct a feature-level Gaussian distribution for each category of the newly added samples.

[0086] In this implementation, the cross entropy loss function of the base stage is

[0087]

[0088] Where exp(·) is the exponential function, f is the encoding feature of the base class sample image, and g y represents the text feature of the corresponding category of the base class sample image encoding feature f, <·,·> represents the calculation of the cosine similarity of two vectors, τ is the temperature coefficient, c represents the category, represents the category set of the sth stage. In the base stage, Contains N categories; is the set of categories from the base stage t to the current incremental stage s; g c is the text feature of category c.

[0089] The cross entropy loss function is minimized as Minimize the loss and use the back-propagation algorithm to learn the prompt words [V]1[V]2…[V] L Make optimizations.

[0090] Furthermore, in the base stage, the base class samples are input into the image encoder to obtain the base class sample image coding features of each image; the feature-level Gaussian distribution of each category is estimated using the image coding features of each base class sample, the distribution parameters are obtained, and the Gaussian distribution parameters of each category are stored:

[0091] The base class Gaussian distribution parameters corresponding to the base class sample feature level Gaussian distribution of each category include the mean vector μ c and the covariance diagonal matrix ∑ c :

[0092]

[0093]

[0094] Where N c is the number of base class samples of category c, f i Encode features for the i-th base class sample image in category c, is the covariance diagonal matrix ∑ c The value of the dth dimension on the diagonal of id f i The d-th dimension encoding feature, μ cd μc The d-th dimension mean vector of .

[0095] The method for calculating the cross entropy loss function of the current training stage is the same as the method for calculating the cross entropy loss function of the base stage; the total number of categories is adaptively updated;

[0096] In the incremental phase, randomly sample B categories from the previous phase Sample a pseudo feature from the feature-level Gaussian distribution corresponding to each of these categories Calculate the similarity score between the pseudo image feature and the text feature, and calculate the cross entropy loss of the previous stage category by comparing the similarity score with the label corresponding to the sampling

[0097] The calculated base class stage cross entropy loss function is

[0098]

[0099] In the formula is the pseudo feature corresponding to the b-th base class Gaussian distribution parameter, Pseudo-feature The text features of the corresponding categories.

[0100] The final cross entropy loss function obtained in the incremental stage is

[0101]

[0102] In the formula is the calculated cross entropy loss function of the current training stage, λ o is the weight coefficient of the classification model. Minimize this loss and use the back propagation algorithm to learn the prompt words [V]1[V]2…[V] L Make optimizations.

[0103] Furthermore, the current training image is input into the image encoder of CLIP to obtain the image feature f; the image feature is input into the learnable MLP to obtain the estimated mean and variance vector Compute the KL loss based on a Gaussian distribution consisting of an estimated mean and variance and a Gaussian distribution with mean 0 and variance 1:

[0104] During the pre-training of the VAE decoder, the divergence loss is

[0105]

[0106] Where KL is the KL divergence, represents Gaussian distribution;

[0107] L-2 loss is

[0108]

[0109] In the formula is the feature of the reconstructed synthetic image, and fz is the encoding feature of the incremental sample image.

[0110] The final loss of VAE is

[0111]

[0112] Where λ r is the weight coefficient of the pre-trained model, usually set to 1. Minimize the final loss of VAE and use the back-propagation algorithm to optimize all learnable MLPs and learnable prompt words in VAE.

[0113] In the incremental stage, a noise vector is randomly sampled from a Gaussian distribution with a mean of 0 and a variance of 1. The noise vector is input into the trained MLP to obtain an estimated bias value. The VAE-trained prompt word is added to the bias value, concatenated with the category noun vector corresponding to a category in the current stage, and input into the text encoder to obtain a synthesized image feature. M image features are synthesized for each category in the current stage. M is usually set to 2 times N c ;

[0114] Input the incremental samples of the current stage into the image encoder respectively to obtain the incremental sample image encoding features of each image Using the image features of each category and the M synthesized image features, estimate the feature-level Gaussian distribution of the category, obtain the distribution parameters (mean vector and diagonal covariance matrix), and store the Gaussian distribution parameters of each category:

[0115]

[0116]

[0117] Where μ cz is the mean vector of the current incremental stage, is the covariance diagonal matrix ∑ of the current incremental stage cz The value of the dth dimension on the diagonal of cdz μ cz The d-th dimension mean vector of for The d-th dimension encoding feature of .

[0118] At this point, this incremental stage ends. If this incremental stage is the last incremental stage, the method ends. Otherwise, it enters the next incremental stage and repeats the above process.

[0119] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that the features described in conjunction with a single embodiment may be used in other described embodiments.

Claims

1. A small sample class incremental image classification method based on cue word fine-tuning and feature replay, characterized in that include, Establish a CLIP classification model, use base class samples to perform base stage training on the CLIP classification model to obtain a base class trained CLIP classification model, then use incremental samples in different small sample incremental sample libraries to perform incremental stage training on the base class trained CLIP classification model, and obtain an incremental stage trained CLIP classification model for incremental image classification; The training process of the CLIP classification model after base class training is obtained from the CLIP classification model includes: Randomly initialize to obtain the learnable prompt word prefix vector, and concatenate it with the N category vectors of the base class samples to obtain N complete prompt texts. The N category text features of the base stage are obtained through the text encoder. Each base class sample is passed through an image encoder to obtain the base class sample image coding features, and the cosine similarity scores of each base class sample image coding feature and the N category text features in the base stage are calculated respectively, and the cross entropy loss function of the current training stage is calculated in combination with the label of each base class sample; during the base stage training process, the cross entropy loss is minimized, and the learnable prompt word prefix vector is optimized using the back propagation algorithm to obtain the optimized prompt word prefix vector in the base stage; the CLIP classification model after base class training is obtained; Obtaining N categories of base class sample feature-level Gaussian distributions and corresponding base class Gaussian distribution parameters from all base class sample image encoding features; storing the N base class Gaussian distribution parameters as base class category knowledge; The training process of obtaining the CLIP classification model after incremental training from the CLIP classification model after base class training includes: The prefix vector of the prompt word after optimization in the base stage is used as the initial prompt word vector of the current incremental stage, and is concatenated with the N1 category vectors composed of the current incremental sample and the base class sample, and then the text features of N1 categories are obtained through the text encoder; The cross entropy loss function of the current training stage is calculated based on each incremental sample, the text features of N1 categories and the label of each incremental sample; Randomly sample B base class Gaussian distribution parameters from the N base class Gaussian distribution parameters of the base class category knowledge, sample the base class sample image encoding features from them, and calculate the base class stage cross entropy loss function with the corresponding base class sample labels; The final cross entropy loss function of the current stage is obtained from the cross entropy loss function of the current training stage and the cross entropy loss function of the base class stage; during the training process of the incremental stage, the final cross entropy loss function of the current stage is minimized, and the initialization prompt word vector of the current stage is optimized using the back propagation algorithm to obtain the optimized prompt word prefix vector of the current incremental stage; and the CLIP classification model after training in the current incremental stage is obtained; Then, the incremental sample category features of N2 categories are obtained from the incremental sample image coding features; at the same time, the pre-trained VAE decoder is used to obtain M reconstructed synthetic image features based on the incremental samples from M Gaussian distribution random sampling noise vectors with a mean of 0 and a variance of 1; then, the incremental sample feature-level Gaussian distribution of each category is estimated from each incremental sample image coding feature and M synthetic image features in the incremental sample category features of each category, and N2 incremental sample Gaussian distribution parameters are obtained; the N2 incremental sample Gaussian distribution parameters are continued to be stored in the base category knowledge to obtain the current incremental category knowledge; N2+N=N1; When continuing to train the incremental samples in the next small sample incremental sample library, the process of obtaining the CLIP classification model after training in the next incremental stage from the CLIP classification model after training in the current incremental stage corresponds to the same training process as that of obtaining the CLIP classification model after training in the incremental stage from the CLIP classification model after training in the base class.

2. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 1 is characterized in that: The method for calculating the cross entropy loss function of the current training stage based on each incremental sample, the text features of N1 categories and the label of each incremental sample is: Each incremental sample is passed through the image encoder to obtain the incremental sample image coding features, and the cosine similarity scores between each incremental sample image coding feature and the N1 category text features of the current incremental stage are calculated respectively. The cross entropy loss function of the current training stage is calculated in combination with the label of each incremental sample.

3. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 2 is characterized in that: The method for calculating the cross entropy loss function at the base class stage is: B base class Gaussian distribution parameters are randomly sampled from the N base class Gaussian distribution parameters of the base class category knowledge, and a base class sample image encoding feature is sampled from the base class sample feature-level Gaussian distribution corresponding to the B base class Gaussian distribution parameters to obtain B pseudo-features. The cosine similarity scores of the B pseudo-features and the N1 category text features in the current incremental stage are calculated respectively, and the cross entropy loss function of the base class stage is calculated in combination with the corresponding base class sample labels.

4. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 3 is characterized in that: The pre-trained VAE decoder includes a decoding multi-layer perceptron MLP and a decoding partial text encoder. The pre-training process of the VAE decoder includes: The incremental samples of the current incremental stage are passed through the image encoder to obtain the incremental sample image coding features, and then passed through the VAE encoder to obtain the estimated mean and the estimated variance vector By the estimated mean and the estimated variance vector The Gaussian distribution composed of the two components is used to calculate the divergence loss with a Gaussian distribution with a mean of 0 and a variance of 1. At the same time, a noise vector is randomly sampled from a Gaussian distribution with a mean of 0 and a variance of 1, and the estimated mean is used. and the estimated variance vector Perform linear transformation to obtain linear transformation features; input the linear transformation features into the decoding multi-layer perceptron MLP to obtain an estimated bias value; add the estimated bias value to the preset learnable vector to obtain a reconstructed prompt word, concatenate the reconstructed prompt word with the corresponding category vector, and then obtain the reconstructed synthetic image features through the decoding partial text encoder; calculate the L-2 loss from the reconstructed synthetic image features and the incremental sample image coding features; The final loss of VAE is obtained by the divergence loss and L-2 loss; Minimize the final loss of VAE, use the back-propagation algorithm to optimize the VAE encoder, decoding multi-layer perceptron MLP and preset learnable vectors to obtain the pre-trained VAE decoder.

5. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 4 is characterized in that: The cross entropy loss function at the base stage is Where exp(·) is the exponential function, f is the encoding feature of the base class sample image, and g y represents the text feature of the corresponding category of the base class sample image encoding feature f, τ is the temperature coefficient, c represents the category, represents the category set of the sth stage. In the base stage, Contains N categories; is the set of categories from the base stage t to the current incremental stage s; g c is the text feature of category c.

6. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 5, characterized in that: In the base stage, the base class Gaussian distribution parameters corresponding to the feature-level Gaussian distribution of the base class samples of each category include the mean vector μ c and the covariance diagonal matrix ∑ c : Where N c is the number of base class samples of category c, f i Encode features for the i-th base class sample image in category c, is the covariance diagonal matrix Σ c The value of the dth dimension on the diagonal of id f i The d-th dimension encoding feature, μ cd μ c The d-th dimension mean vector of .

7. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 6, characterized in that: In the incremental phase, the method for calculating the cross entropy loss function of the current training phase is the same as the method for calculating the cross entropy loss function of the base phase; the total number of categories is adaptively updated; In the incremental stage, the base class stage cross entropy loss function is calculated as In the formula is the pseudo feature corresponding to the b-th base class Gaussian distribution parameter, Pseudo-feature The text features of the corresponding categories.

8. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 7, characterized in that: The final cross entropy loss function obtained in the incremental stage is In the formula is the calculated cross entropy loss function of the current training stage, λ o is the weight coefficient of the classification model.

9. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 8, characterized in that: During the pre-training of the VAE decoder, the divergence loss is Where KL is the KL divergence, represents Gaussian distribution; L-2 loss is In the formula is the feature of the reconstructed synthetic image, and fz is the encoding feature of the incremental sample image.

10. The small sample class incremental image classification method based on prompt word fine-tuning and feature replay according to claim 9, characterized in that: The final loss of VAE is Where λ r is the weight coefficient of the pre-trained model.

Images