Continuous learning method based on two-stage prompt

Abstract

The invention discloses a two-stage prompt-based continuous learning method, which comprises the following steps of: inputting an input image into a parameter-frozen CLIP image encoder to generate image embedding, and splicing the image embedding and a prefix mark to form spliced image embedding; associating an input text prompt corresponding to the input image with the category name to form a first-level prompt, and inputting the first-level prompt into a parameter frozen CLIP text encoder to generate text embedding; calculating the similarity of image embedding and text embedding, and adjusting the similarity by using an adaptive weight to obtain the maximum K values in the adjusted similarity scores to form a second-level prompt; and inputting the spliced image embedding and the second-level prompt into the pre-training model VIT, outputting to obtain a final embedding, and performing softmax operation on the final embedding to obtain final prediction. The method does not need to finely adjust any parameter, does not need any buffer area to store past instance samples, and is suitable for data privacy sensitive scenes.

Description

Technical Field

[0001] The present invention relates to the field of continuous learning of artificial intelligence, and in particular to a continuous learning method based on two-level prompts. Background Art

[0002] Continuous learning can learn from non-IID data streams. The characteristic of continuous learning is that as the training data stream flows in, the model continues to learn incrementally and eventually completes the training of all tasks. In this process, the model can not only utilize the incoming new data and complete the task well on the new data based on the experience accumulated from previous tasks, but also avoid the forgetting problem and still maintain high accuracy for the tasks that have been trained. Its main challenge is called catastrophic forgetting, which comes from the fact that adaptation to the new distribution usually leads to a significant reduction in the ability to capture the old distribution.

[0003] In recent years, the field of continuous learning has developed rapidly, covering traditional continuous learning paradigms and hint-based continuous learning methods. Traditional de novo training methods aim to prevent knowledge forgetting by protecting important past parameters. Elastic weight integration retains learned knowledge and prevents interference from new information by applying regularization constraints in the loss function of new tasks. In addition, dynamic network architectures prevent the modification of important parameters by isolating previously learned parameters and assigning new parameters to new tasks. Most of these methods focus on solving the forgetting problem (stability) and ignore the model's ability to generalize quickly (plasticity), which requires multiple iterations to converge, posing a huge challenge in scenarios where new targets need to be recognized quickly.

[0004] Pre-trained models trained on large-scale data have introduced new research directions for continuous learning. Inspired by cue learning, these methods freeze the pre-trained Transformer backbone and guide the representation learning of the current task by directly selecting relevant cues from the cue pool. These methods significantly reduce the impact of catastrophic forgetting and allow the model to converge in very few iterations. However, the performance of such methods heavily depends on the selection strategy. As the task sequence lengthens, the number of corresponding cues increases, making the selection of the correct cue increasingly unstable. In addition, very low model iterations may lead to insufficient understanding of new tasks by specific task cues. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a continuous learning method based on two-level prompts.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A first aspect of the present invention provides a continuous learning method based on two-level prompts, comprising:

[0008] Inputting an input image to a CLIP image encoder with frozen parameters to generate an image embedding, and concatenating the image embedding with a prefix tag to form a concatenated image embedding; the prefix tag is obtained by adding a ratio of a trained language tag to a trained class tag, and the language tag and the class tag are trained using a pre-trained model VIT with frozen parameters, a CLIP image encoder with frozen parameters, and a CLIP text encoder with frozen parameters;

[0009] Associating the input text prompt corresponding to the input image with the category name to form the first-level prompt, and inputting the first-level prompt into the parameter-frozen CLIP text encoder to generate text embedding;

[0010] Calculating the similarity between the image embedding and the text embedding, and adjusting the similarity using an adaptive weight, thereby obtaining the largest K values ​​of the adjusted similarity scores to form a second-level prompt;

[0011] The spliced ​​image embedding and the second level prompt are input into the pre-trained model VIT, and the final embedding is output, and the final embedding is further subjected to a softmax operation to obtain the final prediction as the classification category label.

[0012] Furthermore, the input image is input to a CLIP image encoder with frozen parameters to generate an image embedding, specifically:

[0013] q=E vis (x)

[0014] Where q represents image embedding, E vis represents the CLIP image encoder with frozen parameters, x represents the input image;

[0015] Language tag i ∈[l1,l2,…,l D ] and class marker c i ∈[c1,c2,…,c D ], i represents the class code in the current task.

[0016] Furthermore, the input text prompt is associated with the category name to form a first-level prompt, including:

[0017] The learnable context vector p is the input text prompt and the category name y c ∈Y t Associated to form a text description of category c As a first level tip:

[0018]

[0019] where [p]s ∈R D , [CLS] c is the text embedding of the c-th class name, Y t represents the set of all categories in the t-th task; the first-level prompt is input into the parameter-frozen CLIP text encoder to generate text embedding, including:

[0020]

[0021] In the formula, k t represents text embedding, E txt represents the parameter-frozen CLIP text encoder, and t represents the task encoding.

[0022] Further, the calculating the similarity between the image embedding and the text embedding includes:

[0023] By matching query With key To calculate the similarity score to retrieve the second level hints:

[0024] S i,t = i,t ,k i,t >

[0025] where i represents the class encoding in the current task and 〈·,·〉 represents the cosine distance similarity; the similarity scores are then projected into the shared space S t ∈R N×D ;

[0026] The method of adjusting the similarity by using the adaptive weight to obtain the largest K values ​​of the adjusted similarity scores to form the second level prompt includes:

[0027]

[0028] Among them, TOP-K max Represents the operation of selecting the largest K values ​​from a given set of values, W∈R N×D Represents adaptive weight.

[0029] Further, the embedding of the spliced ​​image and the second level prompt are input into the pre-trained model VIT, the final embedding is output, and the final embedding is further subjected to a softmax operation to obtain a final prediction as a classification category label, including:

[0030] The language tag corresponding to the language tag embedding is The class tag corresponding to the class tag is embedded as f θ represents the pre-trained model VIT; the final embedding ​It is obtained by adding the proportion of the trained language token embeddings and the trained class token embeddings, including:

[0031]

[0032] In the formula, α and β are hyperparameters used to adjust the integration process;

[0033] Final embedding Generate the final prediction through softmax:

[0034]

[0035] where φ is the classification head.

[0036] Furthermore, the training of the language tokens and class tokens includes: training the language tokens and class tokens based on the pre-trained model VIT with the total loss function and parameter freezing, performing backpropagation and updating the selected key and prompt sets; among them, the total loss function L total is:

[0037] L total = L O + L SD

[0038] In the formula, L O represents the regularization penalty loss function, specifically:

[0039]

[0040] In the formula, T i represents the current task, past(t) = {t'|t'∈T, t'<i}) represents the previously learned tasks, represents the second-level prompt obtained from the previous task training;

[0041] L SD represents the semantic distillation loss function, specifically:

[0042]

[0043] Among them, λ is the balance hyperparameter, represents the prediction result of the classification head, L CrossEntropy represents the cross-entropy loss, L KD represents the knowledge distillation loss, C i represents the learnable context vector p of the context information of the image.

[0044] The beneficial effects of the present invention are:

[0045] In an exemplary embodiment of the present invention, by using the text input of the CLIP model as the first-level prompt, and selecting the second-level prompt through the query-key matching mechanism (calculating the similarity between image embedding and text embedding, image embedding as query, and text embedding as key), the advantage is that the stability of the model is enhanced, that is, there is no need to select or merge parameters, CLIP only generates semantic embeddings for the current image and text without fine-tuning any parameters; on this basis, this exemplary embodiment introduces a sparse joint prompt input method with adaptive weights (that is, the adaptive weights adjust the similarity), that is, using sparse joint prompts as model input, without any buffer to store past instance samples, so that the method in this exemplary embodiment is suitable for scenarios sensitive to data privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A flowchart of a continuous learning method based on two-level prompts provided for an exemplary embodiment of the present invention;

[0047] Figure 2 A schematic diagram of an overall framework provided for an exemplary embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a two-level prompt architecture provided by an exemplary embodiment of the present invention.

[0049] Figure 4 A schematic diagram of the average accuracy (%) and the number of fine-tuning parameters of four data sets provided for an exemplary embodiment of the present invention when the number of incremental tasks is 10;

[0050] Figure 5 A schematic diagram of the forgetting rate (%) results of 4 data sets provided by an exemplary embodiment of the present invention when the number of incremental tasks is 10;

[0051] Figure 6 A schematic diagram of the results of CIFAR and ImageNet datasets after 5 tasks and 20 tasks, respectively, provided for an exemplary embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the results of CUB and DomainNet datasets after 5 tasks and 20 tasks, respectively, provided for an exemplary embodiment of the present invention;

[0053] Figure 8 A schematic diagram of the accuracy of correctly selecting cues and overall selection accuracy for a 1-st task when using different methods for incremental task training provided by an exemplary embodiment of the present invention;

[0054] Fig. 9Two-level prompt (TP), adaptive weight (AW) and semantic distillation module (L) provided for an exemplary embodiment of the present invention SD ) Schematic diagram of the average accuracy of all tasks combined and the TOP-1 accuracy of the new category in the last task;

[0055] Fig.10 An exemplary embodiment of the present invention provides a method for combining TP and a semantic distillation module L SD Schematic diagram of accuracy applied to L2p. DETAILED DESCRIPTION

[0056] The technical solution of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In the description of the present invention, it should be noted that the directions or positional relationships indicated by "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are directions or positional relationships based on the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0058] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0059] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0060] See also Figure 1 , Figure 1 A flowchart of a continuous learning method based on two-level prompts provided by an exemplary embodiment of the present invention is shown, comprising:

[0061] Inputting an input image to a CLIP image encoder with frozen parameters to generate an image embedding, and concatenating the image embedding with a prefix tag to form a concatenated image embedding; the prefix tag is obtained by adding a ratio of a trained language tag to a trained class tag, and the language tag and the class tag are trained using a pre-trained model VIT with frozen parameters, a CLIP image encoder with frozen parameters, and a CLIP text encoder with frozen parameters;

[0062] Associating the input text prompt corresponding to the input image with the category name to form the first-level prompt, and inputting the first-level prompt into the parameter-frozen CLIP text encoder to generate text embedding;

[0063] Calculating the similarity between the image embedding and the text embedding, and adjusting the similarity using an adaptive weight, thereby obtaining the largest K values ​​of the adjusted similarity scores to form a second-level prompt;

[0064] The spliced ​​image embedding and the second level prompt are input into the pre-trained model VIT, and the final embedding is output, and the final embedding is further subjected to a softmax operation to obtain the final prediction as the classification category label.

[0065] Specifically, in this exemplary embodiment, a novel two-level cue selection strategy is designed, combined with a set of adaptive weights for sparse joint tuning, and the model uses prefix tuning to adapt to downstream tasks; in order to address the inherent plasticity challenges in cue-based continuous learning (CL), the CLIP text encoder is used to extract the semantics of class names and images, aiming to improve the accuracy of cue selection:

[0066] By using the text input of the CLIP model as the first-level prompt and selecting the second-level prompt through the query-key matching mechanism (calculating the similarity between image embedding and text embedding, with image embedding as the query and text embedding as the key), the advantage is that the stability of the model is enhanced, that is, there is no need to select or merge parameters, CLIP only generates semantic embeddings for the current image and text without fine-tuning any parameters; on this basis, this exemplary embodiment introduces a sparse joint prompt input method with adaptive weights (that is, the adaptive weights adjust the similarity), that is, using sparse joint prompts as model input, without the need for any buffer to store past instance samples, so that the method in this exemplary embodiment is suitable for scenarios that are sensitive to data privacy.

[0067] The following content will elaborate on the specific exemplary embodiments of each step. Specifically, the goal of continuous learning is to acquire knowledge from a data stream consisting of T non-overlapping sequential data sets, represented as D = {D1, D2, ..., D t}. Each data set D tCorresponding to a specific task t, it can be represented as a set of data of a specific class, that is, Contains data samples and the corresponding where i represents the i-th class in task t. The goal is to train a mapping function f parameterized by θ θ :X→Y to process T incremental tasks. In the inference phase, f predicts the corresponding label y based on the task-agnostic sample x. Note that during the training phase of the current task, the data of the previous tasks cannot be accessed.

[0068] The following exemplary embodiment is a two-level prompt training method for the t-th task, which is constructed using Python language and PyTorch framework and consists of two parts. The overall framework diagram is as follows: Figure 2 As shown in the figure, the two-level prompt architecture is as follows Figure 3 shown.

[0069] Input: Current data stream Pre-trained CLIP model CLIP text encoder E txt and CLIP image encoder E vis , pre-trained ViTf θ , classification head φ, first level hint pool The second level prompt pool P=P1,…,P t}, language tag l = [l1,l2,…,l D ], category label c = [c1, c2, ..., c D ], adaptive weight set W, number of training cycles E for the t-th task, learning rate γ, and balance parameter λ.

[0070] freeze: P1,…,P t}, f θ , E txt and E vis .

[0071] Initialized using tokenized "XXXX[CLS]" For e=1,…,E, perform the following operations:

[0072] Initialize the selected key and prompt set: K B ={},P B ={}. For each (x i ,y i ) Perform the following steps, steps 1 to 4 as follows Figure 3 As shown:

[0073] 1. Combine the learnable context vector p (text prompt) with the category name y c ∈Y t (Y trepresents the set of all categories in the t-th task) to form a text description of the c-th category As a first level tip:

[0074]

[0075] where [p] i ∈R D , i∈{1,2,…,m}, [CLS] c is the text embedding of the name of class c.

[0076] 2. Take an image x∈R H×W×C and a text embedding Input to CLIP image encoder E vis and text encoder E txt , we get image embedding q = E vis (x) and text embedding

[0077] 3. Embedding by matching the image as query Embedded with text as key To calculate the similarity score to retrieve the second level hints:

[0078] S i,t = i,t ,k i,t >

[0079] where i represents the class encoding in the current task, t represents the task encoding, and <·,·> represents the cosine distance similarity. The similarity scores are then projected into the shared space S t ∈R N×D .

[0080] 4. Use adaptive weights W∈R N×D To adjust the correlation between tasks and prompts, and achieve joint sparse prompt adjustment. The final output is as follows:

[0081]

[0082] Among them, TOP-K max Represents the operation of selecting the largest K values ​​from a given set of values ​​(similarity * weight * second-level prompt pool, select the largest k value). In order to simplify the demonstration, the category information corresponding to each prompt is omitted, that is, in M represents the length of the prompt vector. The initial second-level prompt pool and the initial first-level prompt pool are calculated in the same process, that is, the initial second-level prompt pool is the same as the initial first-level prompt pool.​

[0083] Specifically, the first-level generation class prototypes are used as keys, and these keys are compared with the queries for similarity. Then, the optimal prompt is selected by combining learnable parameters. When fine-tuning the prompt for the current task in this exemplary embodiment, the prompt parameters from the previous task are frozen to prevent them from being affected by the current task.

[0084] 5. As Figure 2 shown, the selected prompt (i.e., the second-level prompt in step 4 ) and the image embedding are fed into the pre-trained Vision Transformer (ViT) model f θ . Prefix tuning is adopted to adapt to the downstream task. To capture the text semantic information of the class name, a language token l i ∈ [l1, l2, …, l D and a class token c i ∈ [c1, c2, …, c D are created, and the language token and the class token are used as the prefix of the image embedding in the form of concatenation.

[0085] After that, the language token embedding and the class token embedding are added together to form a new embedding:

[0086]

[0087] where i represents the class encoding in the current task, and α and β are hyperparameters used to regulate the integration process.

[0088] 6. The final embedding generates the final prediction through softmax:

[0089]

[0090] where φ is the classification head, and the final prediction is the classification class label.

[0091] 7. Regularization penalty is used to prevent the potential similarity between the new and old prompts, which provides more diversity and less homogeneity for the prompts:

[0092]

[0093] where T i represents the current task, and past(t) = {t'|t' ∈ T, t' < i}) represents the previously learned tasks.

[0094] To enhance the discriminative ability of the language token l i , a new loss function called semantic distillation L SDThe text encoder of CLIP is used to extract the embedding features of the category name and the context information of the image C i ∈R D A simple distillation function is used to extract the high-level information of the image from the semantic feature space into language tag features:

[0095]

[0096] where λ is a balancing hyperparameter, Represents the prediction result of the classification head. L CrossEntropy represents the cross entropy loss, L KD represents the knowledge distillation loss, C i Represents the context information of the image.

[0097] Calculate the total loss: L total =L O +L SD .

[0098] 8. Update the selected key and prompt collection:

[0099] 9. Perform back propagation, using renew P t , W, c, l, φ.

[0100] In order to compare the performance between the proposed method (TIPS) of this exemplary embodiment and the recent SOTA (state-of-the-art), detailed reports are provided for each dataset (CIFAR, ImageNet-R, CUB, and DomainNet) under three incremental scenarios (5 tasks, 10 tasks, and 20 tasks), as shown in Figures 4 to 7 As shown. Among them:

[0101] Figure 4 The average accuracy (%) and the number of fine-tuning parameters for 4 datasets with 10 incremental tasks are shown, where DomainNet is a cross-domain dataset. Methods are grouped according to buffer size, where 0 means no replay is required, * means results taken directly from the original paper, - means experiments that could not be completed, and the rest of the experiments were conducted using the code provided by the original paper. Experiments were conducted here using three random seeds: 1993, 1997, and 1999.

[0102] Figure 5 The results of forgetting rate (%) for 4 datasets with an incremental number of tasks of 10 are shown, where lower values ​​indicate better performance.

[0103] Figure 6are the results after 5 and 20 tasks for CIFAR and ImageNet datasets, respectively. AVG represents the overall average accuracy after completing the incremental tasks, while Last represents the average accuracy of the new and old categories in the last task. This exemplary embodiment was experimented with a random seed from 1993.

[0104] Figure 7 are the results after 5 tasks and 20 tasks for the CUB and DomainNet datasets; AVG represents the overall average accuracy after completing the incremental tasks, and Last represents the average accuracy of the new and old categories in the last task.

[0105] For the experimental results:

[0106] CIFAR100: In the 10-task scenario, TIPS has an overall average accuracy that is 2% higher than ESN. However, for longer (20 tasks) and shorter (5 tasks) task sequences, the performance of this exemplary embodiment is close to SOTA. Compared with AttriCLIP, which also uses CLIP assistance, TIPS shows better stability on simpler datasets.

[0107] ImageNet-R: This dataset contains multiple domains and classes, which brings greater challenges and requirements to the model. It is worth noting that the performance of this exemplary embodiment exceeds the SOTA in all three scenarios. It is believed that this is due to the powerful cross-domain recognition ability of CLIP.

[0108] CUB: The distribution of this dataset is significantly different from the pre-training dataset. Therefore, it effectively tests the generalization ability of the model to downstream tasks. Although the performance of this exemplary embodiment on this dataset is slightly insufficient, it is still close to SOTA and shows better anti-forgetting ability. For example, in the scenario of 20 tasks, the best-performing Dualprompt has a sharp drop in accuracy after the 13th incremental session, while TIPS maintains a stable accuracy and even increases slightly.

[0109] DomainNet: This is a domain incremental dataset where TIPS demonstrates strong domain generalization capabilities, surpassing SOTA performance in various scenarios. It is worth noting that this dataset is more challenging than ImageNet-R, causing all methods to experience a rapid drop in accuracy after starting incremental learning. However, TIPS shows less forgetting than other methods, demonstrating its robustness in dealing with this difficulty.

[0110] In summary, the results of various methods of this exemplary embodiment on four datasets (CIFAR, CUB, ImageNet-R and Domainnet) are shown in Figure 4, Figure 5 , Figure 6 and Figure 7 In the examples, the method performs well on the four datasets compared to the most recent state-of-the-art methods without replaying images of old categories. In addition, the method of the example embodiment is also competitive in terms of the number of parameters to be adjusted.

[0111] Figure 8 (a) Comparison of the accuracy of correctly selecting cues for the 1-st task when using different methods for incremental task training. Even after multiple incremental learning tasks, the method of the present exemplary embodiment is still able to generate correct cues for 1-st task queries. (b) In the first two figures, the present exemplary embodiment extends the prompt selection test across incremental learning rounds, showing the overall selection accuracy for each task and comparing it with the baseline method. The results show that the method of the present exemplary embodiment maintains stable prompt selection accuracy across all learning rounds. In the last two figures, the present exemplary embodiment compares the accuracy of each method on new and old categories across different incremental learning rounds. The method of the present exemplary embodiment maintains a leading edge.

[0112] Regarding steps 7 to 9, a semantic distillation module is designed to enhance the generalization ability of unknown new categories by creating a language token and utilizing the encapsulated semantic information of the category name. This module effectively utilizes the semantic information of the category name and enhances the model's understanding of new category knowledge, thereby improving the model's plasticity. Experiments have shown that this method can maintain or exceed the performance of state-of-the-art methods in various data sets and incremental learning scenarios, and can maintain stable prompt selection accuracy after multiple rounds of incremental learning.

[0113] Fig. 9 The ablation experiment of this exemplary embodiment includes three parts: two-level prompt (TP), adaptive weight (AW) and semantic distillation module (L SD ). AVG represents the average accuracy of all tasks, and Last-T represents the TOP-1 accuracy of the new category in the last task.

[0114] Fig.10 It is to combine TP and semantic distillation module L SD Applied to L2p. The experiments are conducted on ImageNet-R, which contains 10 tasks. AVG represents the average accuracy of all tasks, while Last-T represents the accuracy when learning new categories in the last task.

[0115] Obviously, the above embodiments are merely examples for clear explanation, and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A continuous learning method based on two-level prompts, characterized by: include: Inputting an input image to a CLIP image encoder with frozen parameters to generate an image embedding, and concatenating the image embedding with a prefix tag to form a concatenated image embedding; the prefix tag is obtained by adding a ratio of a trained language tag to a trained class tag, and the language tag and the class tag are trained using a pre-trained model VIT with frozen parameters, a CLIP image encoder with frozen parameters, and a CLIP text encoder with frozen parameters; Associating the input text prompt corresponding to the input image with the category name to form the first-level prompt, and inputting the first-level prompt into the parameter-frozen CLIP text encoder to generate text embedding; Calculating the similarity between the image embedding and the text embedding, and adjusting the similarity using an adaptive weight, thereby obtaining the largest K values ​​of the adjusted similarity scores to form a second-level prompt; The spliced ​​image embedding and the second level prompt are input into the pre-trained model VIT, and the final embedding is output, and the final embedding is further subjected to a softmax operation to obtain the final prediction as the classification category label.

2. The continuous learning method based on two-level prompts according to claim 1 is characterized in that: The input image is input to the parameter-frozen CLIP image encoder to generate image embedding, specifically: q=E vis (x) Where q represents image embedding, E vis represents the CLIP image encoder with frozen parameters, x represents the input image; Language tag i ∈[l1,l2,…,l D ] and class marker c i ∈[c1,c2,…,c D ], i represents the class code in the current task.

3. The continuous learning method based on two-level prompts according to claim 2 is characterized in that: Associating the input text prompt with the category name forms the first level prompt, including: The learnable context vector p is the input text prompt and the category name y c ∈Y t Associated to form a text description of category c As a first level tip: where [p] s ∈R D , [CLS] c is the text embedding of the c-th class name, Y t represents the set of all categories in the tth task; The first level of prompts are fed into the parameter-frozen CLIP text encoder to generate text embeddings, including: In the formula, k t represents text embedding, E txt represents the parameter-frozen CLIP text encoder, and t represents the task encoding.

4. The continuous learning method based on two-level prompts according to claim 3 is characterized in that: The calculating the similarity between the image embedding and the text embedding comprises: By matching query With key To calculate the similarity score to retrieve the second level hints: S i,t = i,t ,k i,t >​ where i represents the class encoding in the current task and 〈·,·〉 represents the cosine distance similarity; the similarity scores are then projected into the shared space S t ∈R N×D ; The method of adjusting the similarity by using the adaptive weight to obtain the largest K values ​​of the adjusted similarity scores to form the second level prompt includes: Among them, TOP-K max Represents the operation of selecting the largest K values ​​from a given set of values, W∈R N×D Represents adaptive weight.

5. The continuous learning method based on two-level prompts according to claim 4 is characterized in that: The embedding of the spliced ​​image and the second level prompt are input into the pre-trained model VIT, the final embedding is output, and the final embedding is further subjected to a softmax operation to obtain a final prediction as a classification category label, including: The language tag corresponding to the language tag embedding is The class tag corresponding to the class tag is embedded as f θ represents the pre-trained model VIT; the final embedding It is obtained by adding the ratio of the trained language tag embedding to the trained class tag embedding, including: Where α and β are hyperparameters used to adjust the integration process; Final Embed Generate the final prediction through softmax: where φ is the classification head.

6. The continuous learning method based on two-level prompts according to claim 5 is characterized in that: The training of language tags and class tags includes: training the language tags and class tags based on the pre-trained model VIT with total loss function and parameter freezing, performing back propagation and updating the selected key and prompt sets; wherein the total loss function L total for: L total =L O +L SD Where, L O represents the regularized penalty loss function, specifically: where, T i represents the current task, and past(t) = {t'|t'∈T, t'<i} represents the tasks learned previously, represents the second-level hint obtained from the training of the previous round of tasks; L SD represents the semantic distillation loss function, specifically: Among them, λ is the balancing hyperparameter, represents the prediction result of the classification head, L CrossEntropy represents the cross entropy loss, L KD represents the knowledge distillation loss, C i The context information representing the image is the learnable context vector p.

Images