Two-stage hint-based continual learning method

Abstract

The application discloses a two-stage prompt-based continuous learning method, which comprises the following steps: inputting an input image into a CLIP image encoder with frozen parameters to generate an image embedding, and splicing the image embedding with a prefix label to form a spliced image embedding; associating input text prompts corresponding to the input image with category names to form a first-stage prompt, and inputting the first-stage prompt into a CLIP text encoder with frozen parameters to generate a text embedding; calculating the similarity between the image embedding and the text embedding, adjusting the similarity by using an adaptive weight, and obtaining the maximum K values in the adjusted similarity scores to form a second-stage prompt; inputting the spliced image embedding and the second-stage prompt into a pre-trained model VIT, outputting a final embedding, and performing a softmax operation on the final embedding to obtain a final prediction. The application does not need to fine-tune any parameters, does not need any buffer to store past instance samples, and is suitable for data privacy sensitive scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of continual learning of artificial intelligence, and in particular to a two-level prompt-based continual learning method. BACKGROUND

[0002] Continual learning is capable of learning on non-identically distributed data streams. The characteristic of continual learning is that the model continuously learns incrementally as the training data stream flows in, and eventually completes the training on all tasks. In this process, the model can not only utilize the new data coming in, but also based on the experience accumulated before, complete the task well on the new data; it can also avoid the problem of forgetting, and still maintain high accuracy on the tasks that have been trained before. The main challenge is called catastrophic forgetting, which is caused by the adaptation to the new distribution, which usually leads to a significant reduction in the ability to capture the old distribution.

[0003] In recent years, the field of continual learning has developed rapidly, covering traditional continual learning paradigms and prompt-based continual learning methods. Traditional training from scratch methods aim to prevent knowledge forgetting by protecting important past parameters. Elastic weight consolidation preserves learned knowledge by applying regularization constraints in the loss function of new tasks to prevent new information from interfering. In addition, dynamic network architecture prevents modifying important parameters by isolating previously learned parameters and allocating new parameters for new tasks. Most of these methods focus on solving the forgetting problem (stability), while ignoring the model's rapid generalization ability (plasticity), which requires multiple iterations to converge, bringing great challenges in scenarios that require rapid identification of new targets.

[0004] Pre-trained models trained on large-scale data introduce a new research direction for continual learning. Inspired by prompt learning, these methods freeze the pre-trained Transformer backbone and guide the representation learning of the current task by directly selecting relevant prompts from the prompt pool. These methods significantly reduce the impact of catastrophic forgetting and allow the model to converge in very few iterations. However, the performance of this method is heavily dependent on the selection strategy. As the task sequence lengthens, the number of corresponding prompts increases, making it increasingly unstable to select the correct prompt. In addition, very low model iteration numbers can lead to insufficient understanding of specific task prompts for new tasks. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a two-level prompt-based continual learning method.

[0006] The purpose of the present application is achieved by the following technical solutions:

[0007] In a first aspect of the present application, a two-level prompt-based continual learning method is provided, comprising:

[0008] Input an input image to a CLIP image encoder with frozen parameters to generate an image embedding, and concatenate the image embedding with a prefix tag to form a concatenated image embedding; the prefix tag is obtained by adding the ratio of trained language tags to trained class tags, and the language tags and class tags are trained using a pre-trained model VIT with frozen parameters, a CLIP image encoder with frozen parameters, and a CLIP text encoder with frozen parameters;

[0009] Associating the input text prompt corresponding to the input image with the category name to form the first-level prompt, and inputting the first-level prompt into the parameter-frozen CLIP text encoder to generate text embedding;

[0010] Calculating the similarity between the image embedding and the text embedding, and adjusting the similarity using adaptive weights to obtain the largest K values ​​of the adjusted similarity scores to form a second-level prompt;

[0011] The spliced ​​image embedding and the second level prompt are input into the pre-trained model VIT, and the final embedding is output, and the final embedding is further subjected to a softmax operation to obtain the final prediction as the classification category label.

[0012] Furthermore, the input image is input to a parameter-frozen CLIP image encoder to generate an image embedding, specifically:

[0013] q=E vis (x)

[0014] Where q represents image embedding, E vis represents the CLIP image encoder with frozen parameters, and x represents the input image;

[0015] Language tag i ∈[l1,l2,…,l D ] and class marker c i ∈[c1,c2,…,c D ], i represents the class code in the current task.

[0016] Furthermore, the input text prompt is associated with the category name to form a first-level prompt, including:

[0017] The learnable context vector p is the input text prompt and the category name y c ∈Y t Associate to form a text description of category c As a first level tip:

[0018]

[0019] where [p]s ∈R D , [CLS] c is the text embedding of the c-th class name, Y t denotes the set of all classes in the t-th task; the first-level prompt is input into the parameter-frozen CLIP text encoder to generate a text embedding, including:

[0020]

[0021] where k t denotes the text embedding, E txt denotes the parameter-frozen CLIP text encoder, and t denotes the task code.

[0022] Further, the similarity between the image embedding and the text embedding is calculated, including:

[0023] The similarity score is calculated by matching the query with the key to retrieve the second-level prompt:

[0024] S i,t = <q i,t , k i,t >

[0025] where i represents the class code in the current task, and <·,·> represents the cosine distance similarity; the similarity score is then projected into a shared space S t ∈R N×D ;

[0026] The similarity is adjusted using adaptive weights to obtain the maximum K values of the adjusted similarity score to form the second-level prompt, including:

[0027]

[0028] where TOP-K max denotes the operation of selecting the maximum K values from a given set of values, and W ∈ R N×D denotes the adaptive weight.

[0029] Further, the spliced image embedding and the second-level prompt are input into the pre-trained model VIT to output the final embedding, and the final embedding is further subjected to a softmax operation to obtain the final prediction as the classification class label, including:

[0030] The language token embedding corresponding to the language token is The class token embedding corresponding to the class token is f θ denotes the pre-trained model VIT; and the final embedding The proportion of the trained language token embedding and the trained class token embedding is added to obtain, including:

[0031]

[0032] In the formula, alpha and beta are hyperparameters, used to adjust the integration process;

[0033] Final embedding The final prediction is generated by softmax:

[0034]

[0035] Where phi is the classification head.

[0036] Further, the training of the language token and the class token includes: based on the total loss function and the pre-trained model VIT of the parameter freezing, training the language token and the class token, performing back propagation and updating the selected key and prompt set; wherein, the total loss function L total is:

[0037] L total = L O + L SD

[0038] In the formula, L O represents a regularization penalty loss function, specifically:

[0039]

[0040] In the formula, T i represents the current task, past(t)={t'|t'∈T,t'<i}) represents the previously learned task, represents the second-level prompt obtained by the previous round of task training;

[0041] L SD represents a semantic distillation loss function, specifically:

[0042]

[0043] Where lambda is a balance hyperparameter, represents the prediction result of the classification head, L CrossEntropy represents a cross-entropy loss, L KD represents a knowledge distillation loss, C i represents the context vector p that can learn the context information of the image.

[0044] The beneficial effects of the present application are:

[0045] In an example embodiment of the present application, by using the text input of the CLIP model as the first-level prompt and selecting the second-level prompt through a query-key matching mechanism (calculating the similarity of image embeddings and text embeddings, the image embeddings being the query and the text embeddings being the key), the advantage lies in enhancing the stability of the model, i.e., no need to select or merge parameters, the CLIP only generates semantic embeddings for the current image and text without fine-tuning any parameters; on this basis, the example embodiment introduces a sparse joint prompt input method with adaptive weights (i.e., the adaptive weights adjust the similarity), i.e., using sparse joint prompts as model input, without any buffer to store past instance samples, making the method in the example embodiment applicable to data privacy sensitive scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 Flowchart of the two-level prompt-based continual learning method provided by an example embodiment of the present application;

[0047] Figure 2 Overall framework schematic diagram provided by an example embodiment of the present application;

[0048] Figure 3 Two-level prompt architecture schematic diagram provided by an example embodiment of the present application.

[0049] Figure 4 Schematic diagram of the average accuracy (%) and the number of fine-tuned parameters of four data sets in the case of 10 incremental tasks provided by an example embodiment of the present application;

[0050] Figure 5 Schematic diagram of the results of the forgetting rate (%) of four data sets in the case of 10 incremental tasks provided by an example embodiment of the present application;

[0051] Figure 6 Schematic diagram of the results after 5 tasks and 20 tasks for CIFAR and ImageNet data sets provided by an example embodiment of the present application;

[0052] Figure 7 Schematic diagram of the results after 5 tasks and 20 tasks for CUB and DomainNet data sets provided by an example embodiment of the present application;

[0053] Figure 8 Schematic diagram of the accuracy of correctly selecting prompts for the 1-st task and the overall selection accuracy when using different methods to train incremental tasks provided by an example embodiment of the present application;

[0054] Figure 9The two-stage prompt (TP), the adaptive weight (AW) and the semantic distillation module (L SD An illustration of the average accuracy of all tasks and the TOP-1 accuracy of new classes in the last task in combination with the two-stage prompt (TP), the adaptive weight (AW) and the semantic distillation module (L

[0055] Figure 10 The TP and the semantic distillation module (L SD An illustration of the accuracy applied to L2p. DETAILED DESCRIPTION

[0056] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0057] In the description of the present application, it should be noted that the directions or positional relationships indicated by “center”, “upper”, “lower”, “left”, “right”, “vertical”, “horizontal”, “inner”, “outer” and the like are described based on the directions or positional relationships described in the drawings, and are only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the devices or elements indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, “first” and “second” are only for the purpose of description and cannot be understood as indicating or implying relative importance.

[0058] In the description of the present application, it should be noted that, unless otherwise explicitly specified and limited, “mounting”, “connection” and “connection” should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, and can be the communication between the two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0059] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0060] Referring to Figure 1 , Figure 1 The flowchart of the two-stage prompt-based continual learning method provided by an exemplary embodiment of the present application is shown, which includes:

[0061] inputting an input image into a CLIP image encoder with frozen parameters to generate an image embedding, and splicing the image embedding with a prefix label to form a spliced image embedding; the prefix label is obtained by adding a language label and a class label trained by a pre-trained model VIT with frozen parameters, a CLIP image encoder with frozen parameters, and a CLIP text encoder with frozen parameters;

[0062] associating an input text prompt corresponding to an input image with a category name to form a first-level prompt, and inputting the first-level prompt into a CLIP text encoder with frozen parameters to generate a text embedding;

[0063] calculating a similarity between the image embedding and the text embedding, and adjusting the similarity by using adaptive weights to obtain an adjusted similarity score, and obtaining K maximum values from the adjusted similarity score to form a second-level prompt;

[0064] inputting the spliced image embedding and the second-level prompt into a pre-trained model VIT to output a final embedding, and further performing a softmax operation on the final embedding to obtain a final prediction as a classification category label.

[0065] Specifically, in the present exemplary embodiment, a novel two-level prompt selection strategy is designed, and a set of adaptive weights for sparse joint tuning is combined, and the model uses prefix tuning to adapt to downstream tasks; in order to cope with the inherent plasticity challenge in prompt-based continual learning (CL), the CLIP text encoder is used to extract the semantics of the class name and the image, aiming to improve the accuracy of prompt selection:

[0066] By using the text input of the CLIP model as the first-level prompt, and by using the query-key matching mechanism (calculating the similarity between the image embedding and the text embedding, the image embedding being the query and the text embedding being the key) to select the second-level prompt, the advantage is to enhance the stability of the model, that is, no parameters need to be selected or merged, and the CLIP only generates semantic embeddings for the current image and text without the need to fine-tune any parameters; on this basis, the present exemplary embodiment introduces a sparse joint prompt input method with adaptive weights (i.e., the similarity is adjusted by using adaptive weights), that is, the sparse joint prompt is used as the model input, and no buffer is needed to store past instance samples, so that the method in the present exemplary embodiment is applicable to data privacy sensitive scenarios.

[0067] The following will elaborate on the specific exemplary embodiments of each step in detail. Specifically, the goal of continual learning is to acquire knowledge from a data stream composed of T non-overlapping sequential data sets, denoted as D={D1, D2, …, DT}. t} Each data set D tCorresponding to a specific task t, a set of data can be represented as a specific class, i.e. Contains data samples And the corresponding Where i represents the i-th class in task t. The goal is to train a mapping function f parameterized by θ θ :X→Yto handle T incremental tasks. In the inference phase, f predicts the corresponding label y based on the task-agnostic sample x. Note that the data of previous tasks cannot be accessed during the training phase of the current task.

[0068] The following exemplary embodiment is a two-stage prompt training method for the t-th task, which is built by Python language and PyTorch framework, and consists of two parts. The overall framework diagram is shown in Figure 2 , and the two-stage prompt architecture is shown in Figure 3 .

[0069] Input: current data stream Pre-trained CLIP model CLIP text encoder E txt And CLIP image encoder E vis , pre-trained ViT f θ , classification head φ, first-stage prompt pool Second-stage prompt pool P=P1,…,P t}, language label l=[l1,l2,…,l D ], category label c=[c1,c2,…,c D ], adaptive weight set W, training period number E of the t-th task, learning rate γ, balance parameter λ.

[0070] Freeze: P1,…,P t}, f θ , E txt And E vis .

[0071] Initialize with tokenized "XXXX[CLS]" Perform the following operations for e=1,…,E:

[0072] Initialize the selected key and prompt set: K B = {}, P B = {}. Perform the following steps for each (x i , y i ), steps 1-4 are shown in Figure 3 :

[0073] 1. The learnable context vector p (text prompt) is combined with the category name y c ∈Y t (Y trepresenting the set of all classes in the t-th task) association, forming the text description of the c-th class As the first-level prompt:

[0074]

[0075] where [p] i ∈R D , i∈{1,2,…,m}, [CLS] c is the text embedding of the c-th class name.

[0076] 2. Input an image x∈R H×W×C and a text embedding into the CLIP image encoder E vis and text encoder E txt respectively, to obtain the image embedding q=E vis (x) and the text embedding

[0077] 3. Calculate the similarity score by matching the image embedding as the query with the text embedding as the key to retrieve the second-level prompt:

[0078] S i,t =<q i,t ,k i,t >

[0079] where i represents the class encoding in the current task, t represents the task encoding, and <·,·> represents the cosine distance similarity. The similarity score is then projected into the shared space S t ∈R N×D .

[0080] 4. Use adaptive weights W∈R N×D to adjust the relevance between tasks and prompts to achieve joint sparse prompt tuning. The final output of the second-level prompt is as follows:

[0081]

[0082] where TOP-K max represents the operation of selecting the largest K values from a given set of values (similarity * weight * second-level prompt pool, select the largest k values). To simplify the demonstration, the category information corresponding to each prompt is omitted, that is, where M represents the length of the prompt vector. The initial second-level prompt pool and the initial first-level prompt pool are the same calculation process, that is, the initial second-level prompt pool is the same as the initial first-level prompt pool.

[0083] Specifically, the first-level generation class prototypes are used as keys, and these keys are compared with the query for similarity. Then, the optimal prompt is selected by combining learnable parameters. When fine-tuning the prompt for the current task in this exemplary embodiment, the prompt parameters from the previous task are frozen to prevent them from being affected by the current task.

[0084] 5. As Figure 2 shown, the selected prompt (i.e., the second-level prompt in step 4 ) and the image embedding are fed into the pre-trained Vision Transformer (ViT) model f θ and prefix tuning is adopted to adapt to downstream tasks. To capture the text semantic information of the class name, a language token l i ∈[l1, l2, …, l D and a class token c i ∈[c1, c2, …, c D are created, and the language token and the class token are used as the prefix of the image embedding in the form of concatenation.

[0085] After that, the language token embedding and the class token embedding are added together to form a new embedding:

[0086]

[0087] where i represents the class encoding in the current task, and α and β are hyperparameters used to adjust the integration process.

[0088] 6. The final embedding generates the final prediction through softmax:

[0089]

[0090] where φ is the classification head and the final prediction is the classification class label.

[0091] 7. Regularization penalty is used to prevent the potential similarity between the new and old prompts, which provides more diversity and less homogeneity for the prompts:

[0092]

[0093] where T i represents the current task, and past(t) = {t'|t' ∈ T, t' < i} represents the previously learned tasks.

[0094] To enhance the discrimination ability of the language token l i , a new loss function called semantic distillation L SDThe text encoder of CLIP is used to extract the embedding features of the class name and the context information C of the image i ∈R D A simple distillation function is used to extract the high-level information of the image from the semantic feature space to the language token feature:

[0095]

[0096] where λ is a balance hyperparameter, represents the prediction result of the classification head. L CrossEntropy represents the cross-entropy loss, L KD represents the knowledge distillation loss, C i represents the context information of the image.

[0097] Calculate the total loss: L total = L O + L SD .

[0098] 8. Update the selected key and prompt set:

[0099] 9. Perform backpropagation, using update P t , W, c, l, φ.

[0100] To compare the performance between the method proposed in this exemplary embodiment (TIPS) and the recent SOTA (State-of-the-Art), detailed reports are provided for each dataset (CIFAR, ImageNet-R, CUB, and DomainNet) under three incremental scenarios (5 tasks, 10 tasks, 20 tasks), as shown in Figures 4 to 7 . Among them:

[0101] Figure 4 The average accuracy (%) and the number of fine-tuning parameters of the 4 datasets are shown when the number of incremental tasks is 10, where DomainNet is a cross-domain dataset. The method is grouped according to the buffer size, where 0 means no need to replay, * means the results directly obtained from the original paper, - means the experiment cannot be completed, and the rest of the experiments are performed using the code provided in the original paper. Here, three random seeds are used: 1993, 1997, and 1999 for experiments.

[0102] Figure 5 The results of the forgetting rate (%) of the 4 datasets are shown when the number of incremental tasks is 10, where a lower value indicates better performance.

[0103] Figure 6are the results on CIFAR and ImageNet datasets after 5 and 20 tasks, respectively. AVG represents the overall average accuracy after completing the incremental tasks, while Last represents the average accuracy of new and old classes in the last task. This example embodiment uses a random seed of 1993 for the experiment.

[0104] Figure 7 are the results on CUB and DomainNet datasets after 5 and 20 tasks. AVG represents the overall average accuracy after completing the incremental tasks, while Last represents the average accuracy of new and old classes in the last task.

[0105] For the experimental results:

[0106] CIFAR100: In the scenario of 10 tasks, the overall average accuracy of TIPS is 2% higher than that of ESN. However, for longer (20 tasks) and shorter (5 tasks) task sequences, the performance of this example embodiment is close to SOTA. Compared with AttriCLIP, which also uses CLIP assistance, TIPS shows better stability on simpler datasets.

[0107] ImageNet-R: This dataset contains multiple domains and classes, posing greater challenges and requirements for the model. Notably, in all three scenarios, the performance of this example embodiment exceeds SOTA. It is believed that this is due to the strong cross-domain recognition ability of CLIP.

[0108] CUB: The distribution of this dataset is significantly different from the pre-training dataset. Therefore, it effectively tests the model's generalization ability to downstream tasks. Although this example embodiment performs slightly less on this dataset, it is still close to SOTA and shows better anti-forgetting ability. For example, in the scenario of 20 tasks, the best-performing Dualprompt has a sharp drop in accuracy after the 13th incremental session, while TIPS maintains stable accuracy and even slightly increases.

[0109] DomainNet: This is a domain-incremental dataset, and TIPS demonstrates strong domain generalization ability, outperforming SOTA in various scenarios. Notably, this dataset is more challenging than ImageNet-R, causing all methods to have a rapid drop in accuracy after starting incremental learning. However, TIPS shows less forgetting compared to other methods, demonstrating its robustness in dealing with this difficulty.

[0110] In summary, the results of various methods on four datasets (CIFAR, CUB, ImageNet-R, and Domainnet) of this example embodiment are shown in Figure 4, Figure 5 , Figure 6 and Figure 7 , respectively, for 10, 10, 5, and 20 tasks. Overall, the method of the present example embodiment outperforms the state-of-the-art methods on these four datasets without replaying old class images. Moreover, the method of the present example embodiment is also competitive in terms of the number of adjusted parameters.

[0111] Figure 8 is (a) the accuracy of correctly selecting hints for the 1-st task when training on incremental tasks using different methods. Even after multiple incremental learning tasks, the method of the present example embodiment is still able to generate correct hints for the 1-st task query. (b) In the first two figures, the present example embodiment extends the hint selection test for incremental learning rounds, showing the overall selection accuracy for each task and comparing with baseline methods. The results show that the method of the present example embodiment maintains stable hint selection accuracy across all learning rounds. In the last two figures, the present example embodiment compares the accuracy of new and old classes for different incremental learning rounds for each method. The method of the present example embodiment maintains a leading advantage.

[0112] Regarding steps 7-9, a semantic distillation module is designed to enhance the generalization ability for unknown new classes by creating a language token and utilizing the encapsulated semantic information of class names. This module effectively utilizes the semantic information of class names to enhance the model's understanding of new class knowledge, thereby improving the model's plasticity. Experiments show that this method can maintain or surpass the performance of state-of-the-art methods in various datasets and incremental learning scenarios, and still maintain stable hint selection accuracy after multiple rounds of incremental learning.

[0113] Figure 9 is an ablation experiment of the present example embodiment, including three parts: two-stage prompt (TP), adaptive weight (AW), and semantic distillation module (L SD ). AVG represents the average accuracy of all tasks, and Last-T represents the TOP-1 accuracy of new classes in the last task.

[0114] Figure 10 is the application of TP and semantic distillation module L SD to L2p. The experiment is conducted on ImageNet-R, which contains 10 tasks. AVG represents the average accuracy of all tasks, and Last-T represents the accuracy when learning new classes in the last task.

[0115] Obviously, the above embodiments are merely example for clearly illustrating but not limitation to the embodiments, and on the basis of the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments need not and can not be enumerated. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A method for continual learning based on two-stage prompting, characterized in that: The method comprises the following steps: inputting an input image into a CLIP image encoder with frozen parameters to generate an image embedding; associating an input text prompt corresponding to the input image with a category name to form a first-level prompt, and inputting the first-level prompt into a CLIP text encoder with frozen parameters to generate a text embedding; calculating the similarity between the image embedding and the text embedding, adjusting the similarity by using an adaptive weight to obtain the maximum K values in the adjusted similarity score to form a second-level prompt; splicing the image embedding with a prefix mark to form a spliced image embedding; the prefix mark is obtained by adding the proportion of a trained language mark and a trained category mark, and the language mark and the category mark are trained by using a pre-trained model VIT with frozen parameters, a CLIP image encoder with frozen parameters and a CLIP text encoder with frozen parameters; inputting the spliced image embedding and the second-level prompt into the pre-trained model VIT to output a final embedding, and further performing a softmax operation on the final embedding to obtain a final prediction as a classification category label; associating the input text prompt with the category name to form the first-level prompt, comprising: The learnable context vector p, i.e. the input text prompt, is associated with the category name ∈ to form a text description of the c-th category as a first level prompt: ； wherein ∈ , is a text embedding of the c-th name, denotes the set of all classes in the t-th task; inputting the first-level prompt into the CLIP text encoder with frozen parameters to generate the text embedding, comprising: ； wherein k t represents a text embedding, represents a CLIP text encoder with parameter freezing, and t represents a task encoding; the calculation of the similarity between the image embedding and the text embedding, comprising: By matching the query With the key To compute a similarity score to retrieve a second-level hint: ； where i represents the class code in the current task, denotes cosine distance similarity; the similarity score is then projected to the shared space ; the adjustment of the similarity by using the adaptive weight to obtain the maximum K values in the adjusted similarity score to form the second-level prompt, comprising: ； wherein represents an operation of selecting the largest K values from a given set of values, represents an adaptive weight.

2. The method of claim 1, wherein: the inputting of the input image into the CLIP image encoder with frozen parameters to generate the image embedding, specifically: ； In the formula, q represents an image embedding, a CLIP image encoder indicating parameter freezing, x represents an input image; Language tag And class tag i represents the class encoding in the current task.

3. The method of claim 1, wherein: the inputting of the spliced image embedding and the second-level prompt into the pre-trained model VIT to output the final embedding, and further performing the softmax operation on the final embedding to obtain the final prediction as the classification category label, comprising: The language mark corresponding language mark embedding is The class mark corresponding class mark embedding is , The pre-training model VIT represents; the prefix mark is the sum of the trained language mark embedding and the trained class mark embedding ratio, comprising: ； wherein, α and β are hyperparameters, used for adjusting the integration process; the final embedding generates the final prediction through the softmax: ； wherein is a classification header.

4. The method of claim 3, wherein: The training of the language tag and the class tag comprises: training the language tag and the class tag based on a total loss function and a pre-trained model VIT with parameter freezing, performing back propagation and updating the selected key and prompt set; wherein the total loss function is: Ltotal = Lclass + Llang. ； In the formula, represents a regularization penalty loss function, specifically: ； In the formula, represents a current task, represents a previously learned task, represents a second-level prompt obtained by training a previous round of tasks. denotes the semantic distillation loss function, in particular: ； where λ is a balanced hyper-parameter, denotes the prediction result of the classification head, denotes the cross-entropy loss, denotes the knowledge distillation loss, denotes the context information of the image, i.e., the learnable context vector p.

Images