Class incremental learning method, system and device based on pre-training model and medium

Abstract

The invention discloses a class incremental learning method, system and device based on a pre-training model and a medium. The method comprises the following steps: step 1, in class incremental learning of a first stage, aiming at a first task D1, finely adjusting a pre-training model ViT through a Prompt embedding method; obtaining a pre-training model after fine tuning; 2, combining and freezing the embedding layer function in the fine-tuning pre-training model obtained in the step 1 and the embedding layer function of the pre-training model ViT before fine tuning to obtain a combined model; step 3, performing class increment training from the beginning by using the combined model obtained in the step 2, and finally obtaining a class increment model with strong generalization and robustness; and 4, predicting a test sample by using the class increment model obtained in the step 3, and calculating the final classification precision. According to the method, the generalization ability of the model is kept while the adaptability of the model is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of class incremental learning, and specifically relates to a class incremental learning method, system, device and medium based on a pre-training model. Background Art

[0002] Currently, deep learning often faces streaming data, which requires the model to continuously learn new knowledge. Class incremental learning aims to simulate the learning method of the human brain and continuously adjust the model parameters to adapt to new data. The main difficulty faced by class incremental learning is the catastrophic forgetting problem, that is, when training on a new task data set, the model is more inclined to learn and remember the features of the new data, and will forget the knowledge learned on the old tasks, resulting in a significant drop in accuracy on the old task test set. How to enable the class incremental model to integrate all knowledge and construct a general classifier is a problem that needs to be solved.

[0003] Among the existing class incremental learning algorithms:

[0004] The algorithm based on data replay directly stores the original samples or uses a generative model to generate pseudo samples. When learning new tasks, catastrophic forgetting is alleviated by replaying previous samples. However, this method may cause overfitting of stored samples and is also limited by joint training.

[0005] Algorithms based on parameter isolation use different model parameters for each task. When there is no limit on the size of the architecture, the parameters of the previous task can be frozen and a new branch can be added for the new task. However, this method is limited to multi-head tasks, and as the number of tasks increases, the over-expansion of the model will make training more difficult.

[0006] Regularization-based algorithms do not store the original samples, but introduce a regularization term in the loss function to integrate the knowledge of the old tasks when learning on the new task data. However, this method has an obvious flaw, which is that it is not applicable when there is a domain conversion between tasks.

[0007] The invention is named "Incremental learning method based on joint distillation replay strategy" and the application publication number is (CN118072099A). The invention proposes a method for incremental learning based on joint distillation replay strategy. First, the distillation model of the initial task is trained; an additional buffer is set up to store the distillation data of the initial task; then a joint distillation model based on the distillation data of the old task and the original data of the new task is established; the joint distillation data is replayed during the training process of the next task to alleviate the catastrophic forgetting problem.

[0008] This method is a data replay algorithm that opens up additional memory space to store old task data and trains an incremental model from scratch. Summary of the invention

[0009] In order to overcome the deficiencies of the above-mentioned prior art, the purpose of the present invention is to provide a class incremental learning method, system, device and medium based on a pre-trained model. The invention utilizes the strong generalization ability and robust representation ability of the pre-trained model in downstream tasks, and further alleviates the catastrophic forgetting problem. That is, in the class incremental scenario setting, only the data set of the current task can be used for training. After the model parameters are updated, the classification effect on the old task data set will be greatly reduced. In addition, Prompt is used to fine-tune the pre-trained model to compensate for the domain difference between the pre-training data set and the class incremental data set. While ensuring the adaptability of the model, the generalization ability of the model is also retained. It can learn the knowledge of new tasks while not forgetting the knowledge of old tasks, and the image classification effect remains good.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is:

[0011] A class incremental learning method based on a pre-trained model comprises the following steps;

[0012] Step 1: In the first stage of class incremental learning, for the first task D 1 , fine-tune the pre-trained model ViT by embedding the prompt information Prompt; obtain the fine-tuned pre-trained model;

[0013] Step 2: Merge and freeze the embedding layer function in the fine-tuned pre-trained model obtained in step 1 and the embedding layer function of the pre-trained model ViT before fine-tuning to obtain a merged model;

[0014] Step 3: Use the merged model obtained in step 2 to perform class-incremental training from scratch, and finally obtain a class-incremental model with strong generalization and robustness;

[0015] Step 4: For the test sample, use the class increment model obtained in step 3 to make predictions and calculate the final classification accuracy.

[0016] In step 1, the first task D 1The dataset uses the top 10 classes on CIFAR100, the top 20 classes on CUB200, ImageNetR, and ImageNetA. The pre-trained model ViT is fine-tuned through the first task to make up for the domain difference between the ImageNet21k dataset. Since there is no category overlap between different tasks in the class incremental learning scenario, only the current dataset can be used to update the model parameters during training. After the update, the classification effect on the old task categories will be greatly reduced. As the number of training tasks increases, this phenomenon will be further aggravated, which is also called the catastrophic forgetting problem.

[0017] The specific process of fine-tuning the pre-trained model in step 1 is as follows:

[0018] Step 1.1: For T incremental tasks {D 1 ,D 2 ,…,D T}, is the t-th incremental task, including N t samples, is the i-th sample of the t-th class incremental task, is the label value of the i-th sample of the corresponding t-th class incremental task, for the first task The query function is used to obtain the query value q(x) from the input image x in , where the query value refers to the global features of the input image x extracted by the pre-trained model ViT without fine-tuning;

[0019] Step 1.2 first obtains the Hadamard product between the attention vector A and the query value q(x), and then calculates the cosine similarity between the Hadamard product and the key K to obtain the weight, as shown in the following formula:

[0020] α=cos(q(x)⊙A,K)

[0021] α={cos(q(x)⊙A1,K1),cos(q(x)⊙A2,K2),…,cos(q(x)⊙A M ,K M )}

[0022] The attention vector A and the key value K are a set of learnable parameters, where each key value K corresponds to an attention vector A. D represents the dimension of the embedding layer vector of the pre-trained model ViT, M represents the length of the prompt pool, ⊙ is the Hadamard product, cos(·) is the calculated cosine similarity,

[0023] Step 1.3: Perform a weighted summation of the weight α calculated in step 1.2 and the prompt information Prompt to obtain the prompt information of the new task. The formula is as follows:

[0024]

[0025] α m represents the mth weight, Indicates the current prompt information, L p is the size of the prompt information, D represents the dimension of the pre-trained ViT embedding layer vector, and M represents the length of the Prompt prompt pool;

[0026] Step 1.4 The embedding function φ(·) of the pre-trained model ViT divides the input image x into L patches, flattens them, and linearly projects them into a high-dimensional space using the fully connected layer of the pre-trained ViT. Then add it to the position code to get the input embedding feature of the self-attention layer Query h Q , key h K , value h V , where h = h Q =h K =h V , the output of the multi-head self-attention layer is expressed as:

[0027] MSA(h Q ,h K ,h V )=concat(h1,…,h m )W O

[0028] h i =Attention(h Q W i Q ,h K W i K ,h V W i V )

[0029] Here W O , W i Q , W i K , W i V are all projection matrices, m is the number of heads in the multi-head self-attention layer, concat(·) means connection, and Attention(·) represents the attention mechanism in ViT;

[0030] The suggestive information p obtained in step 1.3 is embedded into the multi-head self-attention layer of the pre-trained model ViT, and the formula is as follows:

[0031] f(p,h)=MSA(h Q ,[p K ;h K ],[p V ;h V ])

[0032] Here f(·) represents the output of the self-attention layer, which divides the prompt information p into two parts

[0033] Step 1.5 Input image x into the multi-head attention layer f embedded with the prompt information Prompt obtained in step 1.4 p,h (·) and then enter the final classifier layer Make predictions in , add query loss for optimization, and get the final loss function, which is expressed as:

[0034]

[0035] Here L(·) represents the cross entropy classification loss, λ is the balancing hyperparameter, and cos(·) is the cosine similarity.

[0036] Step 1.6 back-propagates the loss function obtained in step 1.5, updates the prompt information Prompt, key K, attention vector A, and finally obtains the fine-tuned pre-trained model, thereby obtaining the embedding function φ in the model. * (·), embedding function φ * (x) will extract the patch features and global information of the input image x.

[0037] The specific process of merging and freezing the model embedding layer function in step 2 is as follows:

[0038] After obtaining the fine-tuned pre-trained model, the embedding function φ * (·) is merged with the embedding function φ(·) of the original pre-trained model. The two have the same dimensions and are directly concatenated along the last dimension to obtain the merged model embedding function [φ * (·),φ(·)], in order to ensure the generalization of the model, the embedding layer function is frozen, the parameters of the embedding function are no longer updated during the training process, and the requires_grad of the embedding function parameters is set to false.

[0039] The specific process of the class incremental training in step 3 is as follows:

[0040] Step 3.1 Get the class incremental task {D 1 ,D 2 ,…,D T}, the embedding layer function [φ * (·),φ(·)], extract the prototype vector of each class, the formula is as follows:

[0041]

[0042] Here Incremental Task D t represents the total number of samples of category i, p i It is the prototype vector of the final category i.

[0043] Step 3.2 uses the prototype vector of each category obtained in step 3.1 to replace the weight of the classifier. The formula is as follows:

[0044] ω i =p i

[0045] Step 3.3 uses the classifier obtained in step 3.2 to predict the sample and obtain the loss function. The formula is as follows:

[0046]

[0047] Here L(·) represents the cross entropy classification loss.

[0048] The specific process of using the class increment model to make predictions in step 4 is as follows:

[0049] For a new test sample x, use the merged model embedding function [φ * (·),φ(·)] calculate the corresponding embedding vector, and then compare the result with the prototype in the classifier by cosine similarity. The category of the prototype feature closest to it is the category of the test sample.

[0050] A class incremental learning system based on a pre-trained model, comprising a fine-tuning model module, a model merging and freezing module, and a classifier module;

[0051] The fine-tuning model module adopts the prompt information Prompt embedding method to fine-tune the pre-trained model VisionTransformer (ViT);

[0052] The model merging and freezing module merges and freezes the embedding layer function of the fine-tuned pre-training model and the embedding layer function of the original pre-training model;

[0053] The classifier module uses the prototype vector of each class extracted by the merged model to replace the weight of the classifier and make predictions for the samples.

[0054] A quasi-incremental learning device based on a pre-trained model, comprising:

[0055] Memory: used to store a computer program for implementing the aforementioned quasi-incremental learning method based on a pre-trained model;

[0056] Processor: used to implement the class incremental learning method based on a pre-trained model when executing the computer program.

[0057] A computer-readable storage medium, comprising:

[0058] The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a class incremental learning method based on a pre-trained model.

[0059] Beneficial effects of the present invention:

[0060] The present invention uses the Vision Transformer (ViT) trained on ImageNet21k as a pre-trained model. Compared with the traditional method of training the model from scratch as the data stream arrives, that is, randomly initializing the model weights, the present invention utilizes the strong generalization ability and robust representation ability of the pre-trained model in downstream tasks, further alleviating the catastrophic forgetting problem.

[0061] The present invention uses a prompt information embedding method to fine-tune the pre-trained model, obtains features for a specific data set in step 1, solves the domain drift problem between the pre-trained data set and the class incremental data set, and compensates for the distribution difference.

[0062] The present invention uses an attention-based prompt weighted summation scheme to promote knowledge reuse, and the prompt information of the new task will include knowledge from the old task.

[0063] The present invention freezes the model embedding function and retains the knowledge of old tasks.

[0064] The present invention uses the prototype vector of each class in the merged model to replace the weight of the classifier, and encodes the specific information of each class while ensuring the general information, thereby increasing the plasticity of the model and alleviating the catastrophic forgetting problem of old tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The figure is a flow chart of the method of the present invention.

[0066] Figure 2This is a method of using prompt information Prompt embedding to fine-tune the pre-trained model module diagram of the present invention.

[0067] Figure 3 This is a graph showing the accuracy changes of the present invention when the CIFAR100 dataset is divided into 10 tasks on average.

[0068] Figure 4 This is the accuracy change diagram of the present invention on the CIFAR100 dataset, when the first task includes 50 categories and the number of categories in the remaining tasks is 5.

[0069] Figure 5 This is a comparison chart of the total parameters of the present invention and other methods. DETAILED DESCRIPTION

[0070] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0071] See also Figure 1 , a class incremental learning method based on a pre-trained model, comprising the following steps:

[0072] The datasets used are CIFAR100, CUB200, ImageNetR, and ImageNetA. The categories of the datasets are shuffled by random seeds. In order to compare with the classic method, each task includes the same number of categories. For example, CIFAR100-10 means that it is divided into 10 tasks, each task includes 10 categories. During the training process, batch_size=128, the Adam optimizer is used for optimization, the learning rate is set to 0.9, the learning rate decay adopts the cosine annealing strategy, and the training is performed for 40 epochs.

[0073] Step 1: In the first stage of class incremental learning, for the first task D 1 , for example, image classification tasks on the first 10 classes on the CIFAR100 dataset. The pre-trained model Vision Transformer (ViT) is fine-tuned by embedding prompt information. Features specific to a specific dataset are obtained, solving the domain drift problem between the pre-trained dataset and the class increment dataset, and compensating for the distribution difference.

[0074] See also Figure 2 , Step 1.1 For T class incremental tasks {D 1 ,D 2 ,…,D T}, is the t-th incremental task, including N t samples, is the i-th sample of the t-th class incremental task, is the label value of the i-th sample of the corresponding t-th class incremental task.

[0075] For the first task The query function is used to obtain the query value q(x) from the input image x in . The query value here refers to the global features of the input image x extracted by the pre-trained model ViT without fine-tuning;

[0076] Step 1.2 first obtains the Hadamard product between the attention vector A and the query value q(x), and then calculates the cosine similarity between the Hadamard product and the key K to obtain the weight, as shown in the following formula:

[0077] α=cos(Q(x)⊙A,K)

[0078] α={cos(q(x)⊙A1,K1),cos(q(x)⊙A2,K2),…,cos(q(x)⊙A M ,K M )}

[0079] The attention vector A and the key value K are a set of learnable parameters, where each key value K corresponds to an attention vector A. D represents the dimension of the embedding layer vector of the pre-trained model ViT, and M represents the length of the prompt pool. ⊙ is the Hadamard product, cos(·) is the calculated cosine similarity,

[0080] Step 1.3: Perform a weighted summation of the weight α calculated in step 1.2 and the prompt information Prompt to obtain the prompt information of the new task. The formula is as follows:

[0081]

[0082] Here Indicates the current prompt information. p is the Prompt size of the prompt information, D represents the dimension of the pre-trained ViT embedding layer vector, and M represents the Prompt prompt pool length.

[0083] In the experiment, the dimension of the embedding layer vector of the pre-trained model ViT is D=768, the length of the prompt pool is M=100, and the size of the prompt information is L p =8.

[0084] Step 1.4 The embedding function φ(·) of the pre-trained model ViT divides the input image x into L patches, flattens them, and linearly projects them into a high-dimensional space using the fully connected layer of the pre-trained ViT. Then add it to the position code to get the input embedding feature of the self-attention layer Query hQ , key h K , value h V , where h = h Q =h K =h V , the output of the multi-head self-attention layer is expressed as:

[0085] MSA(h Q ,h K ,h V )=concat(h1,…,h m )W O

[0086] h i =Attention(h Q W i Q ,h K W i K ,h V W i V )

[0087] Here W O , W i Q , W i K , W i V They are all projection matrices, m is the number of heads in the multi-head self-attention layer, concat(·) means connection, and Attention(·) represents the attention mechanism in ViT.

[0088] The suggestive information p obtained in step 1.3 is embedded into the multi-head self-attention layer of the pre-trained model ViT, and the formula is as follows:

[0089] f(p,h)=MSA(h Q ,[p K ;h K ],[p V ;h V ])

[0090] Here f(·) represents the output of the self-attention layer, which divides the prompt information p into two parts

[0091] In the experiment, the hint information p is inserted into the 1st to 5th layers of the self-attention layer of the pre-trained model ViT. The features of the specific dataset are obtained, providing additional task context and enhancing the adaptability of the pre-trained model to specific tasks.

[0092] Step 1.5 Input image x into the multi-head attention layer f embedded with the prompt information Prompt obtained in step 1.4 p,h (·) and then enter the final classifier layer Make predictions in , add query loss for optimization, and get the final loss function, which is expressed as:

[0093]

[0094] Here L(·) represents the cross entropy classification loss, λ is the balancing hyperparameter, and cos(·) is the cosine similarity.

[0095] In the experiment, λ=0.75 to balance the classification loss and query loss and improve the stability of model training.

[0096] Step 1.6 back-propagates the loss function obtained in step 1.5, updates the prompt information Prompt, key K, and attention vector A, and finally obtains the fine-tuned pre-trained model. Thus, the embedding function φ in the model is obtained. * (·), embedding function φ * (x) will extract the patch features and global information of the input image x.

[0097] Step 2: Merge and freeze the embedding layer function in the fine-tuned pre-trained model obtained in step 1 and the embedding layer function of the pre-trained model ViT before fine-tuning to obtain a merged model; compensate for the domain difference between the pre-trained dataset and the class increment dataset. While ensuring the adaptability of the model, the generalization ability of the model is also retained.

[0098] The specific process is:

[0099] After obtaining the fine-tuned pre-trained model, the embedding function φ * (·) is merged with the embedding function φ(·) of the original pre-trained model. The two have the same dimensions and are directly concatenated along the last dimension. The merged model embedding function [φ * (·),φ(·)], in order to ensure the generalization of the model, the embedding layer function is frozen, the parameters of the embedding function are no longer updated during the training process, and the requires_grad of the embedding function parameters is set to false.

[0100] Step 3: Use the merged model obtained in step 2 to perform incremental training from scratch, and finally obtain an incremental model with strong generalization and robustness. This increases the plasticity of the model while alleviating the problem of catastrophic forgetting of old tasks.

[0101] Step 3.1 Get the class incremental task {D 1 ,D2 ,…,D T}, the embedding layer function [φ * (·),φ(·)], extract the prototype vector of each class, the formula is as follows:

[0102]

[0103] Here Incremental Task D t Represents the total number of samples of category i. i It is the prototype vector of the final category i.

[0104] Step 3.2 uses the prototype vector of each category obtained in step 3.1 to replace the weight of the classifier. The formula is as follows:

[0105] ω i =p i

[0106] Step 3.3 uses the classifier obtained in step 3.2 to predict the sample and obtain the loss function. The formula is as follows:

[0107]

[0108] Here L(·) represents the cross entropy classification loss.

[0109] Step 4: For the test sample, use the class increment model obtained in step 3 to make predictions and calculate the final classification accuracy.

[0110] The specific process is:

[0111] For a new test sample x, use the merged model embedding function [φ * (·),φ(·)] to calculate the corresponding embedding vector. The specific steps are to divide the input image x into multiple patches, then linearly project it into a high-dimensional space, add position encoding information, and finally extract the output of the category token as the global embedding vector. Then compare the result with the prototype in the classifier by cosine similarity, and the category of the prototype feature closest to it is the category of the test sample.

[0112] In order to evaluate the performance of the proposed class incremental learning method based on pre-trained models, the method of this application (Ours) is compared with several advanced class incremental learning methods, namely Finetune, Finetune Adapter, LwF, DER, L2P, FOSTER, LAE, DualPrompt. For the datasets CIFAR100, CUB200, ImageNetR, and ImageNetA, they are divided into 20 tasks, each task contains 5 classes, 10 classes, 5 classes, and 10 classes respectively. The experiments report the average precision And the final accuracy A last , the experimental results are shown in Table 1.

[0113] Table 1 Comparison of the accuracy of different methods on multiple datasets

[0114]

[0115] From the results, we can see that although the performance of our method (Ours) is slightly lower than LAE on CIFAR100 and ImageNetR datasets, it performs well on CUB200 and ImageNetA datasets. It also exceeds most advanced methods. In addition, our method is also related to the number of categories in the first dataset. For the convenience of comparison, we use the method of equally dividing the number of categories.

[0116] See also Figure 3 , which is a precision change chart divided into 10 tasks on the CIFAR100 dataset. Each task contains 10 classes. It can be seen that the overall precision drops relatively slowly and eventually reaches 82.36%, effectively alleviating the catastrophic forgetting problem.

[0117] See also Figure 4 In order to explore the impact of the first dataset, this application conducted experiments on the CIFAR100 dataset. On the CIFAR100 dataset, it is divided into 11 tasks. The first task contains 50 classes, and the remaining 10 tasks each contain 5 classes. It can be seen that at the beginning of training, the accuracy reached 92.34%, which compensated for the distribution difference of the training dataset. And after the training of 11 tasks is completed, the accuracy can also reach 84.21%. Compared with the method of equally dividing the number of categories for each task, the first task contains more categories, which can achieve better results and further alleviate the problem of catastrophic forgetting.

[0118] See also Figure 5Since the Vision Transformer (ViT) is used as the backbone network, the problem of parameter quantity is involved. This application compares the total parameter quantity of the networks used by different methods. It can be seen that compared with the DualPrompt method that also uses Prompt prompt information, the total parameter quantity of the method of this application (Ours) is smaller under the same conditions, and compared with the DER based on model expansion, it is half as low, which also proves that our method (Ours) is more suitable for real environments and reduces memory consumption.

[0119] A quasi-incremental learning device based on a pre-trained model, comprising:

[0120] Memory: used to store a computer program for implementing the aforementioned quasi-incremental learning method based on a pre-trained model;

[0121] Processor: used to implement the class incremental learning method based on a pre-trained model when executing the computer program.

[0122] A computer-readable storage medium, comprising:

[0123] The computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a class incremental learning method based on a pre-trained model.

Claims

1. A class incremental learning method based on a pre-trained model, characterized in that: The steps include: Step 1: In the first stage of class incremental learning, for the first task D 1 , fine-tune the pre-trained model ViT by embedding the prompt information Prompt; obtain the fine-tuned pre-trained model; Step 2: Merge and freeze the embedding layer function in the fine-tuned pre-trained model obtained in step 1 and the embedding layer function of the pre-trained model ViT before fine-tuning to obtain a merged model; Step 3: Use the merged model obtained in step 2 to perform class-incremental training from scratch, and finally obtain a class-incremental model with strong generalization and robustness; Step 4: For the test sample, use the class increment model obtained in step 3 to make predictions and calculate the final classification accuracy.

2. The class incremental learning method based on a pre-training model according to claim 1, characterized in that: In step 1, the first task D 1 The dataset uses the first 10 classes on CIFAR100, the first 20 classes on CUB200, ImageNetR, and ImageNetA. The pre-trained model ViT is fine-tuned through the first task to make up for the domain difference between the ImageNet21k dataset and the original image.

3. The class incremental learning method based on a pre-training model according to claim 1, characterized in that: The specific process of fine-tuning the pre-trained model in step 1 is as follows: Step 1.1: For T incremental tasks {D 1 ,D 2 ,…,D T }, is the t-th incremental task, including N t samples, is the i-th sample of the t-th class incremental task, is the label value of the i-th sample of the corresponding t-th class incremental task, for the first task The query function is used to obtain the query value q(x) from the input image x in , where the query value refers to the global features of the input image x extracted by the pre-trained model ViT without fine-tuning; Step 1.2: First, obtain the Hadamard product between the attention vector A and the query value q(x), and then calculate the cosine similarity between the Hadamard product and the key K to obtain the weight. The formula is as follows: α=cos(q(x)⊙A,K) α={cos(q(x)⊙A1,K1),cos(q(x)⊙A2,K2),…,cos(q(x)⊙A M ,K M )} The attention vector A and the key value K are a set of learnable parameters, where each key value K corresponds to an attention vector A. D represents the dimension of the embedding layer vector of the pre-trained model ViT, M represents the length of the prompt pool, ⊙ is the Hadamard product, cos(·) is the calculated cosine similarity, Step 1.3: Take the weighted sum of the weight α calculated in step 1.2 and the prompt information Prompt to obtain the prompt information of the new task. The formula is as follows: α m represents the mth weight, Indicates the current prompt information, L p is the size of the prompt information, D represents the dimension of the pre-trained ViT embedding layer vector, and M represents the length of the Prompt prompt pool; Step 1.4: The embedding function φ(·) of the pre-trained model ViT divides the input image x into L patches, flattens them, and linearly projects them into a high-dimensional space using the fully connected layer of the pre-trained ViT. Then add it to the position code to get the input embedding feature of the self-attention layer Query h Q , key h K , value h V , where h = h Q =h K =h V , the output of the multi-head self-attention layer is expressed as: MSA(h Q ,h K ,h V )=concat(h1,…,h m )W O h i =Attention(h Q W i Q ,h K W i K ,h V W i V ) Here W O , W i Q , W i K , W i V are all projection matrices, m is the number of heads in the multi-head self-attention layer, concat(·) means connection, and Attention(·) means the head self-attention layer in ViT; The suggestive information p obtained in step 1.3 is embedded into the multi-head self-attention layer of the pre-trained model ViT, and the formula is as follows: f(p,h)=MSA(h Q ,[p K ;h K ],[p V ;h V ]) Here f(·) represents the output of the self-attention layer, which divides the prompt information p into two parts Step 1.5 Input image x into the multi-head attention layer f embedded with the prompt information Prompt obtained in step 1.4 p,h (·) and then enter the final classifier layer Make predictions in , add query loss for optimization, and get the final loss function, which is expressed as: Here L(·) represents the cross entropy classification loss, λ is the balancing hyperparameter, and cos(·) is the cosine similarity. Step 1.6 back-propagates the loss function obtained in step 1.5, updates the prompt information Prompt, key K, attention vector A, and finally obtains the fine-tuned pre-trained model, thereby obtaining the embedding function φ in the model. * (·), embedding function φ * (x) will extract the patch features and global information of the input image x.

4. The class incremental learning method based on a pre-training model according to claim 3 is characterized in that: The specific process of merging and freezing the model embedding layer function in step 2 is as follows: After obtaining the fine-tuned pre-trained model, the embedding function φ * (·) is merged with the embedding function φ(·) of the original pre-trained model. The two have the same dimensions and are directly concatenated along the last dimension to obtain the merged model embedding function [φ * (·),φ(·)], freeze the embedding layer function, no longer update the parameters of the embedding function during training, and set requires_grad of the embedding function parameters to false.

5. The class incremental learning method based on a pre-training model according to claim 4 is characterized in that: The specific process of the class incremental training in step 3 is as follows: Step 3.1 Get the class incremental task {D 1 ,D 2 ,…,D T }, the embedding layer function [φ * (·),φ(·)], extract the prototype vector of each class, the formula is as follows: Here Incremental Task D t represents the total number of samples of category i, p i It is the prototype vector of the final category i. Step 3.2 uses the prototype vector of each category obtained in step 3.1 to replace the weight of the classifier. The formula is as follows: oh i =p i Step 3.3 uses the classifier obtained in step 3.2 to predict the sample and obtain the loss function. The formula is as follows: Here L(·) represents the cross entropy classification loss.

6. The class incremental learning method based on a pre-training model according to claim 5, characterized in that: The specific process of using the class increment model to make predictions in step 4 is as follows: For a new test sample x, use the merged model embedding function [φ * (·),φ(·)] calculate the corresponding embedding vector, and then compare the result with the prototype in the classifier by cosine similarity. The category of the prototype feature closest to it is the category of the test sample.

7. A pre-trained model-based quasi-incremental learning system for implementing the method described in any one of claims 1 to 6, characterized in that: Includes fine-tuning model module, model merging and freezing module, and classifier module; The fine-tuning model module adopts the prompt information Prompt embedding method to fine-tune the pre-trained model VisionTransformer (ViT); The model merging and freezing module merges and freezes the embedding layer function of the fine-tuned pre-training model and the embedding layer function of the original pre-training model; The classifier module uses the prototype vector of each class extracted by the merged model to replace the weight of the classifier and make predictions for the samples.

8. A quasi-incremental learning device based on a pre-trained model, comprising: Memory: used to store a computer program for implementing a quasi-incremental learning method based on a pre-trained model as described in any one of claims 1 to 6; Processor: used to implement the class incremental learning method based on a pre-trained model when executing the computer program.

9. A computer-readable storage medium, comprising: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement any one of claims 1-6, a class incremental learning method based on a pre-trained model.

Images