A pre-trained model-based class-incremental learning method, system, device and medium

Abstract

The application discloses a class incremental learning method, system, device and medium based on a pre-training model, which comprises the following steps: in the first stage of class incremental learning, a first task D 1 is fine-tuned by embedding a prompt information Prompt into a pre-training model ViT; a fine-tuned pre-training model is obtained; in step 2, the embedding layer function of the fine-tuned pre-training model obtained in step 1 and the embedding layer function of the pre-training model ViT before fine-tuning are merged and frozen to obtain a merged model; in step 3, the merged model obtained in step 2 is used for head class incremental training, and finally a class incremental model with strong generalization and robustness is obtained; in step 4, the class incremental model obtained in step 3 is used for prediction on test samples, and the final classification accuracy is calculated. The application can ensure the adaptability of the model while retaining the generalization ability of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of class-incremental learning, and particularly relates to a class-incremental learning method based on a pre-trained model, a system, a device and a medium. BACKGROUND

[0002] At present, deep learning often faces streaming data, which requires the model to continuously learn new knowledge, and class-incremental learning aims to simulate the learning mode of the human brain, and continuously adjust the model parameters to adapt to new data. The main difficulty of class-incremental learning is the problem of catastrophic forgetting, that is, when training on a new task dataset, the model tends to learn and remember the features of new data, and forget the knowledge learned on the old task, which will cause a significant decrease in accuracy on the old task test set. How to make the class-incremental model integrate all the knowledge and construct a general classifier is a problem to be solved.

[0003] In the existing class-incremental learning algorithm:

[0004] The algorithm based on data replay is to directly store the original samples or use a generative model to generate pseudo samples. When learning a new task, the previous samples are replayed to alleviate the problem of catastrophic forgetting. However, this method may cause overfitting when storing samples, and is also limited by joint training.

[0005] The algorithm based on parameter isolation uses different model parameters for each task. When there is no restriction on the size of the architecture, the parameters of the previous task can be frozen, and a new branch can be added for the new task. However, this method is limited to multi-head tasks, and as the number of tasks increases, the over-expansion of the model will make training more difficult.

[0006] The algorithm based on regularization does not store the original samples, but introduces a regularization term in the loss function part to integrate the knowledge of the old task when learning on the new task data. However, this method has a significant drawback, which is not applicable when there is a domain shift between tasks.

[0007] The invention with the title "Class-incremental learning method based on joint distillation replay strategy" and the application publication number (CN118072099A). The invention proposes a class-incremental learning method based on a joint distillation replay strategy. First, train the distillation model of the initial task; additionally set up a buffer to store the distillation data of the initial task; then establish a joint distillation model based on the distillation data of the old task and the original data of the new task; replay the joint distillation data during the training process of the next task to alleviate the problem of catastrophic forgetting.

[0008] This method belongs to the data replay algorithm, and additional memory space is opened to store the old task data, and the class-incremental model is trained from scratch. SUMMARY

[0009] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a class-incremental learning method, system, device and medium based on a pre-trained model. The application utilizes the strong generalization ability and robust representation ability of the pre-trained model in downstream tasks, further reducing the catastrophic forgetting problem. That is, in the class-incremental scenario, only the current task dataset can be used for training. After updating the model parameters, the classification effect on the old task dataset will be greatly reduced. In addition, the pre-trained model is fine-tuned using Prompt, which compensates for the domain difference between the pre-trained dataset and the class-incremental dataset. While ensuring model adaptability, the model's generalization ability is also preserved, allowing it to learn new task knowledge while not forgetting old task knowledge, and maintaining good image classification results.

[0010] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0011] A class-incremental learning method based on a pre-trained model, comprising the following steps:

[0012] Step 1: In the first phase of class-incremental learning, for the first task D 1 , the pre-trained model ViT is fine-tuned by embedding the prompt information Prompt; obtaining the fine-tuned pre-trained model;

[0013] Step 2: Merge and freeze the embedding layer function of the fine-tuned pre-trained model obtained in step 1 and the embedding layer function of the pre-trained model ViT before fine-tuning, to obtain a merged model;

[0014] Step 3: Use the merged model obtained in step 2 to perform class-incremental training from scratch, and finally obtain a class-incremental model with strong generalization and robustness;

[0015] Step 4: For test samples, use the class-incremental model obtained in step 3 to make predictions and calculate the final classification accuracy.

[0016] In the step 1, the first task D 1The dataset of the first task adopts the first 10 classes on CIFAR100, the first 20 classes on CUB200, ImageNetR, ImageNetA, and the domain difference between the pre-training model ViT and the ImageNet21k dataset is compensated. Since there is no overlap between different tasks in the class-incremental learning scenario, only the current dataset can be used to update the model parameters during training, and after updating, the classification effect on the old task class will be greatly reduced. With the increase of the number of training tasks, this phenomenon will further aggravate, which is also called the catastrophic forgetting problem.

[0017] The specific process of the step 1 of fine-tuning the pre-training model is as follows:

[0018] Step 1.1: For T class-incremental tasks {D 1 ,D 2 ,…,D T}, is the t-th class-incremental task, including N t samples, is the i-th sample of the t-th class-incremental task, is the label value of the corresponding i-th sample of the t-th class-incremental task, and the input picture x in the first task is obtained by querying function, wherein the query value refers to the global feature of the input picture x extracted by the pre-training model ViT without fine-tuning;

[0019] Step 1.2: first obtain the Hadamard product between the attention vector A and the query value q(x), and then calculate the cosine similarity between the Hadamard product and the key K to obtain the weight, the formula is as follows:

[0020] α=cos(q(x)⊙A,K)

[0021] α={cos(q(x)⊙A1,K1),cos(q(x)⊙A2,K2),…,cos(q(x)⊙A M ,K M )}

[0022] The attention vector A and the key value K are a set of learnable parameters, wherein each key value K corresponds to an attention vector A, D represents the dimension of the pre-training model ViT embedding layer vector, M represents the length of the Prompt prompt pool, is the Hadamard product, and cos(·) is the calculation of cosine similarity,

[0023] Step 1.3 Weighted sum of the weight α calculated in step 1.2 and the prompt information Prompt to obtain the prompt information of the new task, the formula is as follows:

[0024]

[0025] α m represents the mth weight, represents the current Prompt information, L p is the size of the prompt information Prompt, D represents the dimension of the pre-trained ViT embedding layer vector, and M represents the length of the Prompt prompt pool;

[0026] Step 1.4 The embedding function φ(·) of the pre-trained model ViT divides the input picture x into L patches for flattening, and uses the fully connected layer of the pre-trained ViT to linearly project to a high-dimensional space And then add the position encoding to get the input embedding feature of the self-attention layer Query h Q , key h K , value h V , where h = h Q = h K = h V The output of the multi-head self-attention layer is represented by the formula:

[0027] MSA(h Q ,h K ,h V ) = concat(h1,…,h m )W O

[0028] h i = Attention(h Q W i Q ,h K W i K ,h V W i V )

[0029] Here W O , W i Q , W i K , W i V are projection matrices, m is the number of heads of the multi-head self-attention layer, concat(·) represents connection, and Attention(·) represents the attention mechanism in ViT;

[0030] The prompt information p obtained in step 1.3 is embedded into the multi-head self-attention layer of the pre-trained model ViT, and the formula is as follows:

[0031] f(p, h) = MSA(h Q , [p K ; h K ], [p V ; h V ])

[0032] Here, f(·) represents the output of the self-attention layer, and the prompt information p is evenly divided into two parts

[0033] After the input picture x in step 1.5 is embedded in the multi-head attention layer f p,h (·) obtained in step 1.4, it enters the final classifier layer for prediction, and the query loss is added for optimization to obtain the final loss function, which is represented by the formula:

[0034]

[0035] Here, L(·) represents the cross-entropy classification loss, λ is the balance hyperparameter, and cos(·) is the cosine similarity,

[0036] Step 1.6: The loss function obtained in step 1.5 is backpropagated to update the prompt information Prompt, the key K, and the attention vector A, and finally the fine-tuned pre-trained model is obtained, thereby obtaining the embedding function φ * (·) in the model. The embedding function φ * (x) will extract the patch features and global information of the input picture x.

[0037] The specific process of merging and freezing the model embedding layer function in step 2 is as follows:

[0038] After obtaining the fine-tuned pre-trained model, the embedding function φ * (·) and the original pre-trained model embedding function φ(·) are merged, and their dimensions are the same. Along the last dimension, they are directly spliced to obtain the merged model embedding function [φ * (·), φ(·)]. In order to ensure the generalization of the model, the embedding layer function is frozen, and the embedding function parameters are not updated during training. Set requires_grad of the embedding function parameters to false.

[0039] The specific process of class-incremental training in step 3 is as follows:

[0040] Step 3.1 Obtain class-incremental task D 1 ,D 2 ,…,D T} by the embedding layer function [φ * (·),φ(·)] of the merged model obtained in step 2, extract the prototype vector of each class, as follows:

[0041]

[0042] Here, In the class-incremental task D t represents the total number of samples of class i, and p i is the final prototype vector of class i.

[0043] Step 3.2 Replace the weights of the classifier with the prototype vector of each class obtained in step 3.1, as follows:

[0044] ω i =p i

[0045] Step 3.3 Use the classifier obtained in step 3.2 to make predictions for samples, and obtain the loss function, as follows:

[0046]

[0047] Here, L(·) represents the cross-entropy classification loss.

[0048] The specific process of step 4 is to use the class-incremental model to make predictions, as follows:

[0049] For a new test sample x, use the embedding function [φ * (·),φ(·)] of the merged model to calculate the corresponding embedding vector, and then compare the result with the prototype in the classifier, and the class of the nearest prototype feature is the class of the test sample.

[0050] A class-incremental learning system based on a pre-trained model, comprising a fine-tuned model module, a model merging and freezing module, and a classifier module;

[0051] The fine-tuned model module uses a prompt information Prompt embedding method to fine-tune the pre-trained model VisionTransformer (ViT);

[0052] The model merging and freezing module uses the embedding layer function of the fine-tuned pre-trained model and the embedding layer function of the original pre-trained model to merge and then freeze;

[0053] The classifier module replaces the weights of the classifier with the prototype vector of each class extracted by the merging model to make a prediction for the sample.

[0054] A pre-trained model-based class incremental learning device, comprising:

[0055] A memory for storing a computer program implementing the pre-trained model-based class incremental learning method;

[0056] A processor for implementing the pre-trained model-based class incremental learning method when executing the computer program.

[0057] A computer-readable storage medium, comprising:

[0058] The computer-readable storage medium stores a computer program, which can implement a pre-trained model-based class incremental learning method when executed by a processor.

[0059] The beneficial effects of the present application are:

[0060] The present application uses a Vision Transformer (ViT) trained on ImageNet21k as a pre-trained model. Compared with the traditional training model from scratch as the data stream arrives, that is, randomly initializing the weights of the model, the present application utilizes the strong generalization ability and robust representation ability of the pre-trained model in downstream tasks to further alleviate the catastrophic forgetting problem.

[0061] The present application uses a method of embedding prompt information to fine-tune the pre-trained model, obtains features for specific data sets in step 1, solves the domain drift problem between the pre-trained data set and the class incremental data set, and compensates for the distribution difference.

[0062] The present application uses a scheme of attention-based Prompt weighted summation to promote knowledge reuse, and the prompt information of the new task will include knowledge from the old task.

[0063] The present application freezes the model embedding function, preserving the knowledge of the old task.

[0064] The present application uses the prototype vector of each class of the merging model to replace the weights of the classifier, encodes the specific information of each class while ensuring general information, increases the plasticity of the model, and alleviates the catastrophic forgetting problem of the old task. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 The method flowchart of the present application.

[0066] Figure 2Fine-tune the pre-trained model module graph by the method of embedding prompt information for the use of the present application.

[0067] Figure 3 The precision change graph of the present application when divided into 10 tasks on the CIFAR100 dataset.

[0068] Figure 4 The precision change graph of the present application on the CIFAR100 dataset, the first task includes 50 classes, and the remaining task class number is 5.

[0069] Figure 5 The total parameter amount comparison graph of the present application and other methods. DETAILED DESCRIPTION

[0070] The present application will be further described in detail below with reference to the accompanying drawings.

[0071] Referring to Figure 1 , a pre-trained model-based class incremental learning method comprises the following steps:

[0072] The dataset adopts CIFAR100, CUB200, ImageNetR, and ImageNetA. The classes of the dataset are shuffled by random seeds. In order to compare with the classical method, the same number of classes are included in each task, such as CIFAR100-10, which means that it is divided into 10 tasks, and each task includes 10 classes. In the training process, batch_size=128, the Adam optimizer is used for optimization, the learning rate is set to 0.9, the learning rate decay adopts the cosine annealing strategy, and 40 epochs are trained.

[0073] Step 1: In the first stage of class incremental learning, for the first task D 1 , such as the image classification task on the first 10 classes on the CIFAR100 dataset. The pre-trained model Vision Transformer (ViT) is fine-tuned by the method of embedding prompt information. The characteristics for a specific dataset are obtained, which solves the domain drift problem between the pre-training dataset and the class incremental dataset, and compensates for the distribution difference.

[0074] Referring to Figure 2 , step 1.1 is for T class incremental tasks {D 1 ,D 2 ,…,D T}, is the tth class incremental task, including N t samples, is the ith sample of the tth class incremental task, is the label value of the i-th sample of the corresponding t-th class incremental task.

[0075] The input picture x in the first task The query value q(x) is obtained by querying the function. The query value here refers to the global feature of the input picture x extracted by the unadjusted pre-trained model ViT;

[0076] Step 1.2 first obtains the Hadamard product between the attention vector A and the query value q(x), and then calculates the cosine similarity between the Hadamard product and the key K to obtain the weight, as follows:

[0077] α=cos(Q(x)⊙A,K)

[0078] α={cos(q(x)⊙A1,K1),cos(q(x)⊙A2,K2),…,cos(q(x)⊙A M ,K M )}

[0079] The attention vector A and the key value K are a set of learnable parameters, where each key value K here corresponds to an attention vector A, D represents the dimension of the pre-trained model ViT embedding layer vector, and M represents the length of the Prompt prompt pool. ⊙ is the Hadamard product, and cos(·) is the cosine similarity,

[0080] Step 1.3 weights the sum of the weights α calculated in step 1.2 and the prompt information Prompt to obtain the prompt information of the new task, as follows:

[0081]

[0082] Here, represents the current Prompt information. L p is the size of the prompt information Prompt, D represents the dimension of the pre-trained ViT embedding layer vector, and M represents the length of the Prompt prompt pool.

[0083] In the experiment, the dimension of the pre-trained model ViT embedding layer vector D=768, the length of the Prompt prompt pool M=100, and the size of the prompt information Prompt L p =8.

[0084] Step 1.4 The embedding function φ(·) of the pre-trained model ViT divides the input picture x into L patches for flattening, and uses the fully connected layer of the pre-trained ViT to linearly project to a high-dimensional space and then adds the position encoding to obtain the input embedding feature of the self-attention layer query hQ , key h K , value h V where h = h Q = h K = h V The output of the multi-head self-attention layer is represented by the formula:

[0085] MSA(h Q , h K , h V ) = concat(h1, …, h m ) W O

[0086] h i = Attention(h Q W i Q , h K W i K , h V W i V )

[0087] Here W O , W i Q , W i K , W i V are projection matrices, m is the number of heads of the multi-head self-attention layer, concat(·) represents concatenation, and Attention(·) represents the attention mechanism in ViT.

[0088] The prompt information p obtained in step 1.3 is embedded into the multi-head self-attention layer of the pre-trained model ViT, and the formula is as follows:

[0089] f(p, h) = MSA(h Q , [p K ; h K ], [p V ; h V ])

[0090] Here f(·) represents the output of the self-attention layer, and the prompt information p is evenly divided into two parts

[0091] In the experiment, the prompt information p is inserted into the 1-5 layers of the self-attention layer of the pre-trained model ViT. The characteristics of the specific data set are obtained, the additional task context is provided, and the adaptability of the pre-trained model to the specific task is enhanced.

[0092] Step 1.5. The input picture x is input into the multi-head attention layer f embedded with the prompt information obtained in step 1.4. p,h After that, it enters the final classifier layer The prediction is made in the middle, and the query loss is added for optimization to obtain the final loss function, which is expressed as:

[0093]

[0094] Here, L(·) represents the cross-entropy classification loss, λ is the balance hyperparameter, and cos(·) is the cosine similarity calculation,

[0095] In the experiment, λ = 0.75. The classification loss and the query loss are balanced to improve the stability of model training.

[0096] Step 1.6. The loss function obtained in step 1.5 is backpropagated to update the prompt information Prompt, the key K, and the attention vector A, and finally the fine-tuned pre-trained model is obtained. Thus, the embedding function φ * (·) in the model is obtained. * (x) will extract the patch features and global information of the input picture x.

[0097] Step 2: The embedding layer function of the fine-tuned pre-trained model obtained in step 1 and the embedding layer function of the pre-trained model ViT before fine-tuning are merged and frozen to obtain a merged model; the domain difference between the pre-training data set and the class incremental data set is compensated. While ensuring the adaptability of the model, the generalization ability of the model is also preserved.

[0098] The specific process is as follows:

[0099] After obtaining the fine-tuned pre-trained model, the embedding function φ * (·) and the embedding function φ(·) of the original pre-trained model are merged, and their dimensions are the same, and they are directly spliced along the last dimension. The merged model embedding function [φ * (·), φ(·)] is obtained. In order to ensure the generalization of the model, the embedding layer function is frozen, and the parameters of the embedding function are not updated during training. Set requires_grad of the embedding function parameters to false.

[0100] Step 3: Use the merged model obtained in step 2 to perform class incremental training from scratch to obtain a class incremental model with strong generalization and robustness. Increase the plasticity of the model while reducing the problem of catastrophic forgetting of old tasks.

[0101] Step 3.1: Obtain the class incremental task {D 1 , D2 ,…,D T}, the embedding layer function [φ * (·),φ(·)] of the merged model obtained by step 2, extract the prototype vector of each class, the formula is as follows:

[0102]

[0103] Here In the class incremental task D t Indicates the total number of samples of class i. p i is the prototype vector of the final class i.

[0104] Step 3.2 replaces the weights of the classifier with the prototype vector of each class obtained in step 3.1, and the formula is as follows:

[0105] ω i =p i

[0106] Step 3.3 uses the classifier obtained in step 3.2 to make predictions for the samples, and obtains the loss function, the formula is as follows:

[0107]

[0108] Here L(·) represents the cross-entropy classification loss.

[0109] Step 4: For test samples, use the class incremental model obtained in step 3 to make predictions, and calculate the final classification accuracy.

[0110] The specific process is as follows:

[0111] For a new test sample x, use the embedding function [φ * (·),φ(·)] of the merged model to calculate the corresponding embedding vector, the specific steps are to divide the input picture x into multiple patches, then project it to a high-dimensional space by linear projection, add position encoding information, and finally extract the output of the class token as the global embedding vector. Then compare the result with the prototype in the classifier, and the class of the nearest prototype feature is the class of the test sample.

[0112] To evaluate the performance of the proposed class-incremental learning method based on pre-trained model, the method of the present application (Ours) is compared with a variety of advanced class-incremental learning methods, i.e. Finetune, Finetune Adapter, LwF, DER, L2P, FOSTER, LAE, DualPrompt. For the data sets CIFAR100, CUB200, ImageNetR, ImageNetA are all divided into 20 tasks, each task contains 5 classes, 10 classes, 5 classes and 10 classes respectively. The average accuracy and final accuracy A last are reported in the experiment, and the experimental results are shown in Table 1.

[0113] Table 1 Accuracy comparison of different methods on multiple data sets

[0114]

[0115] From the results, it can be seen that the method of the present application (Ours) is slightly lower than LAE in performance on CIFAR100 and ImageNetR data sets, but has good effect on CUB200 and ImageNetA data sets. And it also exceeds most of the advanced methods. And the method of the present application is also related to the number of classes of the first data set, and here the equal division of class number method is adopted for convenience of comparison.

[0116] Referring to Figure 3 , the accuracy change graph on the CIFAR100 data set is divided into 10 tasks, each task contains 10 classes, it can be seen that the overall accuracy decreases relatively gently, and finally reaches 82.36%, effectively slowing down the catastrophic forgetting problem.

[0117] Referring to Figure 4 , in order to explore the influence of the first data set, the present application carries out experiments on the CIFAR100 data set. On the CIFAR100 data set, it is divided into 11 tasks, the first task contains 50 classes, and the remaining 10 tasks each contain 5 classes. It can be seen that at the beginning of training, the accuracy reaches 92.34%, which compensates for the distribution difference of the training data set. And after the completion of the training of the 11 tasks, the accuracy can also reach 84.21%, compared with the equal division of class number method of each task, the first task contains more classes which can achieve better effect, further alleviating the catastrophic forgetting problem.

[0118] Referring to Figure 5Since the Vision Transformer (ViT) is used as the backbone network, the total parameter amount of the network used by different methods is compared, and it can be seen that, compared with the DualPrompt method which also uses Prompt prompt information, the total parameter amount of the method (Ours) of the application is smaller under the same conditions, and compared with the DER based on model expansion, it is one time lower, which proves that the method (Ours) of the application is more suitable for the real environment and reduces the consumption of memory.

[0119] A pre-trained model-based class-incremental learning device, comprising:

[0120] A memory for storing a computer program for implementing the pre-trained model-based class-incremental learning method;

[0121] A processor for implementing the pre-trained model-based class-incremental learning method when executing the computer program.

[0122] A computer-readable storage medium, comprising:

[0123] The computer-readable storage medium stores a computer program, and the computer program can implement a pre-trained model-based class-incremental learning method when executed by a processor.

Claims

1. A pre-trained model-based incremental learning method, characterized in that, Includes the following steps; Step 1: In the first phase of incremental learning, for the first task D 1 The pre-trained model ViT is fine-tuned by embedding prompt information; the fine-tuned pre-trained model is obtained. Step 2: Merge and freeze the embedding layer functions in the fine-tuned pre-trained model obtained in Step 1 and the embedding layer functions of the pre-trained model ViT before fine-tuning to obtain the merged model; Step 3: Use the merged model obtained in Step 2 to perform class incremental training from scratch, and finally obtain a class incremental model with strong generalization and robustness; Step 4: For the test samples, use the class increment model obtained in Step 3 to make predictions and calculate the final classification accuracy; In step 1, the first task D 1 The dataset uses the top 10 classes on CIFAR100, the top 20 classes on CUB200, ImageNetR, and ImageNetA. The pre-trained model ViT is fine-tuned through the first task to compensate for domain differences with the ImageNet21k dataset. The specific process of fine-tuning the pre-trained model in step 1 is as follows: Step 1.1: For T types of incremental tasks {D 1 D 2 ,…,D T }, It is the t-th type of incremental task, which includes N t One sample, It is the i-th sample of the t-th class of incremental tasks. It is the label value of the i-th sample in the t-th class incremental task, for the first task. The input image x is used to obtain the query value q(x) through the query function. Here, the query value refers to the global features of the input image x extracted by the pre-trained model ViT without fine-tuning. Step 1.2: First, obtain the Hadamard product between the attention vector A and the query value q(x), then calculate the cosine similarity between the Hadamard product and the key K to obtain the weight, as shown in the following formula: α=cos(q(x)⊙A,K) α={cos(q(x)⊙A1,K1),cos(q(x)⊙A2,K2),…,cos(q(x)⊙A M ,K M )} Both the attention vector A and the key values ​​K are a set of learnable parameters, where each key value K corresponds to an attention vector A. D represents the dimension of the embedding layer vector of the pre-trained model ViT, M represents the length of the Prompt pool, ⊙ is the Hadamard product, and cos(·) is used to calculate the cosine similarity. Step 1.3: Calculate the weight α obtained in Step 1.2 and the prompt message Prompt, and sum them to obtain the prompt message for the new task. The formula is as follows: α m This represents the m-th weight. L represents the current Prompt message. p The size of the prompt message is defined as D, where D represents the dimension of the pre-trained ViT embedding layer vectors, and M represents the length of the prompt pool. Step 1.4: The embedding function φ(·) of the pre-trained model ViT divides the input image x into L patches for flattening, and then linearly projects it into a high-dimensional space using the fully connected layer of the pre-trained ViT. Then, add it to the positional encoding to obtain the input embedding feature of the self-attention layer. Query h Q , key h K value h V , where h = h Q =h K =h V The output of the multi-head self-attention layer is expressed by the formula: MSA(h Q ,h K ,h V )=concat(h1,…,h n )W O h i =Attention(h Q W i Q ,h K W i K ,h V W i V ) The W here O W i Q W i K W i V Both are projection matrices, where n is the number of heads in the multi-head self-attention layer, concat(·) indicates making a connection, and Attention(·) represents the head self-attention layer in ViT; The cue information p obtained in step 1.3 is embedded into the multi-head self-attention layer of the pre-trained model ViT, as shown in the following formula: f(p,h)=MSA(h Q ,[p K ;h K ],[p V ;h V ]) Here, f(·) represents the output of the self-attention layer, which divides the cueing information p into two equal parts. Step 1.5 Input image x into a multi-head self-attention layer f embedded with the prompting information obtained in Step 1.

4. p,h After (·), we proceed to the final classifier layer. In the prediction process, query loss is incorporated for optimization, resulting in the final loss function, expressed by the formula: Here, L(·) represents the cross-entropy classification loss, λ is the balancing hyperparameter, and cos(·) is used to calculate the cosine similarity. Step 1.6 Backpropagates the loss function obtained in Step 1.5 to update the prompt information Prompt, the key value K, and the attention vector A, ultimately obtaining a fine-tuned pre-trained model, thus obtaining the embedding function φ in the model. * (·), Embedded function φ * (x) will extract the patch features and global information of the input image x; The specific process of merging and freezing the model embedding layer functions in step 2 is as follows: After obtaining the fine-tuned pre-trained model, the embedding function φ will be... * The embedding function φ(·) of the original pre-trained model is merged with the embedding function φ(·) of the original model. Since the two have the same dimension, they are concatenated directly along the last dimension to obtain the merged model embedding function [φ]. * (·),φ(·)], freezes the embedding layer function, and stops updating the parameters of the embedding function during training, setting the requirements_grad of the embedding function parameters to false.

2. The incremental learning method based on a pre-trained model according to claim 1, characterized in that, The specific process of incremental training in step 3 is as follows: Step 3.1 Obtain the incremental task class {D} 1 D 2 ,…,D T }, the embedding layer function [φ] of the merged model obtained through step 2 * [·),φ(·)], extract the prototype vector for each class, as shown in the following formula: Here In incremental task D t p represents the total number of samples of category i1. i1 It is the prototype vector of the final category i1; Step 3.2 Replace the classifier weights with the prototype vectors of each category obtained in Step 3.1, as shown in the following formula: oh i1 =p i1 Step 3.3 Use the classifier obtained in Step 3.2 to predict the samples and obtain the loss function, as shown in the following formula: Here, L(·) represents the cross-entropy classification loss.

3. The incremental learning method based on a pre-trained model according to claim 2, characterized in that, The specific process of step 4, which uses an incremental model for prediction, is as follows: For a new test sample x, use the merged model embedding function [φ] * The corresponding embedding vector is calculated using (·),φ(·)], and then the result is compared with the prototype in the classifier using cosine similarity. The category of the prototype feature closest to it is the category of the test sample.

4. A pre-trained incremental learning system for implementing the method according to any one of claims 1-3, characterized in that, This includes a model fine-tuning module, a model merging and freezing module, and a classifier module; The fine-tuning model module uses a prompt embedding method to fine-tune the pre-trained model ViT; The model merging and freezing module first merges the embedding layer function of the fine-tuned pre-trained model and the embedding layer function of the original pre-trained model, and then freezes them. The classifier module uses the prototype vector of each class extracted by the merged model to replace the weights of the classifier to predict the samples.

5. An incremental learning-like device based on a pre-trained model, comprising: Memory: for storing a computer program that implements the class incremental learning method based on a pre-trained model as described in any one of claims 1-3; Processor: Used to implement the class incremental learning method based on a pre-trained model when executing the computer program.

6. A computer-readable storage medium comprising: The computer-readable storage medium stores a computer program that, when executed by a processor, can implement the class incremental learning method based on a pre-trained model according to any one of claims 1-3.

Images