Class incremental learning method of progressive prompt integrated network

Abstract

The invention discloses a class incremental learning method of a progressive prompt integrated network, the network gradually fuses global prompt and task prompt on different levels to adapt to an incremental learning task, and the task prompt assists image recognition through a voting strategy. In addition, a prompt query strategy is optimized through a metric learning mechanism, the prompt selection accuracy is improved, and task prompts are accurately selected by adjusting prompt keys. A large number of experiments are carried out on the two reference data sets, and compared with an existing class incremental learning method, the class incremental learning method of the progressive prompt integrated network further improves the image classification accuracy.

Description

Technical Field

[0001] The invention relates to the field of pattern recognition and machine learning, as well as the technical field of class incremental learning, and in particular to a class incremental learning method of a progressive prompt integration network. Background Art

[0002] Class incremental learning (CIL) is an important topic in machine learning, which aims to gradually learn new knowledge based on old knowledge. Unlike traditional machine learning methods, which assume that training data is provided all at once, CIL deals with a series of learning tasks, each of which introduces new categories to learn. Since the data of old tasks are not available when learning new tasks, the model tends to forget old knowledge. Therefore, catastrophic forgetting becomes a key challenge in the CIL process. Many methods have been proposed to address the problem of catastrophic forgetting. Traditional CIL methods are generally divided into three categories: structure-based methods, replay-based methods, and regularization-based methods.

[0003] Structure-based methods mainly add a new branch to the model for each new task. However, these methods increase the size of the model and require more inference time. Replay-based methods propose to keep representative samples of old tasks to replay old knowledge. As the number of tasks increases, this will require larger memory to store old data, limiting their application. Regularization-based methods use knowledge distillation techniques to retain old knowledge. However, regularization constraints may conflict with the learning of new knowledge, resulting in poor performance on new tasks.

[0004] With the rapid development of large-scale language models, pre-trained visual encoders and prompt learning have gained increasing attention in recent years. Some methods propose to use prompt learning techniques to effectively adapt pre-trained models to incremental learning tasks and use the rich knowledge in pre-trained models to alleviate catastrophic forgetting. L2P proposes a prompt pool and selects appropriate prompts to learn image features for recognition. DualPrompt designs global prompts and task prompts to learn more adaptive image features.

[0005] Although the prompt-based methods have made great progress, some problems still exist. First, the prompt selection strategies in current methods may also encounter catastrophic forgetting problems. Existing prompt selection methods only consider the similarity between image features and the current prompt key to learn the prompt key of the current task, ignoring the difference between image features and old prompt keys, which may lead to low accuracy of prompt selection. Second, the prompt information is not fully utilized. Current methods only use image category labels for classification, ignoring valuable prompts. Since prompts interact with image features multiple times through attention mechanisms, they also contain rich discriminative information, which can greatly assist image recognition. Summary of the invention

[0006] To address these challenges, the present invention proposes a novel progressive prompt integration network to achieve more effective class-incremental learning. Our framework improves the prompt selection strategy and integrates task prompts to assist image recognition. In terms of prompt selection, we utilize metric learning techniques to optimize the prompt keys of old tasks, which not only considers the similarity between image queries and corresponding task prompt keys but also the differences with other prompt keys, thereby improving the accuracy of prompt selection. In terms of prompt integration, we introduce a progressive prompt interaction framework and fuse task prompt information for image recognition. The image classification results are integrated with the original predictions through a voting strategy, thus improving the accuracy of the image recognition results.

[0007] The technical solution steps of the present invention are as follows:

[0008] Step 1: Progressive prompt integration;

[0009] S11 Input: Image data of the current task.

[0010] S12 Progressive prompts: Gradually add global prompts and task prompts to different layers of the pre-trained ViT model through prompt tuning techniques.

[0011] S13 Output: Select task prompts through metric learning techniques, add them to the pre-trained ViT model, and integrate task prompts through a voting strategy during the classification stage to assist image recognition and obtain the results of image recognition.

[0012] Step 2: Optimization of prompt keys

[0013] S21 When learning each new task, optimize the cosine distance between the query feature and the prompt key of the current task through the loss function.

[0014] S22 Based on the previous step, fine-tune the prompt keys of other tasks to make their cosine distance from the current query feature as far as possible.

[0015] Compared with the prior art, the present invention proposes a progressive prompt integration network, which helps the pre-trained model adapt to incremental learning tasks by gradually introducing global prompts and task prompts at different levels. Among them, task prompts are integrated through a voting strategy to assist image recognition. In addition, we optimize the prompt query strategy through a metric learning mechanism and adjust the prompt keys to achieve accurate task prompt selection.

[0016] Progressive prompt integration module

[0017] Different from DualPrompt that uses prefix tuning to achieve feature adaptation, we adopt prompt tuning to adapt backbone features to incremental learning tasks. In prefix tuning, a series of learnable vectors are added to each Transformer layer as additional keys (K) and values (V), which can be expressed as:

[0018] K′ = [P K ; K], V′ = [P V ; V]

[0019] where P K and P V represent the learnable prefix key and prefix value respectively. However, these prompts can only be used once in each layer and will disappear in subsequent layers, which may lead to information loss. In contrast, we adopt the prompt tuning technique, and the prompts are added layer by layer incrementally, so that the prompts can continue to be used for information interaction in subsequent layers. This not only promotes the accumulation and integration of information in the model, but also effectively identifies global information and task-specific information. Prompt tuning can be expressed as:

[0020] X′ = [P; X]

[0021] where P represents the learnable prompt and X represents the image token. The global prompt G is gradually added to the first few layers of the network, and they will integrate the low-level features shared by different classes to enhance the generalization ability. The task prompt T is gradually added to the remaining layers to integrate task-specific information for more effective image recognition. Therefore, we can integrate the task prompt information through a voting strategy to assist image recognition. We obtain the updated global prompt task prompt class token and image token We fuse the task prompt information through average pooling and use it to assist image recognition. In our framework, we use the class token and the task prompt to train the image classifier respectively, which is expressed as:

[0022]

[0023] where O c and O t are the outputs of the image classifier, f 1 and f 2 are the image classifiers, and avg represents average pooling. Finally, the classification loss can be expressed as:

[0024] L class = L CE (O c , y) + L CE (O t , y)

[0025] Here, L CE is the cross-entropy loss, and y is the target label.

[0026] Optimization Module for Prompt Keys

[0027] At the start of each new task, the keys of other task prompts are fine-tuned to be as far away as possible from the current sample features. Let the feature representation extracted from the training set by the pre-trained Vision Transformer model (ViT) be F, and let K denote the keys of the task prompts. Let i be the ID of the current task and j be the ID of other tasks. The loss function can be expressed as:

[0028]

[0029] where cos(F, K i ) represents the cosine similarity between F and K i , and α is the hyperparameter for fine-tuning the keys of other tasks. By minimizing the distance between F and the corresponding task prompt key K i , and maximizing the distance between F and the keys of other task prompts K j , the model can effectively select the appropriate task prompts for feature adaptation.

[0030] The final overall loss function is:

[0031] L = L class + βL key

[0032] where L class is the classification loss, L key is the prompt key optimization loss, and β is the hyperparameter.

[0033] Image Classification Phase

[0034] We select task prompts by measuring the similarity between the test image and the prompt keys. Then, we obtain the outputs O c and O t of the image classifier and the prompt classifier respectively, and penalties are imposed for other tasks. Next, we use the softmax function to obtain the class probabilities, and the final class prediction is determined by selecting the maximum response in these two probability distributions:

[0035] P c = softmax(O c ), P t = softmax(O t )

[0036]

[0037] where, Represents the maximum operation at the element level. In this way, we combine the prediction results of the image classifier and the prompt classifier through a voting strategy to improve the accuracy of the image prediction results.

[0038] The present invention proposes a progressive prompt integration network for class-incremental learning, which gradually integrates two types of prompts into a pre-trained model to promote efficient feature adaptation. To improve the accuracy of task prompt selection, we use metric learning methods to optimize the keys of old task prompts, thereby enhancing the distinguishability between image queries and different task prompt keys and achieving more accurate prompt selection in the test phase. In addition, we deeply mine the deeper information contained in the prompts, integrate the task prompt information to assist image recognition, and use a voting strategy to integrate information for prediction, thereby improving the classification accuracy of the model.

[0039] Our extensive experiments on the CIFAR-100 and ImageNet-R datasets show that compared with existing methods, the proposed progressive prompt integration network has achieved a significant improvement in picture classification accuracy. Brief Description of the Drawings

[0040] Figure 1 It is a specific implementation flowchart of the present invention.

[0041] Figure 2 Shows the performance of different incremental learning methods on Split CIFAR-100 and Split ImageNet-R under 10 tasks. A N is the average accuracy, and F N is the forgetting rate.

[0042] Figure 3 Compares the query matching accuracy of the progressive prompt integration network on the CIFAR-100 and ImageNet-R datasets with and without fine-tuning prompt keys. 'Tune' means optimizing the old prompt keys, 'Fix' means keeping the old prompt keys unchanged, and 'Acc' means the matching accuracy of the prompt keys.

[0043] Figure 4 Shows the performance of adding global prompts and task prompts on different layers of ViT, and experiments are conducted on the CIFAR-100 and ImageNet-R datasets respectively. 'G-layers' represents the number of layers where global prompts are added, and 'T-layers' represents the number of layers where task prompts are added. Detailed Description of the Invention

[0044] To accurately evaluate the present invention, we conducted extensive experiments on two class-incremental learning datasets: CIFAR100 and ImageNet-R. Both datasets were divided into 10 tasks for class-incremental learning. CIFAR-100 contains 60,000 images covering 100 categories, with 500 training images and 100 test images for each category. ImageNet-R contains 30,000 images from 200 ImageNet classes, with images in each category presenting different styles.

[0045] We compared our method with recent prompt-based class-incremental learning methods, including L2P, DualPrompt, CODA-Prompt, HiDe-Prompt, DAP, OS-Prompt, and PC. Additionally, we also compared it with other representative methods, such as regularization-based methods and replay-based methods.

[0046] We used two evaluation metrics to assess the methods: (1) Average Accuracy A N , which is the average of the accuracies of N tasks after training the last task; (2) Forgetting Rate F N , which is the average decline in task performance, also based on N tasks. We emphasize that A N is the more important metric, while F N is only used as a measure of catastrophic forgetting during the incremental learning process.

[0047] We compared this method with other methods. Figure 2 The experimental results on the CIFAR-100 and ImageNet-R datasets are shown. It can be seen that replay-based methods require a buffer to store images of old categories, and the performance improves as the buffer size increases. Prompt-based methods do not require a buffer and have higher performance due to their ability to effectively utilize the knowledge in the pre-trained model. Additionally, our method performs best compared to other prompt-based methods. Compared with DualPrompt, it improves the accuracy by 8.02% on ImageNet-R and 10.69% on CIFAR-100. Further, compared with recent methods such as PC and OS-Prompt, our method consistently maintains superior performance, clearly demonstrating its effectiveness on the CIFAR-100 and ImageNet-R datasets. Moreover, our method achieves the minimum forgetting rate on the CIFAR-100 dataset.

[0048] In the progressive prompt integration network, we use metric learning methods to optimize the old task prompt keys, aiming to improve the accuracy of task prompt matching. To verify the effectiveness of our method, we compared it with fixed prompt keys and conducted experiments on the CIFAR-100 and ImageNet-R datasets. The experimental results are as Figure 3 shown. The results show that on the CIFAR-100 dataset, the matching accuracy increased from 93.75% to 98.13%, and the classification accuracy increased from 94.52% to 97.20%, an increase of 2.68%. On the ImageNet-R dataset, the matching accuracy increased from 71.88% to 79.69%, and the classification accuracy increased from 73.80% to 76.15%, an increase of 2.35%. Overall, our fine-tuning mechanism did improve the matching accuracy between the query image and the corresponding key, ensuring that the correct task prompt was selected during the test, thus improving the accuracy of image classification.

Claims

1. A quasi-incremental learning method for a progressive prompt ensemble network, characterized in that: The implementation steps of this method are as follows: Step 1: Progressive prompt integration; S11 input: image data of the current task; S12 Progressive Hints: Global and task hints are gradually added to different layers of the pre-trained ViT model through hint tuning techniques; S13 output: The task prompt is selected through the metric learning technology and added to the pre-trained ViT model. In the classification stage, the task prompt is integrated through the voting strategy to assist image recognition and obtain the image recognition result; Step 2: Optimize the prompt key When learning each new task, S21 optimizes the cosine distance between the query feature and the prompt key of the current task through the loss function; S22 fine-tunes the prompt keys of other tasks based on the previous step S21 so that the cosine distance between them and the current query feature is as far as possible.

2. A quasi-incremental learning method for a progressive prompt integration network according to claim 1, characterized in that: The implementation process of progressive hint integration is as follows: Hint tuning is used to adapt backbone features to incremental learning tasks; in prefix tuning, learnable vectors are added to each Transformer layer as additional keys K and values ​​V, expressed as: K′=[P K ;K],V′=[P V [V] Among them, P K and P V Represent the learnable prefix key and prefix value respectively; the hint can only be used once in each layer and will disappear in the subsequent layers, which may lead to information loss; the hint tuning technology is adopted, and the hint is added gradually layer by layer, so that the hint continues to be used for information interaction in the subsequent layers; the hint tuning is expressed as: X′=[P;X] Among them, P represents the learnable hint and X represents the image tag; the global hint G will be gradually added to the first few layers of the network, which will integrate the low-level features shared by different classes to enhance the generalization ability; the task hint T will be gradually added to the remaining layers to integrate task-specific information for more effective image recognition; therefore, the task hint information is integrated through the voting strategy to assist image recognition; the updated global hint is obtained Task Tips Class Marker and image tags The task prompt information is fused through average pooling and used to assist image recognition; in the framework, class labels and task prompts are used to train image classifiers, respectively, as shown in the following figure: Among them, O c and O t is the output of the image classifier, f1 and f2 are image classifiers, and avg represents average pooling; finally, the classification loss can be expressed as: L class =L CE (O c ,y)+L CE (O t ,y) Here, L CE is the cross entropy loss and y is the target label.

3. The incremental learning method of a progressive prompt integration network according to claim 1, characterized in that: The optimization module implementation process of the prompt key is as follows: Let the feature extracted from the training set by the pre-trained visual transformer model ViT be represented as F, let K represent the key of the task prompt; let i be the ID of the current task, and j be the ID of other tasks; the loss function is expressed as: Among them, cos(F,K i ) represents F and K i The cosine similarity between them, α is a hyperparameter for fine-tuning other task keys; by minimizing F and the corresponding task prompt key K i The distance between them and maximize the F and other task prompt keys K j The distance between them, the model effectively selects task cues for feature adaptation; The final total loss function is: L=L class +βL key Among them, L class is the classification loss, L key is the key optimization loss and β is a hyperparameter.

4. The incremental learning method of a progressive prompt integration network according to claim 1, characterized in that: The implementation process of the image classification stage is as follows: The task prompt is selected by measuring the similarity between the test image and the prompt key; then, the outputs of the image classifier and the prompt classifier are obtained respectively. c and O t , other tasks will be penalized; next, the softmax function is used to obtain the category probability and the final category prediction Determined by choosing the maximum response between these two probability distributions: P c =softmax(O c ),P t =softmax(O t ) in, represents the maximum operation at the element level; in this way, the prediction results of the image classifier and the hint classifier are combined through a voting strategy, thereby improving the accuracy of the image prediction results.

Images