A continuous learning method based on prototype-like prompts and virtual logical values

Abstract

The present invention discloses a continuous learning method based on class prototype prompts and virtual logic values, and relates to the technical field of pattern recognition. The present invention trains a task-specific prompt for each task, and proposes a prompt query mechanism based on class prototypes in terms of the method for selecting and adding prompts, and adds prompt momentum to the originally fixed query function to update, eliminating the problem of uneven addition of prompts. At the same time, this method also proposes to add virtually generated logic to the logic output by the classifier, so that the classifier has a better classification boundary, achieving the purpose of solving the confusion problem between tasks. The present invention solves the problem of uneven prompt query in the existing methods, and the problem of failing to fully alleviate the confusion between tasks, thereby solving the problem of serious catastrophic forgetting.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition, and particularly relates to a continuous learning method based on class prototype prompting and virtual logical values. Background Art

[0002] Continuous learning enables a single model to acquire knowledge from a series of tasks in a non-static data stream, which can not only effectively classify the categories of new tasks but also maintain good performance on old tasks, thus avoiding catastrophic forgetting. However, when the model parameters are fixed during the learning process, the model can only generalize to known categories. If the model parameters are updated with new category samples, the model often has difficulty maintaining its classification performance on old tasks. This limitation severely restricts the potential of deep learning models in practical applications.

[0003] In recent years, the success of pre-trained models and fine-tuning techniques has promoted prompt-based methods as an emerging continuous learning strategy, which outperforms traditional continuous learning methods. Traditional methods usually do not rely on a prompt pool but adopt other strategies to prevent catastrophic forgetting. For example, parameter regularization-based methods enhance the stability of the model by imposing constraints on important parameters, but this leads to a serious overlap of features between new and old categories, making it difficult to distinguish. Data replay-based methods allow saving a small amount of old category data and inputting it together with new data into the model for training. This method is simple and efficient, but it is sensitive to old data and may cause privacy problems. Network structure-based methods dynamically expand the network structure during the incremental learning process, but most of these methods require adding a large number of additional parameters, resulting in a high computational complexity and significant spatio-temporal overhead.

[0004] Existing prompt-based continuous learning methods usually incrementally learn a set of global prompts or task-specific prompts during the training process and, during inference, calculate the cosine distance between the query features extracted by the pre-trained Vision Transformer (ViT) and the query keys through a key-value pair query mechanism, and select appropriate prompts from the prompt pool to input into the ViT together with the samples for classification. For example, the L2P method first introduced prompt learning into continuous learning, trained a prompt pool containing multiple prompt words, and dynamically selected prompt combinations for different samples based on the key-value pair query mechanism. The DualPrompt method further proposed training two types of prompts: global prompts and task-specific prompts, which encode global knowledge and task local knowledge respectively, and the two work together to better complete downstream tasks. However, these methods usually adopt a fixed key-value pair query mechanism to select and optimize fixed prompts from the prompt pool, which is prone to falling into local optima. In addition, their query functions usually directly use the rough query features extracted by the pre-trained ViT model, and these features are difficult to accurately represent each sample, resulting in different categories may tend to use the same prompt.

[0005] Existing methods also generally train classifiers separately for each task during training and combine all classifiers for classification during testing. Since the classifiers are trained independently, the prediction scores between different tasks cannot be directly compared. This may result in the highest-scoring classifier selected not being the optimal one even if the prompt assignment is correct (i.e., the problem of confusion between tasks).

[0006] In summary, the current prompt-based continual learning methods have not effectively solved the problem of uneven prompt queries, and have not fully alleviated the confusion between tasks, thus still suffering from severe catastrophic forgetting. Summary of the Invention

[0007] Aiming at the above deficiencies in the prior art, a continual learning method based on class prototype prompts and virtual logical values provided by the present invention solves the problems of uneven prompt queries in existing methods, as well as the failure to fully alleviate the confusion between tasks, thus suffering from severe catastrophic forgetting.

[0008] To achieve the above invention purpose, the technical solution adopted by the present invention is: a continual learning method based on class prototype prompts and virtual logical values, including the following steps:

[0009] S1: In the query function fine-tuning stage, the images of the current task and the introduced prompts are input into a pre-trained encoder for training, and the classifier parameters and prompts are updated by backpropagation of the cross-entropy loss of the training samples;

[0010] S2: Based on the updated prompts and the prompts in the query function of the previous task, the prompts are updated using momentum, and combined with the pre-trained encoder, the query function of the current task is obtained;

[0011] S3: Based on the query function of the current task and the images of the current task, the query features of the training samples are extracted, and a class prototype is calculated for each class sample of the current task through the query features, and the class prototypes are stored in the prototype pool;

[0012] S4: In the prompt fine-tuning stage based on virtual logical values, the images of the current task and the prompts are input into another pre-trained encoder, the cross-entropy loss of the training samples is calculated by adding virtual logical values, and the classifier and prompts are updated by backpropagation according to the cross-entropy loss, and the updated prompts are stored in the prompt pool;

[0013] S5: Repeat the above steps S1 - S4 until the training of the current task is completed, and based on the stored class prototypes and prompts, the test samples of the current task are used for testing to obtain the classification results, and the continual learning based on class prototype prompts and virtual logical values is completed.

[0014] Further, the S1 includes the following sub-steps:

[0015] S11: Initialize the model parameters, and input the image of the current task and the introduced prompt into the pre-trained encoder to obtain image features;

[0016] S12: Input the image features into the classifier, and obtain the output obtained by forward propagation through the classifier. The formula is:

[0017]

[0018] where, is the output obtained by forward propagation, is the initialized classifier parameters, is the query function of the current task and is the sample;

[0019] S13: Based on the output obtained by forward propagation, calculate the first cross-entropy loss of each sample , and update the classifier and the prompt through backpropagation. The formula is:

[0020]

[0021] where, is the label corresponding to the sample , is the model classifier parameters of the current task. The superscript - distinguishes the query function fine-tuning stage and the prompt fine-tuning stage based on virtual logical values, is the prompt updated by the current task through backpropagation, is the process of backpropagation update, is the query function of the previous task, is the model classifier of the previous task,

[0022] is the training set.

[0022] Furthermore, in S2, based on the updated prompt and the prompt in the query function of the previous task, use momentum to update the prompt. The formula is:

[0023]

[0024] where, is the prompt updated by momentum, is the prompt in the query function of the previous task, is the momentum.

[0025] Furthermore, in S3, calculate a class prototype for each class sample of the current task through the query features. The formula is:

[0026] ​

[0027] Among them, is the class prototype of the th category of the current task, and is the total number of samples in the category . is the feature extracted corresponding to the th sample in the th category, is the query function of the current task updated by momentum, is the feature of the th sample in the th category.

[0028] Furthermore, the S4 includes the following sub-steps:

[0029] S41: Input the image and prompt of the current task into another pre-trained encoder to obtain image features;

[0030] S42: Input the image features into a classifier, and obtain the output logical value obtained by forward propagation through the classifier. The formula is:

[0031]

[0032] Among them, is the output logical value obtained by forward propagation, is the sample;

[0033] S43: Use a virtual logical value generator to generate a virtual logical value , and add the virtual logical value to the output logical value obtained by forward propagation to obtain an extended logical value . The formula is:

[0034]

[0035] Among them, represents the concatenation operation, is the output logical value of the sample corresponding to the th category, is the total number of categories of the current task;

[0036] S44: Use the extended logical value to calculate the second cross-entropy loss of each sample, and update the classifier and prompt through backpropagation. The formula is:

[0037]

[0038] Among them, is the hint updated by backpropagation, is the process of backpropagation update, is the encoder after adding the hint for the previous task ; is the training set, is the second cross-entropy loss;

[0039] S45: According to the obtained by the update, store it in the hint pool, and the formula is:

[0040]

[0041] Among them, is the set of hints for all tasks, is the hint for all tasks.

[0042] Furthermore, the virtual logical value in S43 is:

[0043]

[0044] Among them, is the virtual logical value generator, is the operation of taking a random value, and are hyperparameters.

[0045] Furthermore, in S5, based on the stored class prototypes and hints, use the test samples of the current task to perform tests to obtain classification results, including the following sub-steps:

[0046] S51: Select a test sample from the test set, and input the test sample into the query function

[0047] updated by momentum to extract its query features; S52: Calculate the distance between the extracted query features and the class prototypes stored in the prototype pool, and select the task label

[0048]

[0049] corresponding to the nearest neighbor class prototype, and the formula is: represents the Euclidean distance, means first fixing the task and minimizing for the class , then minimizing for the task finally to obtain the task label corresponding to the nearest neighbor class prototype;

[0050] S53: Based on the task label Select the corresponding prompt stored in the prompt pool , and use the prompt together with the test sample and input them into the model to obtain features ;

[0051] S54: Input the features into the classifier to obtain the classification result.

[0052] The beneficial effects of the present invention are as follows: A continuous learning method based on class prototype prompts and virtual logical values provided by the present invention trains a task-specific prompt for each task, proposes a class prototype-based prompt query mechanism in the aspect of selecting and adding prompts, and adds prompt momentum to update the originally fixed query function, eliminating the problem of uneven addition of prompts. At the same time, this method also proposes to add virtual generated logic to the logic output by the classifier, so that the classifier has a better classification boundary, achieving the purpose of solving the problem of confusion between tasks. Description of the Drawings

[0053] Figure 1 is a flowchart of a continuous learning method based on class prototype prompts and virtual logical values.

[0054] Figure 2 is a model structure diagram of a continuous learning method based on class prototype prompts and virtual logical values. Detailed Embodiments

[0055] The following further describes the present invention in conjunction with the drawings and specific embodiments.

[0056] As Figure 1 shown, a continuous learning method based on class prototype prompts and virtual logical values includes the following steps:

[0057] S1: In the fine-tuning stage of the query function, input the images of the current task and the introduced prompts into the pre-trained encoder for training, and update the classifier parameters and prompts through the cross-entropy loss of the training samples by backpropagation;

[0058] S2: Based on the updated prompts and the prompts in the query function of the previous task, update the prompts using momentum, and combine with the pre-trained encoder to obtain the query function of the current task;

[0059] S3: Based on the query function of the current task and the images of the current task, extract the query features of the training samples, calculate a class prototype for each class sample of the current task through the query features, and store the class prototype in the prototype pool;

[0060] Among them, the stored prototypes will be used as keys to query specific prompts, and are used to find the prompts that need to be added to the input image during the test phase;

[0061] S4: In the prompt fine-tuning phase based on virtual logical values, input the image and prompt of the current task into another pre-trained encoder, calculate the cross-entropy loss of the training samples by adding virtual logical values, and update the classifier and prompts through backpropagation according to the cross-entropy loss. At the same time, store the updated prompts in the prompt pool;

[0062] Among them, the stored prompts will be input into the model together with the image during the test phase to obtain more refined features to instruct the model to make predictions;

[0063] S5: Repeat the above steps S1 - S4 until the training of the current task is completed, and based on the stored class prototypes and prompts, use the test samples of the current task for testing to obtain the classification results, and complete the continual learning based on class prototype prompts and virtual logical values.

[0064] Figure 2 The figure shows an overview of the training phase of the method of the present invention. We divide the entire training process into two phases: the query function fine-tuning phase and the prompt fine-tuning phase based on virtual logical values. The specific steps are as follows:

[0065] The S1 includes the following sub-steps:

[0066] S11: Initialize the model parameters, and input the image of the current task and the introduced prompts into the pre-trained encoder to obtain image features;

[0067] Model parameter initialization: Obtain the encoder of the pre-trained ViT and , the randomly initialized prompt and , the initialized classifier and , where represents the parameters of the model encoder (feature extractor), represents the parameters of the classifier, and the subscript represents the task phase ( ), where 0 represents the initialization phase, and the overline represents the relevant parameters used in the query function fine-tuning phase, which are distinguished from the parameters of the second training phase. The number of tasks , the training set , the number of training epochs ;

[0068] As shown in the query function fine-tuning module in Figure 2 , for a certain task , obtain a mini-batch from the training set , for a sample input in each mini-batch , where is the sample, is the corresponding label, and calculate the output obtained by forward propagation;

[0069] S12: Input the image features into the classifier, and obtain the output obtained by forward propagation through the classifier. The formula is:

[0070]

[0071] where is the output obtained by forward propagation, is the initialized classifier parameter, is the query function of the current task , is the sample;

[0072] S13: Based on the output obtained by forward propagation, calculate the first cross-entropy loss of each sample , and update the classifier and the prompt through backpropagation. The formula is:

[0073]

[0074] where is the sample corresponding label, is the model classifier parameter of the current task, and the superscript - distinguishes the query function fine-tuning stage and the prompt fine-tuning stage based on virtual logical values, is the prompt updated by the current task through backpropagation, is the current task is the prompt updated through backpropagation, is the process of backpropagation update, is the query function of the previous task, is the model classifier of the previous task, is the training set.

[0075] In S2, based on the updated prompt and the prompt in the previous task query function, use momentum to update the prompt. The formula is:

[0076]

[0077] where is the prompt updated by momentum, is the prompt in the previous task query function, is the momentum.

[0078] When​ When it is, directly take . After updating to obtain the prompt , combined with the pre-trained ViT encoder , obtain the query function for the current task (i.e., the structure of the blue dotted line part of the middle module). Figure 2

[0079] Based on these query functions, calculate a class prototype for each category. Specifically, for a given category , use the query function to calculate the sample features of this category, and take the mean of the features as the class prototype .

[0080] In S3 above, calculate a class prototype for each category sample of the current task through query features, and the formula is:

[0081]

[0082] Among them, is the class prototype of the th category of the current task, , is the total number of samples in the category , is the th sample in the th category corresponding to the extracted feature, is the query function updated by momentum for the current task , is the th sample in the th category of features.

[0083] S4 above includes the following sub-steps:

[0084] S41: Input the image and prompt of the current task into another pre-trained encoder to obtain image features;

[0085] S42: Input the image features into the classifier, and obtain the output logical value obtained by forward propagation through the classifier. The formula is:

[0086]

[0087] Among them, is the output logical value obtained by forward propagation, is the sample;

[0088] According to the output logical value obtained by forward propagation, it can be further expressed as:​

[0089]

[0090] S43: Generate a virtual logical value using a virtual logical value generator , and add the said virtual logical value to the output logical value obtained from the forward propagation to obtain an extended logical value , the formula is:

[0091]

[0092] wherein, represents the concatenation operation, is the output logical value of the sample corresponding to the rd category, is the total number of categories of the current task;

[0093] S44: Calculate the second cross-entropy loss of each sample using the extended logical value , and update the classifier and the hint through backpropagation, the formula is:

[0094]

[0095] wherein, is the hint updated through backpropagation, is the process of backpropagation update, is the encoder after adding the hint of the previous task , is the training set, is the second cross-entropy loss;

[0096] S45: According to the updated , store it in the hint pool, the formula is:

[0097]

[0098] wherein, is the set of hints for all tasks, is the hint for all tasks.

[0099] The virtual logical value in the said S43 is:

[0100]

[0101] wherein, is the virtual logical value generator, is the operation of taking a random value, and are hyperparameters.

[0102] Through this virtual logical value generator, virtual logical values can be generated within the range of In this embodiment, the virtual logic generator can generate virtual logical values within the range of These virtual logical values are similar in magnitude to the logical values generated by the classifier.

[0103] In step S5, based on the stored class prototypes and prompts, the test samples of the current task are used for testing to obtain classification results, including the following sub-steps:

[0104] S51: Select a test sample from the test set , and input the test sample into the query function updated by momentum to extract its query features;

[0105] S52: Calculate the distance between the extracted query features and the class prototypes stored in the prototype pool, and select the task label corresponding to the nearest neighbor class prototype , and the formula is:

[0106]

[0107] where represents the Euclidean distance, means first fixing the task and minimizing for the class , and then minimizing for the task to finally obtain the task label corresponding to the nearest neighbor class prototype ;

[0108] S53: Based on the task label select the corresponding prompt stored in the prompt pool , and input the prompt and the test sample into the model to obtain the feature ;

[0109] S54: Input the feature into the classifier to obtain the classification result.

[0110] In the test phase of this embodiment, take a sample from the test set , first input it into the query function obtained from Equation to extract its query features. In the second step, calculate the distance between the query features and the stored class prototypes, and select the task label corresponding to the nearest neighbor class prototype:

[0111]

[0112] In the third step, based on this task label select the corresponding prompt , and input it together with the test sample into the model to obtain features . Finally, input the features into the combined classifier to obtain the classification result.

[0113] When the number of test sets (up to the current task ) has all been tested, this is the end of all training and testing processes for the current task . Until , the training and testing of all tasks are completed.

[0114] In an embodiment of the present invention, the embodiments of the present invention are implemented on four publicly available common datasets: 10-SplitImageNet-R, 20-Split ImageNet-R, 10-Split CIFAR-100, and 10-Split DomainNet. The ViT model pre-trained on ImageNet-21k is used. The length of the prompt is 8. The number of training epochs on ImageNet-R is 100, and on CIFAR-100 and DomainNet is 50. The mini-batch size is uniformly 150. For the virtual outlier parameter, take , .

[0115] The comparison with other methods on 10-Split ImageNet-R, 20-Split ImageNet-R, 10-Split CIFAR-100, and 10-SplitDomainNet is shown in Table 1, where n-split represents dividing different datasets into n consecutive class incremental learning tasks. The evaluation metrics are the final accuracy (Last Accuracy) and the final forgetting rate (LastForgetting), that is, the average accuracy and forgetting rate after the last task is completed. The values after

[0116] represent the standard deviation obtained from multiple experiments. The baseline method in the table is the method that does not use momentum update to fine-tune the query function and the virtual outlier strategy. It can be seen from the table that the present invention improves the accuracy by 0.85% - 4.02% and reduces the forgetting rate by 0.40% - 1.17% compared with the baseline method, and exceeds other advanced methods in terms of accuracy, achieving the best performance.

[0117]

[0118] The present invention was compared with baselines and other methods in long-term (multi-task incremental learning) experiments. Table 2 shows the comparison results of the present invention with baselines and other methods on 50 tasks. It can be seen from the table that the present invention still achieves the best performance on 50 tasks and the improvement is more obvious.

[0119] Table 2 Comparison with other methods on 50 tasks (%)

[0120]

[0121] Table 3 shows the comparison results of the present invention with baselines and other methods on 100 tasks. It can be seen from the table that when the number of tasks increases again, the present invention can still achieve better performance on the two datasets in the table, and there is a more obvious improvement compared with other methods, while obvious catastrophic forgetting phenomena occur in other methods. This shows that the present invention has more obvious advantages in long-term class incremental learning and has the potential for practical applications.

[0122] Table 3 Comparison with other methods on 100 tasks (%)

[0123]

[0124] Those of ordinary skill in the art will realize that the embodiments described herein are for helping readers understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations without departing from the essence of the present invention according to the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the invention.

Claims

1. A continuous learning method based on class prototype prompts and virtual logic values, characterized in that: The following steps are involved: S1: In the query function fine-tuning phase, the image of the current task and the introduced prompt are input into the pre-trained encoder for training, and the classifier parameters and prompts are updated through the cross-entropy loss backpropagation of the training samples; The S1 includes the following sub-steps: S11: Initialize the model parameters and input the image of the current task and the introduced prompts into the pre-trained encoder to obtain image features; S12: Input the image features into the classifier, and obtain the output obtained by forward propagation through the classifier; S13: Based on the output obtained by forward propagation, calculate the first cross entropy loss of each sample, and update the classifier and prompt through back propagation; S2: Based on the updated hint and the hint in the previous task query function, the hint is updated using momentum and combined with the pre-trained encoder to obtain the query function of the current task; S3: Based on the query function of the current task and the image of the current task, the query features of the training samples are extracted, and a class prototype is calculated for each category sample of the current task through the query features, and the class prototype is stored in the prototype pool; S4: In the prompt fine-tuning stage based on virtual logical values, the image and prompt of the current task are input into another pre-trained encoder, the cross entropy loss of the training sample is calculated by adding the virtual logical value, and the classifier and prompt are updated by backpropagation according to the cross entropy loss, and the updated prompt is stored in the prompt pool; The S4 includes the following sub-steps: S41: Input the image and prompt of the current task into another pre-trained encoder to obtain image features; S42: inputting the image features into the classifier, and obtaining the output logic value obtained by forward propagation through the classifier; S43: Generate a virtual logic value using a virtual logic value generator, and add the virtual logic value to the output logic value obtained by forward propagation to obtain an extended logic value; S44: Calculate the second cross entropy loss for each sample using the extended logistic value, and update the classifier and prompt through back propagation; S45: storing the updated prompt in a prompt pool; S5: Repeat the above steps S1-S4 until the training of the current task is completed, and based on the stored class prototypes and prompts, use the test samples of the current task to test, obtain classification results, and complete continuous learning based on class prototype prompts and virtual logical values; In S5, based on the stored class prototypes and prompts, the test samples of the current task are used for testing to obtain the classification results, including the following sub-steps: S51: Select a test sample from the test set , and the test sample Input to the query function updated by momentum Extract its query features; S52: Calculate the distance between the extracted query features and the class prototypes stored in the prototype pool, and select the task label corresponding to the nearest neighbor class prototype , the formula is: in, represents the Euclidean distance, Indicates that the task is fixed first For Category Minimize and then re-assign the task Minimize, and finally get the task label corresponding to the nearest neighbor prototype , For the current task No. The class prototype of each category; S53: Based on task tags Select the corresponding hint stored in the hint pool , and will prompt With test samples Input into the model In the ; S54: Features Input into the classifier to obtain the classification result.

2. A continuous learning method based on class prototype prompts and virtual logic values ​​according to claim 1, characterized in that: The S1 includes the following sub-steps: S11: Initialize the model parameters and input the image of the current task and the introduced prompts into the pre-trained encoder to obtain image features; S12: Input the image features into the classifier, and obtain the output obtained by forward propagation through the classifier. The formula is: in, is the output obtained by forward propagation, are the initialized classifier parameters, For the current task The query function, For samples; S13: Based on the output obtained by forward propagation, calculate the first cross entropy loss of each sample , and update the classifier and prompt through back propagation, the formula is: in, For sample The corresponding label, For the current task The model classifier parameters of , the superscript - distinguishes the query function fine-tuning stage from the prompt fine-tuning stage based on the virtual logical value, For the current task By back-propagating the updated hint, is the process of back-propagation update, is the query function of the previous task, is the model classifier of the previous task, For the training set.

3. A continuous learning method based on class prototype prompts and virtual logic values ​​according to claim 2, characterized in that: In S2, based on the updated prompt and the prompt in the previous task query function, the momentum is used to update the prompt, and the formula is: in, As a reminder of the momentum update, Query the prompt in the previous task function. For momentum.

4. A continuous learning method based on class prototype prompts and virtual logic values ​​according to claim 3, characterized in that: In S3, a class prototype is calculated for each class sample of the current task by querying the features, and the formula is: in, For the current task No. The class prototype of the category, For Category The total number of samples in For the of the categories The samples correspond to the extracted features. For the current task The query function updated by momentum, For the of the categories Characteristics of a sample.

5. A continuous learning method based on class prototype prompts and virtual logic values ​​according to claim 4, characterized in that: The S4 includes the following sub-steps: S41: Input the image and prompt of the current task into another pre-trained encoder to obtain image features; S42: Input the image features into the classifier, and obtain the output logic value obtained by forward propagation through the classifier. The formula is: in, is the output logic value obtained by forward propagation, For samples; S43: Generate a virtual logic value using a virtual logic value generator , and the virtual logic value Add to the output logic value obtained by forward propagation to obtain the extended logic value , the formula is: in, Indicates a connection operation. The sample corresponds to The output logical value of each category, is the total number of categories of the current task; S44: Using extended logic values Calculate the second cross entropy loss for each sample , and update the classifier and prompt through back propagation, the formula is: in, is the hint updated by back-propagation, is the process of back-propagation update, Add a reminder for the previous task After the encoder, is the training set, is the second cross entropy loss; S45: Based on the update , store it in the prompt pool, the formula is: in, A collection of hints for all tasks. Hints for all tasks.

6. A continuous learning method based on class prototype prompts and virtual logic values ​​according to claim 5, characterized in that: The virtual logic value in S43 for: in, is a virtual logic value generator, To get a random value, and is a hyperparameter.

Images