Method and device for establishing continuous learning model based on playback under dual-granularity prompts

Abstract

The present invention discloses a method and device for establishing a continuous learning model based on playback under dual-granularity prompts. The present invention first pre-processes images in a training data set; then constructs a continuous learning model based on playback under dual-granularity prompts, including a feature extractor, a classifier, an input perception prompt module and an agent feature prompt module; the feature extractor is used to extract features from input samples; the classifier is used to classify the features output by the feature extractor; the input perception prompt module is used for additional supplementation of network input, helping to generate more reasonable and diversified input distributions and improving the generalization ability of the network; the agent feature prompt module is used to bridge the knowledge gap between the teacher and student models to maintain consistency during the feature transfer process, and enhance the plasticity and stability of the features. The present invention can efficiently complete continuous learning tasks, and the image classification accuracy is better than that of existing continuous learning methods.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for establishing a continuous learning model based on playback under dual-granularity prompts. Background Art

[0002] Continuous learning plays a vital role in the field of computer vision, especially in scenarios where new knowledge is constantly emerging, such as smart security and autonomous driving. Continuous learning aims to train intelligent agents to simulate the ability of the human brain, and to have the ability to maintain or even integrate and optimize old knowledge while learning and absorbing new knowledge. In the era of big data, a large amount of data information flow is generated every day in the real world, so it is very important for intelligent agents to have the ability to continuously learn.

[0003] One of the most challenging directions in the field of continuous learning is incremental learning. Incremental learning aims to train the model to maintain the ability to distinguish known categories while continuously learning new categories. The main research content of incremental learning is image classification in incremental scenarios. The mainstream research methods today mainly maintain the ability to distinguish old categories in the incremental process by constantly reviewing old samples while learning new knowledge. However, such methods still have defects. On the one hand, a small number of samples from old tasks participate in the model training process together with the online data stream of the current task, which alleviates the model's excessive bias in learning the distribution of new tasks. But at the same time, the addition of old samples brings about changes in the input distribution and reduces the generalization ability of the network. On the other hand, due to the size limit of the rehearsal buffer, only a small number of samples of the old category are usually retained. In order to alleviate the model's forgetting of old knowledge, conventional methods usually directly perform hard feature distillation between the new and old models. In real scenarios, most of the network features between the new and old incremental stages have some differences. If these network features are strictly aligned, the plasticity of the model on new tasks will be significantly reduced. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a method and device for establishing a continuous learning model based on playback under dual-granularity prompts.

[0005] Technical solution: The method for establishing a continuous learning model based on playback under dual-granularity prompts of the present invention includes:

[0006] Step S1: dividing a data set containing a number of images to be classified into a training data set and a test data set, and determining a learning order for images of different categories in the data set;

[0007] Step S2: preprocessing the images in the training data set;

[0008] Step S3: construct a playback-based continuous learning model under dual-granularity prompts, including a feature extractor, a classifier, an input perception prompt module and an agent feature prompt module; the feature extractor is used to extract features from input samples, and is specifically formed by stacking several convolutional layers; the classifier is used to classify the features extracted by the feature extractor, and is specifically composed of a fully connected layer; the input perception prompt module is used for additional supplementation of network input; the agent feature prompt module is used to bridge the knowledge gap between the teacher and student models;

[0009] Step S4: obtaining the preprocessed training data in sequence, and training the playback-based continuous learning model under dual-granularity prompts;

[0010] Step S5: Use deep copy to save a copy that is independent of the current model as the old network for the next incremental stage;

[0011] Step S6: Use the preprocessed test data set to test the performance of the model and obtain the average classification accuracy of the model for all currently known category images;

[0012] Step S7: Return to step S4 and execute until the training of all classes is completed, thereby obtaining a replay-based continuous learning model under dual-granularity prompting that has completed training.

[0013] Furthermore, in step S1, the learning order of determining the samples of different categories in the data set is specifically as follows:

[0014] The images in the dataset are divided into a series of tasks according to their categories. Each task contains an equal number of categories. Task one is trained first, and the remaining tasks are trained in sequence.

[0015] Furthermore, the input perception prompt module in step S3 specifically includes:

[0016] The attention module focuses on key objects or regions in the image, learning and paying attention to objects or features relevant to the task;

[0017] A fully connected layer is used to perform a nonlinear transformation on the input attention vector to generate a weight vector equal to the number of prompts in the prompt pool;

[0018] The input-aware prompt pool contains a series of randomly initialized learnable prompts, which are weighted and summed with the weight vector to generate input-aware prompts for additional supplementation of network input.

[0019] Furthermore, the agent feature prompt module in step S3 specifically includes:

[0020] The proxy feature hint pool contains a set of randomly initialized learnable hints, which are used to subsequently generate proxy feature hints to bridge the knowledge gap between the teacher and student models;

[0021] Convolutional layer, used to capture information between different channels and reduce the channel dimension to the same number of categories currently known to the model;

[0022] A fully connected layer is used to perform a nonlinear transformation on the learnable prompt vector to generate a weight vector equal to the number of categories currently known to the model;

[0023] The projection layer is used to transform the new class features into a learnable transformation of the old class features to ensure the linear separability of the new class and the old class.

[0024] Furthermore, the step S4 specifically includes:

[0025] Step S4-1: Setting training parameters. If the current stage is the base training stage, the continuous learning model described in S3 is randomly initialized. If it is the incremental training stage, the network parameters obtained by the previous stage training are directly used for initialization.

[0026] Step S4-2: For the data in the base training phase or the data belonging to the new class in the incremental training phase, these data are sent to the current network and the cross entropy loss L of the new class samples is performed. ce (D t );

[0027] Step S4-3: If it is not the base training stage, select old class samples with the same amount of data as the current new class from the rehearsal buffer, send them to the current network, and perform the cross entropy loss L of the old class samples respectively. rehearsal-ce (S) and distillation loss L rehearsal-logit (S);

[0028] Step S4-4: If it is not the base training stage, combine the new class samples and the old class samples, and send them to the network trained in the current stage and the frozen network in the previous stage respectively, and perform the proxy feature prompt loss L on the features of the network trained in the current stage and the network in the previous stage f (D t ,S);

[0029] Step S4-5: If it is the base training stage, use the loss calculated in step S4-2 to optimize the network parameters. If it is not the base training stage, combine the losses calculated in steps S4-2 to S4-4 to calculate the total network loss L(D t ,S), optimize the network parameters;

[0030] Step S4-6: Use the class-balanced reservoir sampling algorithm to select a certain number of samples from the samples of the new class currently being trained and update the rehearsal buffer.

[0031] Furthermore, the method for calculating the cross entropy loss of the new class samples in step S4-2 specifically includes:

[0032] Step S4-2-1: Calculate the weight vector according to the following formula:

[0033] α=softmax(FC(Atn(x)))

[0034] In the formula, Atn represents the attention module, FC refers to the fully connected layer, softmax represents the SoftMax activation function, and α represents the final set of weight vectors corresponding to the input x;

[0035] Step S4-2-2: Calculate the input perception prompt according to the following formula:

[0036] p=∑α m P m

[0037] In the formula, p represents the final input perception prompt, α m is the mth weight vector in the weight vector set, P m is the mth cue in the cue pool;

[0038] Step S4-2-3: Calculate the cross entropy loss L of the new class sample according to the following formula ce (D t ):

[0039]

[0040] Where D t represents the tth task, x, y represent the input sample and its corresponding label, cross-entropy represents the cross entropy loss, h θ represents the current model parameterized by θ.

[0041] Furthermore, the cross entropy loss L of the old class samples in step S4-3 is rehearsal-ce (S) and distillation loss L rehearsal-logit The calculation method of (S) specifically includes:

[0042] Step S4-3-1: Calculate the cross entropy loss β of the old class samples according to the following formula Lrehearsal-ce (S):

[0043]

[0044] Where S represents the rehearsal buffer, cross-entropy represents the cross entropy loss, x′, y′ represent the samples from the rehearsal buffer and their corresponding labels, respectively, and h θ represents the current model parameterized by θ;

[0045] Step S4-3-2: Calculate the distillation loss L according to the following formula: rehearsal-logit (S):

[0046]

[0047] Where h′ represents the model saved in the previous incremental stage.

[0048] Furthermore, the calculation method of the proxy feature prompt loss in step S4-4 specifically includes:

[0049] Step S4-4-1: Calculate the proxy feature hint according to the following formula:

[0050] B=∑FC′(β n )Conv(β n )

[0051] In the formula, B represents the proxy feature prompt, FC′ represents a series of fully connected layers, Conv represents the convolution operation, and β n represents the nth cue in the agent feature cue pool;

[0052] Step S4-4-2: Calculate the proxy feature prompt loss L according to the following formula: f (D t ,S):

[0053]

[0054] Where, L f (D t ,S) represents the proxy feature hint loss, M represents the linear transformation matrix, f t represents the feature extractor of the current model, f t-1 represents the feature extractor of the model saved in the previous incremental stage, D t represents the tth task, S represents the rehearsal buffer, x, y represent the input sample of the current task and its corresponding label, x′, y′ represent the sample from the rehearsal buffer and its corresponding label, p is the input perception cue, and B represents the proxy feature cue.

[0055] Furthermore, the total network loss L(D t ,S) is calculated as follows:

[0056] L(D t ,S)=L ce(Dt)+α Lrehearsal-ce (S)+β Lrehearsal-logit (S)+γL f (D t ,S)

[0057] In the formula, L(D t , S) represents the total loss of the model, α, β, γ are hyperparameters used to weigh the contribution of each loss to the total loss, D t represents the tth task, and S represents the rehearsal buffer.

[0058] The continuous learning model device based on playback under dual-granularity prompts described in the present invention includes a processor and a computer program stored in a memory and executable on the processor, and the above method is implemented when the processor executes the program.

[0059] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0060] (1) The input perception prompt module proposed in the present invention can be used as an additional supplement to the network input, helping to generate a more reasonable and diverse input distribution and improving the generalization ability of the network;

[0061] (2) The proxy feature hint module proposed in this paper can be used as a consistency regularizer to avoid feature conflicts caused by differences between network features. It bridges the knowledge gap between the teacher and student models to maintain consistency during feature transfer and better balance the stability and plasticity of the continuous learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flow chart of a method for establishing a continuous learning model based on playback under dual-granularity prompts provided by the present invention;

[0063] Figure 2 It is a structural schematic diagram of the playback-based continuous learning model under dual-granularity prompts of the present invention. DETAILED DESCRIPTION

[0064] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] This embodiment specifically provides a method for establishing a continuous learning model based on playback under dual-granularity prompts, such as Figure 1 and Figure 2 As shown, the specific steps include:

[0067] Step S1: dividing a data set containing a number of images to be classified into a training data set and a test data set, and determining a learning order for images of different categories in the data set;

[0068] This embodiment selects CIFAR10, CIFAR100 and TinyImageNet datasets for continuous learning tasks. Among them, the pictures in the dataset are divided into a series of tasks according to their categories, and the number of categories contained in each task is equal. Task one is trained first, and the remaining tasks are trained in sequence. In this embodiment, the pictures in the dataset are divided into 10 tasks according to their categories.

[0069] Step S2: preprocessing the images in the training data set;

[0070] The preprocessing method is as follows:

[0071] (1) Randomly crop the image to a preset size, which in this embodiment is 32*32;

[0072] (2) randomly flipping the image horizontally with a preset probability; in this embodiment, the preset probability is 0.5;

[0073] (3) Convert the image into tensor format;

[0074] (4) The image is normalized with a preset mean value and standard deviation. In this embodiment, the preset mean value is (0.4914, 0.4822, 0.4465), and the preset standard deviation is (0.247, 0.2435, 0.2615).

[0075] Step S3: constructing a playback-based continuous learning model under dual-granularity prompts, including a feature extractor, a classifier, an input perception prompt module, and an agent feature prompt module;

[0076] The feature extractor is used to extract features from the input samples, using a ResNet-18 network;

[0077] The classifier is used to classify the features extracted by the feature extractor, and is specifically composed of a fully connected layer;

[0078] The input perception prompt module is used to supplement the network input, help generate more reasonable and diverse input distribution, and improve the generalization ability of the network, including:

[0079] The attention module focuses on key objects or regions in the image, learning and paying attention to objects or features relevant to the task;

[0080] A fully connected layer is used to perform a nonlinear transformation on the input attention vector to generate a weight vector equal to the number of prompts in the prompt pool;

[0081] The input-aware prompt pool contains a series of randomly initialized learnable prompts, which are weighted and summed with the weight vector to generate input-aware prompts for additional supplementation of network input.

[0082] Among them, the proxy feature prompt module bridges the knowledge gap between the teacher and student models to maintain consistency during feature transfer and enhance the plasticity and stability of features, including:

[0083] Convolutional layer, used to capture information between different channels and reduce the channel dimension to the same number of categories currently known to the model;

[0084] A fully connected layer is used to perform a nonlinear transformation on the learnable prompt vector to generate a weight vector equal to the number of categories currently known to the model;

[0085] The projection layer, which is implemented by the fully connected layer, is used to transform the new class features into a learnable transformation of the old class features to ensure the linear separability of the new class and the old class.

[0086] Step S4: Obtain the preprocessed training data in sequence and train the playback-based continuous learning model under dual-granularity prompts. The training method specifically includes:

[0087] Step S4-1: Setting training parameters, where the training parameters include necessary parameters such as the number of iterations, learning rate, batch size, rehearsal buffer size, etc. If the current stage is the base training stage, the continuous learning model described in S3 is randomly initialized. If it is the incremental training stage, the network parameters obtained by the previous stage training are directly used for initialization;

[0088] Step S4-2: For the data in the base training phase or the data in the incremental training phase belonging to the new class, these data are sent to the current network, and the new class sample cross entropy loss L is executed. ce (D t ), enhance the ability of continuous learning models to distinguish new types of images;

[0089] Among them, the calculation method of the cross entropy loss of new class samples specifically includes:

[0090] Step S4-2-1: Calculate the weight vector according to the following formula:

[0091] α=softmax(FC(Atn(x)))

[0092] In the formula, Atn represents the attention module, FC refers to the fully connected layer, softmax represents the SoftMax activation function, and α represents the final set of weight vectors corresponding to the input x;

[0093] Step S4-2-2: Calculate the input perception prompt according to the following formula:

[0094] p=∑α m P m

[0095] In the formula, p represents the final input perception prompt, α m is the mth weight vector in the weight vector set, P m is the mth cue in the cue pool;

[0096] Step S4-2-3: Calculate the cross entropy loss L of the new class sample according to the following formula ce (D t ):

[0097]

[0098] Where D t represents the tth task, x, y represent the input sample and its corresponding label, cross-entropy represents the cross entropy loss, h θ represents the current model parameterized by θ.

[0099] Step S4-3: If it is not the base training stage, select old class samples with the same amount of data as the current new class from the rehearsal buffer, send them to the current network, and perform the cross entropy loss L of the old class samples respectively. rehearsal-ce (S) and distillation loss L rehearsal-logit (S);

[0100] Among them, the cross entropy loss and distillation loss of the above old class samples are calculated as follows:

[0101] Step S4-3-1: Calculate the cross entropy loss L of the old class samples according to the following formula: rehearsal-ce (S):

[0102]

[0103] Where S represents the rehearsal buffer, cross-entropy represents the cross entropy loss, x′, y′ represent the samples from the rehearsal buffer and their corresponding labels, respectively, and h θ represents the current model parameterized by θ;

[0104] Step S4-3-2: Calculate the distillation loss L according to the following formula: rehearsal-logit (S):

[0105]

[0106] Where h′ represents the model saved in the previous incremental stage.

[0107] Step S4-4: If it is not the base training stage, combine the new class samples and the old class samples, and send them to the network trained in the current stage and the frozen network in the previous stage respectively, and perform the proxy feature prompt loss L on the features of the network trained in the current stage and the network in the previous stage f (D t ,S);

[0108] Among them, the calculation method of the proxy feature prompt loss specifically includes:

[0109] Step S4-4-1: Calculate the proxy feature hint according to the following formula:

[0110] B=∑FC′(β n )Conv(β n )

[0111] In the formula, B represents the proxy feature prompt, FC′ represents a series of fully connected layers, Conv represents the convolution operation, and β n represents the nth cue in the agent feature cue pool;

[0112] Step S4-4-2: Calculate the proxy feature prompt loss L according to the following formula: f (D t ,S):

[0113]

[0114] Where, L f (D t ,S) represents the proxy feature hint loss, M represents the linear transformation matrix, f t represents the feature extractor of the current model, f t-1 represents the feature extractor of the model saved in the previous incremental stage, D t represents the tth task, S represents the rehearsal buffer, x, y represent the input sample of the current task and its corresponding label, x′, y′ represent the sample from the rehearsal buffer and its corresponding label, p is the input perception cue, and B represents the proxy feature cue.

[0115] Step S4-5: If it is the base training stage, use the loss calculated in step S4-2 to optimize the network parameters. If it is not the base training stage, combine the losses calculated in steps S4-2 to S4-4 to calculate the total network loss L(D t ,S), optimize the network parameters;

[0116] Among them, the total network loss L(Dt ,S) is calculated as follows:

[0117] L(D t ,S)=L ce( Dt)+α Lrehearsal-ce (S)+β Lrehearsal-logit (S)+γL f (D t ,S)

[0118] In the formula, L(D t , S) represents the total loss of the model, α, β, γ are hyperparameters used to weigh the contribution of each loss to the total loss, D t represents the tth task, and S represents the rehearsal buffer.

[0119] Step S4-6: Use the class-balanced reservoir sampling algorithm to select a certain number of samples from the samples of the new class currently being trained and update the rehearsal buffer.

[0120] Among them, the arrangement buffer is updated, and the specific steps are: traverse the current training samples one by one, and when the rehearsal buffer is not full, put the training samples into the rehearsal buffer in turn; when the rehearsal buffer is full, generate a random number for each training sample in turn, and the generation range of the random number is the total number of samples that participate in the selection and construction of the rehearsal buffer up to the current sample. When the random number size exceeds the range of the rehearsal buffer size, the sample is not selected and put into the data buffer; when the random number size is within the range of the rehearsal buffer size, count the number of samples of all old categories in the rehearsal buffer, find the old category with the largest number of samples in the rehearsal buffer and build its sample index set, randomly select an index from the index set, remove the corresponding old sample from the rehearsal buffer, and add the current training sample to the position corresponding to the index.

[0121] Step S5: Use deep copy to save a copy that is independent of the current model as the old network for the next incremental stage;

[0122] The method of using deep copy to save a copy independent of the current model is as follows:

[0123] Another network model is created, and the data of the current model is recursively copied to the newly created network model, so there is no data sharing between the newly created network model and the current model, which serves as the old network in the next incremental stage.

[0124] Step S6: Use the preprocessed test data set to test the performance of the model and obtain the average classification accuracy of the model for all currently known category images;

[0125] Among them, the image preprocessing method of the test data set is as follows:

[0126] Step S6-1: convert the image into a tensor format;

[0127] Step S6-2: Normalize the image with a preset mean value and standard deviation. In this embodiment, the preset mean value is (0.4914, 0.4822, 0.4465), and the preset standard deviation is (0.247, 0.2435, 0.2615).

[0128] Step S7: Return to step S4 and execute until the training of all classes is completed, thereby obtaining a replay-based continuous learning model under dual-granularity prompting that has completed training.

[0129] This embodiment also provides a playback-based continuous learning model device under dual-granularity prompts, including a processor and a computer program stored in a memory and executable on the processor, and the processor implements the above method when executing the program.

[0130] The quantitative results of this embodiment are compared with other methods on the continuous learning task datasets CIFAR10, CIFAR100 and TinyImageNet, and the results are shown in Table 1. This embodiment selects FAA (final average accuracy) as an indicator for evaluating the classification performance of the network, where a higher FAA indicates a more accurate classification accuracy.

[0131] Table 1 Quantitative comparison results of this embodiment and other methods on CIFAR10, CIFAR100 and TinyImageNet

[0132]

[0133] The present invention discloses a method and device for establishing a continuous learning model based on playback under dual-granularity prompts. The present invention first pre-processes images in a training data set; then constructs a continuous learning model based on playback under dual-granularity prompts, including a feature extractor, a classifier, an input perception prompt module and an agent feature prompt module; the feature extractor is used to extract features from input samples; the classifier is used to classify the features output by the feature extractor; the input perception prompt module is used for additional supplementation of network input, helping to generate more reasonable and diversified input distributions and improving the generalization ability of the network; the agent feature prompt module is used to bridge the knowledge gap between the teacher and student models to maintain consistency during the feature transfer process, thereby enhancing the plasticity and stability of the features. The constructed continuous learning model based on playback under dual-granularity prompts is trained using the pre-processed training data; the performance of the model is tested using the pre-processed test data set to obtain the average classification accuracy of the model for all currently known category images, and continue to train the next category; until the training of all categories is completed, a trained continuous learning model based on playback under dual-granularity prompts is obtained. The present invention can efficiently complete continuous learning tasks, and the image classification accuracy is better than that of existing continuous learning methods.

Claims

1. A method for establishing a continuous learning model based on playback under dual-granularity prompts, characterized in that: The method includes: Step S1: dividing a data set containing a number of images to be classified into a training data set and a test data set, and determining a learning order for images of different categories in the data set; Step S2: preprocessing the images in the training data set; Step S3: construct a playback-based continuous learning model under dual-granularity prompts, including a feature extractor, a classifier, an input perception prompt module and an agent feature prompt module; the feature extractor is used to extract features from input samples, and is specifically formed by stacking several convolutional layers; the classifier is used to classify the features extracted by the feature extractor, and is specifically composed of a fully connected layer; the input perception prompt module is used for additional supplementation of network input; the agent feature prompt module is used to bridge the knowledge gap between the teacher and student models; Step S4: obtaining the preprocessed training data in sequence, and training the playback-based continuous learning model under dual-granularity prompts; Step S5: Use deep copy to save a copy that is independent of the current model as the old network for the next incremental stage; Step S6: Use the preprocessed test data set to test the performance of the model and obtain the average classification accuracy of the model for all currently known category images; Step S7: Return to step S4 and execute until the training of all classes is completed, and a continuous learning model based on playback under dual-granularity prompting is obtained; The input perception prompt module in step S3 specifically includes: The attention module focuses on key objects or regions in the image, learning and paying attention to objects or features relevant to the task; A fully connected layer is used to perform a nonlinear transformation on the input attention vector to generate a weight vector equal to the number of prompts in the prompt pool; The input-aware prompt pool contains a series of randomly initialized learnable prompts, which are weighted and summed with the weight vector to generate input-aware prompts for additional network input. The agent feature prompt module in step S3 specifically includes: The proxy feature prompt pool contains a set of randomly initialized learnable prompts, which are used to subsequently generate proxy feature prompts to bridge the knowledge gap between the teacher and student models; Convolutional layer, used to capture information between different channels and reduce the channel dimension to the same number of categories currently known to the model; A fully connected layer is used to perform a nonlinear transformation on the learnable prompt vector to generate a weight vector equal to the number of categories currently known to the model; Projection layer, used to transform new class features into learnable transformations of old class features.

2. The method for establishing a continuous learning model based on playback under dual-granularity prompts according to claim 1, characterized in that: In step S1, the learning order of samples of different categories in the data set is determined as follows: The images in the dataset are divided into a series of tasks according to their categories. Each task contains an equal number of categories. Task one is trained first, and the remaining tasks are trained in sequence.

3. The method for establishing a continuous learning model based on playback under dual-granularity prompts according to claim 1, characterized in that: The step S4 specifically includes: Step S4-1: Setting training parameters. If the current stage is the base training stage, the continuous learning model described in S3 is randomly initialized. If it is the incremental training stage, the network parameters obtained by the previous stage training are directly used for initialization. Step S4-2: For the data in the base training phase or the data belonging to the new class in the incremental training phase, these data are sent to the current network and the cross entropy loss L of the new class samples is performed. ce (D t ); Step S4-3: If it is not the base training stage, select old class samples with the same amount of data as the current new class from the rehearsal buffer, send them to the current network, and perform the cross entropy loss L of the old class samples respectively. rehearsal-ce (S) and distillation loss L rehearsal-logit (S); Step S4-4: If it is not the base training stage, combine the new class samples and the old class samples, and send them to the network trained in the current stage and the frozen network in the previous stage respectively, and perform the proxy feature prompt loss L on the features of the network trained in the current stage and the network in the previous stage f (D t ,S); Step S4-5: If it is the base training stage, use the loss calculated in step S4-2 to optimize the network parameters. If it is not the base training stage, combine the losses calculated in steps S4-2 to S4-4 to calculate the total network loss L(D t ,S), optimize the network parameters; Step S4-6: Select a certain number of samples from the samples of the new class currently being trained using a class-balanced reservoir sampling algorithm to update the rehearsal buffer; The method for calculating the proxy feature prompt loss in step S4-4 specifically includes: Step S4-4-1: Calculate the proxy feature hint according to the following formula: B=∑FC′(β n )Conv(b n ) In the formula, B represents the proxy feature prompt, FC′ represents a series of fully connected layers, Conv represents the convolution operation, and β n represents the nth cue in the agent feature cue pool; Step S4-4-2: Calculate the proxy feature prompt loss L according to the following formula: f (D t ,S): Where, L f (D t ,S) represents the proxy feature hint loss, M represents the linear transformation matrix, f t represents the feature extractor of the current model, f t-1 represents the feature extractor of the model saved in the previous incremental stage, D t represents the tth task, S represents the rehearsal buffer, x, y represent the input sample of the current task and its corresponding label, x′, y′ represent the sample from the rehearsal buffer and its corresponding label, p is the input perception cue, and B represents the proxy feature cue.

4. The method for establishing a continuous learning model based on playback under dual-granularity prompts according to claim 3 is characterized in that: The method for calculating the cross entropy loss of the new class samples in step S4-2 specifically includes: Step S4-2-1: Calculate the weight vector according to the following formula: α=softmax(FC(Atn(x))) In the formula, Atn represents the attention module, FC refers to the fully connected layer, softmax represents the SoftMax activation function, and α represents the final set of weight vectors corresponding to the input x; Step S4-2-2: Calculate the input perception prompt according to the following formula: p=∑α m P m Where p represents the input perception prompt, α m is the mth weight vector in the weight vector set, P m is the mth cue in the cue pool; Step S4-2-3: Calculate the cross entropy loss L of the new class sample according to the following formula ce (D t ): Where D t represents the tth task, x, y represent the input sample and its corresponding label, cross-entropy represents the cross entropy loss, h θ represents the current model parameterized by θ.

5. The method for establishing a continuous learning model based on playback under dual-granularity prompts according to claim 3 is characterized in that: In step S4-3, the cross entropy loss L of the old class samples rehearsal-ce (S) and distillation loss L rehearsal-logit The calculation method of (S) specifically includes: Step S4-3-1: Calculate the cross entropy loss L of the old class samples according to the following formula: rehearsal-ce (S): Where S represents the rehearsal buffer, cross-entropy represents the cross entropy loss, x′, y′ represent the samples from the rehearsal buffer and their corresponding labels, respectively, and h θ represents the current model parameterized by θ; Step S4-3-2: Calculate the distillation loss L according to the following formula: rehearsal-logit (S): Where h′ represents the model saved in the previous incremental stage.

6. The method for establishing a continuous learning model based on playback under dual-granularity prompts according to claim 3 is characterized in that: The total network loss L(D t ,S) is calculated as follows: L(D t ,S)=L ce (D t )+αL rehearsal-ce (S)+βL rehearsal-logit (S)+γL f (D t ,S) In the formula, L(D t , S) represents the total loss of the model, α, β, γ are hyperparameters used to weigh the contribution of each loss to the total loss, D t represents the tth task, and S represents the rehearsal buffer.

7. A playback-based continuous learning model device under dual-granularity prompts, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images