An image classification method, device, medium and product based on incremental learning

Abstract

The application discloses an image classification method and device based on incremental learning, a medium and a product, and relates to the technical field of machine learning. The method comprises the following steps: inputting image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data, and determining a general prompt vector corresponding to each image data through a pre-trained general prompt generator; generating a first input vector based on the specific prompt vector and the general prompt vector, and generating a second input vector based on the specific prompt vector; optimizing a target image classification model by using the first input vector and the second input vector, and classifying images to be classified in an image classification task by using the optimized target image classification model. Through adaptive and general prompt-based double prompt adaptive incremental learning, the model can be prevented from suffering from catastrophic forgetting during incremental learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and particularly relates to an image classification method, device, medium and product based on incremental learning. BACKGROUND

[0002] Incremental learning is a learning method in the field of machine learning. Compared with traditional batch learning, incremental learning hopes that the model can gradually learn from new data, rather than starting from scratch. This means that incremental learning requires the model to retain previous knowledge and only update the part related to new data, which can save computing resources and time. Generally speaking, for an incremental learning model, since incremental learning is a continuous learning process, the model needs to continuously learn from new data samples, rather than just learning in the initial stage. At the same time, the model needs to retain the learned knowledge and only update the newly added or changed knowledge. An effective strategy is needed to handle the integration of new and old knowledge. On this basis, the model needs to have strong adaptability to adapt to data changes in real time and maintain high accuracy.

[0003] In incremental learning, based on the characteristics of incremental learning, it is easy to cause the problems of data distribution change and concept shift. Generally, incremental learning may change over time in practical applications, which is a common technical difficulty of incremental learning and a major problem that incremental learning hopes to solve when it is proposed. Currently, the incremental learning model is expected to adapt to such distribution changes, and the model can adapt to such changes when it encounters new data different from the training data distribution before, so as to maintain the performance of the model. Concept shift is similar to data distribution change, which generally refers to the change of the target concept in the data over time, and the current incremental learning team change is expected to detect and adapt, so as to maintain its performance. Due to the occurrence of data distribution change and concept shift, the problem of catastrophic forgetting often occurs. In many ways, the problem of catastrophic forgetting is a core problem in incremental learning. Generally, the problem of catastrophic forgetting refers to the problem that the neural network often has large-scale forgetting of previously learned knowledge when learning new tasks or new data, which is obviously not very common in the learning process of human beings. Generally, in incremental learning, the problem of catastrophic forgetting is caused by the joint optimization of the parameters of the neural network on all task data. When the model encounters a new task, the model will adjust to adapt to the new data. However, in the adjustment process, the data of the old task cannot participate in the parameter update process together, so the adjustment process of the new task often accompanies the destruction of the performance of the old task, that is, the knowledge of the old task is forgotten in the process of the model processing and learning the new task, resulting in a decline in the performance of the old task.

[0004] Currently, in solving the problem of catastrophic forgetting in incremental learning, one method is replay-based incremental learning, which can store the samples of old classes and mix them with new class samples for training. However, due to the need to store old class data, the memory cost problem limits the further development of the technology, and the storage and replay of data may involve sensitive information, which may cause privacy and security problems. Another method is parameter regularization-based incremental learning, which can prevent the model from overfitting when training new classes through regularization terms, but it is easy to overprotect old knowledge, which limits the generalization ability and flexibility of the model. Another method is knowledge distillation-based incremental learning, which can "distill" the knowledge of the old model into the new model to avoid catastrophic forgetting. However, due to its conservatism, the model tends to retain the old class model, and it is not convenient for long-term learning.

[0005] Therefore, how to more effectively solve the catastrophic forgetting of the model in incremental learning is a problem that needs to be solved by those skilled in the art. SUMMARY

[0006] The embodiment of the present application aims to provide an image classification method, device, medium and product based on incremental learning, which can solve the problem of catastrophic forgetting of the model in incremental learning through double prompt adaptive class incremental learning based on adaptive prompt and general prompt.

[0007] To solve the above technical problems, in a first aspect, the present application discloses an image classification method based on incremental learning, comprising:

[0008] Inputting the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data, and determining a general prompt vector corresponding to each image data through a pre-trained general prompt generator;

[0009] Generating a first input vector based on the specific prompt vector and the general prompt vector, and generating a second input vector based on the specific prompt vector;

[0010] Optimizing a target image classification model using the first input vector and the second input vector, and classifying the images to be classified in the image classification task using the optimized target image classification model.

[0011] In some embodiments, the inputting the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data comprises:

[0012] Inputting the image data corresponding to each incremental learning task into the preset adaptive specific prompt generator, and transposing the image data using a first input transposer of the preset adaptive specific prompt generator, and performing horizontal normalization on the transposed image data;

[0013] Inputting the horizontally normalized image data into a multi-layer perception layer of the preset adaptive specific prompt generator, extracting a feature representation of the image data through the multi-layer perception layer, and generating the specific prompt vector corresponding to the image data based on the feature representation.

[0014] In some embodiments, after the multi-layer perception layer extracts the feature representation of the image data, the method further comprises:

[0015] Judging the current training number of the multi-layer perception layer, and freezing the multi-layer perception layer after the current training number of the multi-layer perception layer reaches a preset training number.

[0016] In some embodiments, the generating the specific prompt vector corresponding to the image data based on the feature representation comprises:

[0017] inputting the feature representation corresponding to the image data into a linear conversion layer of the preset adaptive specific prompt generator, and adjusting the feature representation based on a preset scaling factor and a preset offset factor by using the linear conversion layer;

[0018] generating a plurality of adaptive prompts based on the adjusted feature representation, and transposing the adaptive prompts by using a second input transposer of the preset adaptive specific prompt generator to generate the specific prompt vector corresponding to each image data.

[0019] In some embodiments, before the transposing the adaptive prompts by using the second input transposer of the preset adaptive specific prompt generator, the method further comprises:

[0020] determining the adaptive prompt corresponding to each image data from the plurality of adaptive prompts based on a preset parameter of learning dynamic prompt, so as to transpose the adaptive prompt by using the second input transposer of the preset adaptive specific prompt generator.

[0021] In some embodiments, the determining the adaptive prompt corresponding to each image data from the plurality of adaptive prompts based on the preset parameter of learning dynamic prompt comprises:

[0022] optimizing the preset parameter according to a preset label of a target transformer layer in the target image classification model by using a preset matching loss function, and determining the adaptive prompt corresponding to each image data from the plurality of adaptive prompts based on the optimized preset parameter.

[0023] In some embodiments, the optimizing the target image classification model based on the first input vector and the second input vector comprises:

[0024] constructing an optimization function based on a preset loss function parameter, the preset similarity function, the first input vector, a generator parameter of the preset adaptive specific prompt generator, and a classifier parameter of the target image classification model;

[0025] optimizing the target image classification model based on the first input vector and the second input vector by using the optimization function.

[0026] In some embodiments, the optimizing the target image classification model based on the first input vector and the second input vector by using the optimization function comprises:

[0027] When the backbone network parameters of the visual transformer in the target image classification model are frozen, the generator parameters and the classifier parameters are updated to update the optimization function based on the updated generator parameters and the classifier parameters.

[0028] In some embodiments, the optimizing the target image classification model using the first input vector and the second input vector comprises:

[0029] The first input vector and the second input vector are respectively input into a first test layer and a second test layer in the target image classification model to optimize the target image classification model; the model depth where the first test layer is located is the depth where the general prompt and the specific prompt jointly act, and the model depth where the second test layer is located is the depth where the specific prompt alone acts.

[0030] In some embodiments, after the optimizing the target image classification model using the first input vector and the second input vector, the method further comprises:

[0031] The general prompt vector in the first test layer is parameter-optimized based on the image data, and a corresponding target prompt vector is generated according to the optimized general prompt vector, so as to perform model testing on the target image classification model based on the target prompt vector.

[0032] In some embodiments, before the determining the general prompt vector corresponding to each image data by the pre-trained general prompt generator, the method further comprises:

[0033] The general prompt training depth of the target image classification model is determined according to preset general prompt training parameters, and the general prompt training depth is taken as the model depth where the first test layer is located, and the corresponding general prompt generator is trained on the first test layer based on the image data of each incremental learning task.

[0034] In some embodiments, before the optimizing the target image classification model using the first input vector and the second input vector, the method further comprises:

[0035] The image data is segmented into corresponding image blocks by the target image classification model, and the image blocks are mapped to a preset embedding space, so as to optimize the target image classification model according to the image blocks by using the multi-layer perceptron in the target image classification model.

[0036] In a second aspect, the present application discloses an electronic device, comprising:

[0037] a memory for storing a computer program;

[0038] A processor is configured to execute the computer program to implement the steps of the image classification method based on incremental learning.

[0039] In a third aspect, the present application discloses a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the image classification method based on incremental learning.

[0040] In a fourth aspect, the present application discloses a computer program product, and the computer program product comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the steps of the image classification method based on incremental learning.

[0041] From the above technical solution, the image data corresponding to each incremental learning task can be input into the preset adaptive specific prompt generator to generate the specific prompt vector corresponding to each image data, and the universal prompt vector corresponding to each image data is determined through the pre-trained universal prompt generator, then the first input vector is generated based on the specific prompt vector and the universal prompt vector, and the second input vector is generated based on the specific prompt vector, so that the target image classification model is optimized by using the first input vector and the second input vector, and the image classification task is classified by using the optimized target image classification model.

[0042] The present application has the advantages that: first, the adaptive specific prompt generator is used to generate specific prompts for different inputs, and then the universal prompt generator is used to generate universal prompts for different inputs, through the double-layer prompt mechanism, the process of human learning knowledge is simulated, on the basis of using the adaptive prompter to generate specific prompts for each task, the concept of universal prompt is proposed to mine the unified logic of different knowledge, so as to further improve the anti-forgetting ability of the image classification model by combining the specific prompt and the universal prompt, solve the problem of catastrophic forgetting, and improve the practical application ability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0044] Figure 1 A flow chart of an image classification method based on incremental learning is provided for the embodiments of the present application.

[0045] Figure 2 An image classification model architecture based on incremental learning is provided for the embodiments of the present application.

[0046] Figure 3 A self-adaptive specific prompt generator structure schematic diagram is provided for the embodiment of the present application.

[0047] Figure 4 A model optimization method flowchart based on incremental learning is provided for the embodiment of the present application.

[0048] Figure 5 A specific image classification model architecture diagram based on incremental learning is provided for the embodiment of the present application.

[0049] Figure 6 An electronic device structure diagram is provided for the embodiment of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0051] The terms “include” and “have” and any variations thereof in the specification and above drawings of the present application are intended to cover the inclusions without the exclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can include steps or units not listed.

[0052] In order to enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0053] In the incremental learning, the catastrophic forgetting problem is caused by the fact that the parameters of the neural network are jointly optimized on the data of all tasks. When the model encounters a new task, the model's description will be adjusted to adapt to the new data. However, during the adjustment process, the data of the old task cannot participate in the parameter update process, so the adjustment process of the new task often causes the performance of the old task to be destroyed. In the present application, in order to solve the problem of catastrophic forgetting in the incremental learning, specific knowledge adaptive prompts and general knowledge global prompts are constructed in the prompt-based incremental learning to improve the efficiency of the prompts and solve the problem of catastrophic forgetting. In the traditional prompt-based incremental learning method using a prompt pool, the performance of the model is often affected by the size of the prompt pool, so the present application uses a non-pooling technique to generate specific prompts for each task. At the same time, it is obvious that learning special prompts for each task does not conform to the mechanism of human learning new knowledge, so on the one hand, learning can be performed through specific prompts, and on the other hand, learning can be performed through prompts shared by multiple knowledge to avoid catastrophic forgetting.

[0054] Next, the image classification method based on incremental learning provided by the embodiment of the present application will be described in detail. Figure 1 The image classification method based on incremental learning provided by the embodiment of the present application comprises:

[0055] In step S11, the image data corresponding to each incremental learning task is input into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data, and a general prompt vector corresponding to each image data is determined by a pre-trained general prompt generator.

[0056] As shown in Figure 2 Based on the deep convolutional neural network, the adaptive prompt model of the prompt learning is combined with the double-layer prompt method and the pre-trained model VIT (vision Transformer) to form a basic framework of the incremental learning, and through the framework, the image data corresponding to each incremental learning task 1, 2…n is input into the preset adaptive specific prompt generator to generate a specific prompt vector E corresponding to each image data, and a general prompt vector g corresponding to each image data is determined by a pre-trained general prompt generator, so that the VIT neural network is used as the backbone network, and when each task arrives, the specific task-specific prompt is generated by the adaptive prompter, and then the trained general task prompt is added to the VIT network for joint training, and then the classifier is used to eliminate the classification bias of the model.

[0057] Specifically, as shown in Figure 3As shown, when generating the specific prompt vector corresponding to each image data, first, the image data corresponding to each incremental learning task is input into the preset adaptive specific prompt generator, and the first input transpositioner (Transpose) of the preset adaptive specific prompt generator is used to transpose the image data , and the transposed image data is subjected to horizontal normalization (LN, layer normalize), then the horizontally normalized image data is input into the multilayer perception layer (MLP, Multilayer Perceptron) of the preset adaptive specific prompt generator, and the feature representation of the image data is extracted through the multilayer perception layer to generate the specific prompt vector corresponding to the image data based on the feature representation. It can be understood that in this embodiment, the LN layer is used instead of the traditional batch normalization (Batch Normalization), which does not need to rely on the batch size, and is beneficial to the subsequent stage. In this way, through the adaptive specific prompt generator structure in this embodiment, the prompt can be generated through the mode of independent generator, and the domain-related knowledge corresponding to the target task is encoded. Specifically, the information specific to the instance based on the transposed input can be extracted through the feedforward network and linear transformation, and the adaptive prompt is created in time, so that the specific prompt is generated in the mode of independent generator, which helps the plasticity of the model and can enhance the resistance of the model to catastrophic forgetting. In the above generation of the specific prompt vector corresponding to the image data based on the feature representation, the feature representation corresponding to the image data can be input into the linear transformation layer (Linear Transform) of the preset adaptive specific prompt generator, and the feature representation is adjusted based on the preset scaling factor and the preset offset factor through the linear transformation layer, then a plurality of adaptive prompts are generated based on the adjusted feature representation, and the adaptive prompts are transposed through the second input transpositioner of the preset adaptive specific prompt generator, to generate the specific prompt vector corresponding to each image data. Specifically, in the linear transformation layer, two parameter factors need to be maintained, gamma (scaling factor): it is multiplied by the standardized data, used to adjust the size of the data, represented by ; beta (offset factor): it is added to the standardized data, used to adjust the offset of the data, represented by , so that the input can be converted into a prompt, as shown in the following formula:

[0058] ;

[0059] wherein, is the offset factor, and after processing by the linear transformation layer, the corresponding scaling factor and the offset factor are obtained, and , wherein e indicatesFigure 2 The image data corresponding to each task shown, In the middle p, d, namely The dimension range of the corresponding task.

[0060] And it needs to be pointed out that, as Figure 3 As shown, after extracting the feature representation of the image data through the multi-layer perception layer, it is necessary to determine the current training round of the multi-layer perception layer, and freeze the multi-layer perception layer after the current training number of the multi-layer perception layer reaches the preset training number. Through the MLP layer, the deep feature representation of the input data is extracted, and through the multi-layer nonlinear transformation, the complex patterns and relationships in the data can be captured to find specific prompts, so as to better distinguish the feature characteristics of different classes between similar domains, and freezing the MLP layer after several training stages also helps to alleviate the phenomenon of catastrophic forgetting.

[0061] It also needs to be pointed out that before the adaptive prompt is transposed by the second input transposer of the preset adaptive specific prompt generator, the adaptive prompt corresponding to each image data can also be determined from a plurality of adaptive prompts based on the preset parameters of learning dynamic prompts, so as to transpose the adaptive prompt by the second input transposer of the preset adaptive specific prompt generator. And in the process of determining the adaptive prompt corresponding to each image data from a plurality of adaptive prompts, the preset matching loss function can be used to optimize the preset parameters according to the preset mark of the target transformer layer in the target image classification model, and the adaptive prompt corresponding to each image data is determined from a plurality of adaptive prompts based on the optimized preset parameters. In this embodiment, when generating specific prompts, it is mainly based on L2P (Learning To Prompt for Continual Learning) to achieve, which maintains a learnable parameter named designated task vector k t , which is used to maintain the pre-trained model to be closer to all training instances D t belonging to the target task t, where the target task refers to the corresponding target category of the input data, and the training instance refers to the training data belonging to the category. In order to optimize k t , L2P uses the similarity function as follows:

[0062] ;

[0063] Where S C represents the cosine similarity, f b (f p (·)) [CLS] is the last transformer layer of the frozen pre-trained model (i.e. Figure 2the [CLS] token (without prompt) of the rightmost Transformer layer in BERT) and fb represents a frozen pre-trained model encoder, and fp represents a function that generates a corresponding adaptive prompt representation according to the model input x, that is, the preset matching loss function. In this way, in the model of the embodiment, k t complementary instructions beneficial to prediction are embedded, and the instance-level prompt is created. t The complementary instructions beneficial to prediction are obtained according to the possible target categories obtained by k t corresponding specific prompts are obtained according to the possible target categories obtained by k

[0064] Based on the above technical solutions, as shown in Figure 3 When the data enters the adaptive specific prompt generator, an input transposition is performed on the data by the input transposer, and then the data is subjected to horizontal normalization and LN processing, and then is sent to the multi-layer perceptron layer to extract deep feature representations of the input data through the MLP layer. Subsequently, a linear conversion layer is used to further encode the input. In this way, through the adaptive specific prompt generator structure in the embodiment, the prompt is generated in the mode of an independent generator, and the domain-related knowledge corresponding to the target task is encoded. That is, by using the feedforward network and the linear conversion to extract the instance-specific information based on the transposed input, the adaptive prompt can be created in time, and the specific prompt is generated in the mode of an independent generator, which is helpful for better prediction and creation of the instance-level parameter, and can improve the plasticity of the model and enhance the resistance of the model to catastrophic forgetting. That is, the incremental learning model based on the adaptive specific prompt generator and the general prompt generator in the embodiment uses the adaptive specific prompt and the general prompt to prompt the input, realizes the functions of the instance-level specific prompt and the general prompt of the whole input, and does not need to save the past data, only needs to perform incremental training normally, and does not have the problems of data privacy and memory cost. On the other hand, the no-pooling improvement based on the prompt-based class incremental learning method makes it impossible to have a class incremental learning prompt pool, so that different inputs are prompted at the instance level to assist the model to avoid catastrophic forgetting.

[0065] In step S12, a first input vector is generated based on the specific prompt vector and the general prompt vector, and a second input vector is generated based on the specific prompt vector.

[0066] In the embodiment, as shown in Figure 2 the prompt generated after the specific prompt generator can be used as represents a specific prompt vector that only belongs to the input, and in a specific dual prompt-based class-incremental learning algorithm model, for an input that specifies the model depth using a general prompt It should be noted that it should receive two layers of processing, one layer is through the adaptive specific prompt processor to obtain the specific prompt , and one layer is through the general prompt to add the general prompt G, at this time, the first input vector can be generated based on the combination of the specific prompt vector and the general prompt vector corresponding to the input, which can be specifically represented by the following formula:

[0067] ;

[0068] Among them represents the input after two layers of prompt processing, and L represents the number of model inputs.

[0069] If the input is not an input that belongs to the specified depth level of the model, then at this time of the second input vector is as follows:

[0070] ;

[0071] At this time, the input only needs to pass through the adaptive specific prompt generator to generate a prompt, and does not need to pass through the general prompt to add the general prompt vector. In this way, by highly decoupling the adaptive specific prompt generator, the general prompt generator and the different modules of the pre-training model, further improvement is facilitated, and through class-incremental learning, the machine learning model can adapt to the continuous updating of data changes without starting from scratch to retrain, thereby realizing continuous learning and adapting to new environments, and can well solve the pain points of batch learning models to better adapt to the dynamic changes in the current real environment.

[0072] Step S13, optimizing the target image classification model by using the first input vector and the second input vector, and classifying the to-be-classified image in the image classification task by using the optimized target image classification model.

[0073] In this embodiment, the first input vector and the second input vector can be used to optimize the target image classification model, and the optimized target image classification model can be used to classify the to-be-classified image in the image classification task. For example Figure 2As shown, before optimizing the target image classification model, the image data can also be preprocessed, and the preprocessing process can include: using the target image classification model to segment the image data into corresponding image blocks, and mapping the image blocks to a preset embedding space, so as to optimize the target image classification model according to the image blocks using the multi-layer perceptron in the target image classification model. The specific image preprocessing process can segment and flatten the image into blocks by Tokenizer, and map these blocks to the embedding space by Projection, so that the Transformer model can process and learn the complex relationships in the image, so that VIT can use the Transformer architecture to process image data. Through the above technical solution, using VIT neural network as the backbone network, when each task arrives, first generate a specific task specified prompt through the adaptive prompter, and then add the trained general task prompt to enter the VIT network for joint training, and then eliminate the classification bias of the model through the classifier. The incremental learning model architecture based on the double prompts of the adaptive specific prompt generator and the general prompt generator can better decouple different inputs, so that different inputs get instance-level prompts, and at the same time, it can better capture the general prompt knowledge between different inputs across the neighborhood, so that the model can better understand the feature relationship between different inputs to help the model better remember different knowledge, thereby greatly enhancing the ability of the model to resist catastrophic forgetting.

[0074] Specifically, when optimizing the target image classification model, an optimization function can be constructed based on the preset loss function parameter, the preset similarity function, the first input vector, the generator parameter of the preset adaptive specific prompt generator, and the classifier parameter of the target image classification model, and the target image classification model is optimized based on the first input vector and the second input vector using the optimization function. That is, based on the input containing the prompt vector obtained in the previous step After that, for a single-layer loss function, it can be expressed as follows:

[0075] ;

[0076] In the above formula, (x, y) ∈ Dt, represents the cross-entropy loss, represents the adaptive specific prompt generator parameter, and θ represents the classifier parameter, represents the frozen pre-trained model encoder, represents the classifier, when the parameters of the VIT backbone network are frozen, only the parameters of the adaptive prompt generator are updated and the parameters of the classifier θ, where λ is a balance between the two losses, which is generally fixed at 0.1. That is, in the process of optimizing the target image classification model, the generator and the classifier are unfrozen parts that automatically update their own parameters during the training stage to achieve better ability to identify the input category. When the backbone network parameters of the visual transformer in the target image classification model are frozen, the generator parameters and the classifier parameters can be updated to update the optimization function based on the updated generator parameters and the classifier parameters. In this way, through the above optimization function, an adaptive specific prompt generator can be trained for each layer, and a general prompt generator is trained for all inputs at a specified depth.

[0077] Through the above technical solutions, the graph classification model in this embodiment has better effects and better performance in the number of prompts, the quality of prompts, and the action mechanism of prompts. Specifically, through the model constructed based on the adaptive prompt generator, the VIT frozen layer, the general prompt generator, and the classifier, the specified depth action result is verified by the experimental data of the performance of the dual prompt effect in the test set. The generation of the general prompt is only performed at the depth of the first three layers of the model. For an encoder at a specified depth, it includes an adaptive specific prompt generator, a frozen pre-trained model layer, and a general prompt generator. When an input arrives, it will be processed by the adaptive specific prompt generator and the general prompt generator. The adaptive specific prompt generator will generate corresponding specific prompts according to different inputs. The general prompt generator will generate a general prompt for each input. Then the two prompts and the corresponding input will be spliced, and then sent to the frozen pre-trained model including the attention layer and the MLP layer for corresponding optimization training. When the training level reaches the specified general prompt depth layer, the input will not need to attach the general prompt vector, but only needs to be sent to the pre-trained model after being spliced with the corresponding specific prompt generated by the adaptive specific prompt generator for optimization. After completing the multi-layer training and optimization, the data will be sent to the classifier layer. Through such a model design, the mechanisms of the general prompt and the adaptive prompt generator can be combined well. At the same time, by freezing the pre-trained model VIT, the resource and training difficulty requirements can be greatly reduced. Compared with other prompt-based incremental learning mechanisms, the adaptive prompt generator generates independent prompts for each input, and the general prompt generator maintains the same knowledge between different inputs. Through such an idea, the adaptive incremental learning mechanism based on dual prompts is realized, thereby further improving the anti-forgetting ability of the image classification model and solving the problem of catastrophic forgetting.

[0078] Based on the previous embodiment, it can be seen that the present invention can optimize the image classification model by combining specific prompts with general prompts to further improve the anti-forgetting ability of the image classification model. Next, the optimization process of the above model will be explained in detail in this embodiment. Figure 4 An embodiment of the present invention provides a model optimization method based on incremental learning, the method comprising:

[0079] Step S21: input the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data, and determine the general prompt vector corresponding to each image data through a pre-trained general prompt generator.

[0080] Step S22: Generate a first input vector based on the specific hint vector and the general hint vector, and generate a second input vector based on the specific hint vector.

[0081] Step S23: input the first input vector and the second input vector into the first test layer and the second test layer of the target image classification model respectively to optimize the target image classification model; the model depth where the first test layer is located is the depth where the general prompt and the specific prompt work together, and the model depth where the second test layer is located is the depth where the specific prompt works alone.

[0082] It can be understood that in this embodiment, the use of the prompt pool in the prompt pool-based learning can force the input to be associated and explored in the form of a group. From a certain perspective, it is an application of the common logic of different knowledge. In this embodiment, when the adaptive specific generator is used to provide a specific prompt for a group of inputs, it only contains the knowledge system of specific knowledge. Therefore, Figure 5 As shown, a general prompt G-Prompt is added in this embodiment, denoted by G. The general prompt is used to find common general knowledge between different inputs. Specifically, the prompt block (PROMPT-BLOCK) and the Transformer encoder layer can be combined with the general prompt to obtain a more appropriate specific prompt corresponding to the input image, so as to output the final result through the classifier. In this way, the knowledge system of the model can be better optimized by optimizing and using G. It should be pointed out that, whether in the testing phase or in the training phase, the model depth used by the general prompt G needs to be explored. Experiments have shown that using G at different levels will have different effects. This is more obvious in both the testing phase and the training phase. Experiments have shown that generally speaking, the model performance is best when the model depth is about three to four layers. Therefore, the model depth determined based on the use of general prompts can be determined as follows: Figure 2The general prompt specific prompt dual role depth, and it can be understood that the above model depth can be adjusted according to the actual model training, and after the completion of the current stage of incremental learning, the model can also be adjusted according to the test situation of the obtained image classification model, so as to adjust the model depth of the joint action of the general prompt and the specific prompt in the next model training, so as to further improve the catastrophic forgetting problem of the model.

[0083] Therefore, in a specific embodiment, the general prompt training depth of the target image classification model can be determined according to the preset general prompt training parameters before determining the general prompt vector corresponding to each image data, and the general prompt training depth is used as the model depth of the first test layer, and the general prompt generator is trained based on the image data of each incremental learning task on the first test layer. In this way, the first input vector and the second input vector can be input into the first test layer and the second test layer in the target image classification model respectively to optimize the target image classification model. The model depth of the first test layer is the depth of the joint action of the general prompt and the specific prompt, and the model depth of the second test layer is the depth of the single action of the specific prompt.

[0084] And after optimizing the target image classification model in this embodiment, the general prompt vector in the first test layer can also be parameter optimized based on the image data, and the corresponding target prompt vector can be generated according to the optimized general prompt vector, so as to test the target image classification model based on the target prompt vector. Specifically as Figure 2 As shown, since G can participate in all inputs and extract the general feature optimization logic of the input, g is used to participate in the training of each group during the training of each layer, wherein G and g are different expressions of the general prompt, and after each group training is completed, G is updated using the following formula:

[0085] ;

[0086] That is, the use logic of the general prompt G in the embodiment is divided into two parts of a training stage and a test stage. In the process of the test stage, G participates in the training process of the model in the form of g in each layer of the Block. For each input of the same layer, the initial g is the same. Then each input performs a parameter optimization process on the corresponding g to extract its general knowledge. After the single-layer training process is completed, g is updated to G in the form of a formula. In this way, the generated G can have the refinement of the general knowledge of all inputs. The process is repeated in each layer. In the test stage after the model training is completed, g does not participate in the test. In the test process, G participates in the test process in the form of an unchanged vector, as a global general prompt to guide the learning process of the model. In this way, on the basis of using the non-pooling technology, the prompt is generated by the adaptive prompter, thereby solving the dependence on the prompt pool and avoiding the influence of the fixed size of the prompt pool itself on the performance of the model, so as to better generate the prompt. Meanwhile, the double-layer prompt mechanism is further proposed to simulate the process of human learning knowledge. On the basis of generating the specific prompt of each task by the adaptive prompter, the concept of general prompt is proposed. The common prompt features of each task are extracted by the parameter accumulation method to mine the unified logic inside different knowledge, thereby further improving the anti-forgetting ability of the model and solving the problem of catastrophic forgetting.

[0087] More specific processing procedures about the above steps S21 and S22 can be referred to the corresponding content disclosed in the foregoing embodiments, which will not be described here in detail.

[0088] Through the above technical solution, the double-layer prompt-based incremental learning method based on the adaptive prompter is proposed in the embodiment. The problem of the limitation of the replay memory space and the data privacy problem of the traditional replay-based incremental learning method is solved. When performing the incremental learning, the memory cost limitation and the privacy security problem do not need to be disturbed, and the current social development status can be better adapted. At the same time, the existing prompt learning based on the prompt pool is optimized without pooling. The adaptive prompter is used to generate the specified prompt specific to the task to optimize the recognition ability of the corresponding new specific knowledge. Meanwhile, the specified prompt specific to the task and the general prompt between different tasks are distinguished. The general prompt mechanism between different tasks is added to find the commonality between different tasks to improve the efficiency of the prompt. And by combining the specified prompt of the specific task with the common prompt of all tasks, a more effective prompt learning strategy than the traditional prompt pool is constructed, thereby greatly improving the performance of the model and further reducing the catastrophic forgetting situation.

[0089] Further, the embodiment of the present application also discloses an electronic device,Figure 6 is shown according to an exemplary embodiment, the contents of the figure cannot be considered as any limitation on the use range of the present application. The electronic device can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to realize the related steps in the image classification method based on incremental learning disclosed in any of the preceding embodiments. In addition, the electronic device in the present embodiment can be an electronic computer.

[0090] In the present embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol it follows can be any communication protocol applicable to the technical solution of the present application, which is not limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and the specific interface type can be selected according to the specific application needs, which is not limited here.

[0091] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0092] Among them, the operating system 221 is used to manage and control each hardware device on the electronic device and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the image classification method based on incremental learning executed by the electronic device disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0093] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by the processor to realize the image classification method based on incremental learning disclosed in the preceding embodiment. The specific steps of the method can refer to the corresponding contents disclosed in the preceding embodiments, which will not be repeated here.

[0094] Further, the present application also discloses a computer program product, including computer programs / instructions, wherein the computer programs / instructions are executed by the processor to realize the image classification method based on phonemes disclosed in the preceding embodiment. The specific steps of the method can refer to the corresponding contents disclosed in the preceding embodiments, which will not be repeated here.

[0095] The various embodiments described in this specification are presented for the purpose of illustration and description. Each of the embodiments highlights different aspects of the various embodiments, and the various embodiments are not mutually exclusive, and can be combined.

[0096] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or both. The disclosure is not limited to any particular hardware or software implementation unless specifically recited in the specification. The purpose behind this disclosure is to enable the flexible implementation of the described functionality, and the disclosure should not be construed as requiring a particular implementation in hardware or software, unless specifically recited in the specification.

[0097] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The disclosure is not limited to any particular functional implementation of the steps of the methods or algorithms described herein.

[0098] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and are not necessarily intended to denote the order, quantity, combination, or importance of such elements. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0099] The above detailed description of the technical solutions provided by the present application has been described in detail, and the principles and implementation modes of the present application have been described in the text. The above description of the embodiments is only to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description of the specification should not be understood as limiting the present application.

Claims

1. An image classification method based on incremental learning, characterized in that: include: Inputting the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data, and determining a universal prompt vector corresponding to each image data through a pre-trained universal prompt generator; generating a first input vector based on the specific hint vector and the general hint vector, and generating a second input vector based on the specific hint vector; Optimizing a target image classification model using the first input vector and the second input vector, and classifying an image to be classified in an image classification task using the optimized target image classification model; The step of inputting the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data includes: Inputting the image data corresponding to each incremental learning task into the preset adaptive specific prompt generator, transposing the image data using a first input transposer of the preset adaptive specific prompt generator, and performing horizontal normalization on the transposed image data; Inputting the horizontally normalized image data into a multi-layer perception layer of the preset adaptive specific cue generator, extracting feature representations of the image data through the multi-layer perception layer, and generating the specific cue vector corresponding to the image data based on the feature representations; Furthermore, generating the specific hint vector corresponding to the image data based on the feature representation includes: Inputting the feature representation corresponding to the image data into a linear transformation layer of the preset adaptive specific cue generator, and using the linear transformation layer to adjust the feature representation based on a preset scaling factor and a preset offset factor; generating a plurality of adaptive hints based on the adjusted feature representation, and transposing the adaptive hints through a second input transposer of the preset adaptive specific hint generator to generate the specific hint vector corresponding to each of the image data; The optimizing the target image classification model by using the first input vector and the second input vector includes: The first input vector and the second input vector are respectively input into the first test layer and the second test layer in the target image classification model to optimize the target image classification model; the model depth where the first test layer is located is the depth where general prompts and specific prompts act together, and the model depth where the second test layer is located is the depth where specific prompts act alone.

2. The image classification method based on incremental learning according to claim 1, characterized in that After extracting the feature representation of the image data through the multi-layer perception layer, the method further includes: Determine a current training round of the multi-layer perception layer, and freeze the multi-layer perception layer after the current training number of the multi-layer perception layer reaches a preset training number.

3. The image classification method based on incremental learning according to claim 1, characterized in that Before transposing the adaptive hint by the second input transposer of the preset adaptive specific hint generator, the method further includes: An adaptive prompt corresponding to each image data is determined from a plurality of adaptive prompts based on preset parameters of the learned dynamic prompt, so as to transpose the adaptive prompt through the second input transposer of the preset adaptive specific prompt generator.

4. The image classification method based on incremental learning according to claim 3, characterized in that: The step of determining the adaptive prompt corresponding to each image data from the plurality of adaptive prompts based on the preset parameters of the learning dynamic prompt includes: The preset parameters are optimized according to the preset labels of the target transformer layer in the target image classification model using a preset matching loss function, and the adaptive prompt corresponding to each image data is determined from the plurality of adaptive prompts based on the optimized preset parameters.

5. The image classification method based on incremental learning according to claim 4, characterized in that: The optimizing the target image classification model by using the first input vector and the second input vector includes: constructing an optimization function based on preset loss function parameters, a preset similarity function, the first input vector, generator parameters of the preset adaptive specific prompt generator, and classifier parameters of the target image classification model; The target image classification model is optimized using the optimization function based on the first input vector and the second input vector.

6. The image classification method based on incremental learning according to claim 5, characterized in that: The process of optimizing the target image classification model based on the first input vector and the second input vector by using the optimization function further includes: When the backbone network parameters of the visual transformer in the target image classification model are frozen, the generator parameters and the classifier parameters are updated to update the optimization function based on the updated generator parameters and the classifier parameters.

7. The image classification method based on incremental learning according to claim 1, characterized in that: After optimizing the target image classification model using the first input vector and the second input vector, the method further includes: Parameters of the general prompt vector in the first test layer are optimized based on the image data, and a corresponding target prompt vector is generated according to the optimized general prompt vector, so as to perform model testing on the target image classification model based on the target prompt vector.

8. The image classification method based on incremental learning according to claim 1, characterized in that: Before determining the universal prompt vector corresponding to each of the image data by the pre-trained universal prompt generator, the method further includes: The universal prompt training depth of the target image classification model is determined according to the preset universal prompt training parameters, and the universal prompt training depth is used as the model depth of the first test layer, and the corresponding universal prompt generator is trained on the first test layer based on the image data of each incremental learning task.

9. The image classification method based on incremental learning according to any one of claims 1 to 8, characterized in that: Before optimizing the target image classification model using the first input vector and the second input vector, the method further includes: The target image classification model is used to segment the image data into corresponding image blocks, and the image blocks are mapped to a preset embedding space, so that the target image classification model is optimized according to the image blocks using a multilayer perceptron in the target image classification model.

10. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the image classification method based on incremental learning as claimed in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the image classification method based on incremental learning according to any one of claims 1 to 9.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the image classification method based on incremental learning according to any one of claims 1 to 9 are implemented.

Images