Image classification method and device based on incremental learning, medium and product

Abstract

The invention discloses an image classification method and device based on incremental learning, a medium and a product, and relates to the technical field of machine learning, and the method comprises the steps: inputting image data corresponding to each incremental learning task into a preset adaptive specific prompt generator, so as to generate a specific prompt vector corresponding to each image data, determining a universal prompt vector corresponding to each piece of image data through a pre-trained universal prompt generator; generating a first input vector based on the specific prompt vector and the universal prompt vector, and generating a second input vector based on the specific prompt vector; and optimizing a target image classification model by using the first input vector and the second input vector, and classifying to-be-classified images in an image classification task by using the optimized target image classification model. Through dual-prompt adaptive incremental learning based on adaptive prompt and general prompt, disastrous forgetting of the model during incremental learning can be avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to an image classification method, device, medium and product based on incremental learning. Background Art

[0002] Incremental learning is a learning method in the field of machine learning. Compared with traditional batch learning, incremental learning hopes that the model can gradually learn from new data instead of learning from scratch. This means that incremental learning requires the model to retain previous knowledge and only update the parts related to new data, which can save computing resources and time. Generally speaking, for an incremental learning model, since incremental learning is a continuous learning process, the model needs to continuously learn from new data samples instead of just learning in the initial stage. At the same time, the model needs to retain the knowledge that has been learned and only update the new knowledge or changed knowledge. It needs to have an effective strategy to handle the integration of new and old knowledge. On this basis, the model needs to have strong adaptability, be able to adapt to data changes in real time, and maintain a high accuracy rate.

[0003] In incremental learning, based on the characteristics of incremental learning, it is easy to cause problems such as changes in data distribution and concept shift. Generally speaking, incremental learning may often change over time in actual applications. This is a very common technical difficulty in incremental learning and a major problem that incremental learning hopes to solve when it is proposed. At present, it is hoped that the incremental learning model can adapt to this distribution change, and when the model encounters new data with a different distribution from the previous training data, it can adapt to this change, so as to maintain the performance of the model. Concept shift is similar to the change in data distribution. Generally, it means that the target concept in the data may change over time. At present, it is hoped that the incremental learning team can detect and adapt to the change, so as to maintain its performance. It is precisely because of the occurrence of phenomena such as data distribution changes and concept shift that catastrophic forgetting problems are usually caused. In many ways, catastrophic forgetting is a core problem in incremental learning. Usually, catastrophic forgetting refers to the problem that when neural networks learn new tasks or new data, they often forget the previously learned knowledge on a large scale, which is obviously not very common in the human learning process. Usually in incremental learning, the catastrophic forgetting problem occurs because the parameters of the neural network are jointly optimized on the data of all tasks. When the model encounters a new task, the model's explanation will be adjusted to adapt to the new data. However, in the process of adjustment, the data of the old task cannot be involved in the parameter update process. Therefore, the adjustment process of the new task is often accompanied by the destruction of the performance of the old task. In other words, in the process of the model processing and learning new tasks, the knowledge of the old task is forgotten, resulting in a decline in the performance of the old task.

[0004] Currently, when solving the problem of catastrophic forgetting in incremental learning, one method is incremental learning based on replay, which can store samples of old categories and mix them with samples of new categories for training. However, the need to store old category data has resulted in memory cost issues that limit the further development of the technology, and the storage and replay of data may involve sensitive information, which may cause privacy and security issues. Another method is incremental learning based on parameter regularization, which can prevent the model from overfitting when training new categories through regularization terms, but it is easy to over-protect old knowledge, resulting in limited generalization ability and insufficient flexibility of the model. Another method is incremental learning based on knowledge distillation, which can "distill" the knowledge of the old model into the new model to avoid catastrophic forgetting. However, due to being too conservative, the model tends to retain the old category model, and this makes long-term learning inconvenient.

[0005] It can be seen that how to more effectively solve the catastrophic forgetting caused by the model during incremental learning is a problem that technical personnel in this field need to solve. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide an image classification method, device, medium and product based on incremental learning, which can help solve the catastrophic forgetting problem caused by the model during incremental learning through dual prompt adaptive incremental learning based on adaptive prompts and general prompts.

[0007] In order to solve the above technical problems, in a first aspect, the present invention discloses an image classification method based on incremental learning, comprising:

[0008] Inputting the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each of the image data, and determining a general prompt vector corresponding to each of the image data through a pre-trained general prompt generator;

[0009] generating a first input vector based on the specific hint vector and the general hint vector, and generating a second input vector based on the specific hint vector;

[0010] The target image classification model is optimized using the first input vector and the second input vector, and the optimized target image classification model is used to classify the image to be classified in the image classification task.

[0011] In some embodiments, the step of inputting the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data includes:

[0012] Inputting the image data corresponding to each of the incremental learning tasks into the preset adaptive specific prompt generator, transposing the image data using a first input transposer of the preset adaptive specific prompt generator, and horizontally normalizing the transposed image data;

[0013] The horizontally normalized image data is input into the multi-layer perception layer of the preset adaptive specific prompt generator, the feature representation of the image data is extracted through the multi-layer perception layer, and the specific prompt vector corresponding to the image data is generated based on the feature representation.

[0014] In some embodiments, after extracting the feature representation of the image data through the multi-layer perception layer, the method further includes:

[0015] Determine a current training round of the multi-layer perception layer, and freeze the multi-layer perception layer after the current training number of the multi-layer perception layer reaches a preset training number of times.

[0016] In some embodiments, generating the specific prompt vector corresponding to the image data based on the feature representation includes:

[0017] Inputting the feature representation corresponding to the image data into a linear transformation layer of the preset adaptive specific prompt generator, and using the linear transformation layer to adjust the feature representation based on a preset scaling factor and a preset offset factor;

[0018] A plurality of adaptive hints are generated based on the adjusted feature representation, and the adaptive hints are transposed through a second input transposer of the preset adaptive specific hint generator to generate the specific hint vector corresponding to each image data.

[0019] In some embodiments, before the second input transposer of the preset adaptive specific prompt generator transposes the adaptive prompt, the method further includes:

[0020] An adaptive prompt corresponding to each of the image data is determined from a plurality of the adaptive prompts based on preset parameters of the learning dynamic prompt, so as to transpose the adaptive prompt through the second input transposer of the preset adaptive specific prompt generator.

[0021] In some embodiments, the determining of the adaptive prompt corresponding to each of the image data from a plurality of the adaptive prompts based on the preset parameters of the learning dynamic prompt includes:

[0022] The preset parameters are optimized according to the preset labels of the target transformer layer in the target image classification model using a preset matching loss function, and the adaptive prompt corresponding to each image data is determined from a plurality of adaptive prompts based on the optimized preset parameters.

[0023] In some embodiments, optimizing the target image classification model using the first input vector and the second input vector includes:

[0024] constructing an optimization function based on preset loss function parameters, the preset similarity function, the first input vector, generator parameters of the preset adaptive specific prompt generator, and classifier parameters of the target image classification model;

[0025] The target image classification model is optimized using the optimization function based on the first input vector and the second input vector.

[0026] In some embodiments, the process of optimizing the target image classification model based on the first input vector and the second input vector by using the optimization function further includes:

[0027] When the backbone network parameters of the visual transformer in the target image classification model are frozen, the generator parameters and the classifier parameters are updated to update the optimization function based on the updated generator parameters and the classifier parameters.

[0028] In some embodiments, optimizing the target image classification model using the first input vector and the second input vector includes:

[0029] The first input vector and the second input vector are respectively input into the first test layer and the second test layer in the target image classification model to optimize the target image classification model; the model depth where the first test layer is located is the depth where general prompts and specific prompts work together, and the model depth where the second test layer is located is the depth where specific prompts work alone.

[0030] In some embodiments, after optimizing the target image classification model using the first input vector and the second input vector, the method further includes:

[0031] Parameters of the general prompt vector in the first test layer are optimized based on the image data, and a corresponding target prompt vector is generated according to the optimized general prompt vector, so as to perform model testing on the target image classification model based on the target prompt vector.

[0032] In some embodiments, before determining the universal prompt vector corresponding to each of the image data by a pre-trained universal prompt generator, the method further includes:

[0033] The general prompt training depth of the target image classification model is determined according to preset general prompt training parameters, and the general prompt training depth is used as the model depth of the first test layer, and the corresponding general prompt generator is trained on the first test layer based on the image data of each incremental learning task.

[0034] In some embodiments, before optimizing the target image classification model using the first input vector and the second input vector, the method further includes:

[0035] The target image classification model is used to segment the image data into corresponding image blocks, and the image blocks are mapped to a preset embedding space, so that the target image classification model is optimized according to the image blocks using a multi-layer perceptron in the target image classification model.

[0036] In a second aspect, the present invention discloses an electronic device, comprising:

[0037] Memory for storing computer programs;

[0038] A processor is used to execute the computer program to implement the steps of the aforementioned image classification method based on incremental learning.

[0039] In a third aspect, the present invention discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the aforementioned image classification method based on incremental learning are implemented.

[0040] In a fourth aspect, the present invention discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned image classification method based on incremental learning.

[0041] It can be seen from the above technical scheme that the present invention can input the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data, and determine the general prompt vector corresponding to each image data through a pre-trained general prompt generator, and then generate a first input vector based on the specific prompt vector and the general prompt vector, and generate a second input vector based on the specific prompt vector, so as to optimize the target image classification model using the first input vector and the second input vector, and classify the images to be classified in the image classification task using the optimized target image classification model.

[0042] The beneficial effects of the present invention are as follows: first, an adaptive specific prompt generator is used to provide specific prompts for different inputs, and then a universal prompt generator is used to provide universal prompts for different inputs. Through a double-layer prompt mechanism, the process of human learning knowledge is simulated. On the basis of using an adaptive prompter to generate unique prompts for each task, the concept of universal prompts is proposed to explore the unified logic within different knowledge, thereby combining specific prompts with universal prompts to further improve the anti-forgetting ability of the image classification model, solve the problem of catastrophic forgetting, and improve the practical application ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flowchart of an image classification method based on incremental learning provided by an embodiment of the present invention;

[0045] Figure 2 An image classification model architecture diagram based on incremental learning provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the structure of an adaptive specific prompt generator provided by an embodiment of the present invention;

[0047] Figure 4 A flow chart of a model optimization method based on incremental learning provided by an embodiment of the present invention;

[0048] Figure 5 A specific incremental learning-based image classification model architecture diagram provided in an embodiment of the present invention;

[0049] Figure 6 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0052] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0053] In incremental learning, the catastrophic forgetting problem is that the parameters of the neural network are jointly optimized on the data of all tasks. When the model encounters a new task, in order to adapt to the new data, the model's description will be adjusted. However, in the process of adjustment, since the data of the old task cannot participate in the parameter update process together, the adjustment process of the new task is often accompanied by the destruction of the performance of the old task. In the present invention, in order to solve the problem of catastrophic forgetting in class incremental learning, in the prompt-based incremental learning, specific knowledge adaptive prompts and general knowledge global prompts are constructed to improve the efficiency of prompts and solve the problem of catastrophic forgetting. In the traditional prompt-based class incremental learning method using a prompt pool, since the size of the prompt pool itself is fixed, the performance of the model is often affected by the size of the prompt pool. Therefore, the present invention uses a non-pooling technology to generate unique prompts for each task. At the same time, considering that it is obviously not in line with the mechanism of human learning new knowledge to perform separate special prompt learning for each task, on the one hand, learning can be carried out through specific prompts, and on the other hand, learning can be carried out through prompts shared by multiple knowledge to avoid catastrophic forgetting.

[0054] Next, an image classification method based on incremental learning provided by an embodiment of the present invention is introduced in detail. Figure 1 An image classification method based on incremental learning is provided in an embodiment of the present invention, and the method includes:

[0055] Step S11, input the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data, and determine the general prompt vector corresponding to each image data through a pre-trained general prompt generator.

[0056] In this embodiment, Figure 2 As shown, based on a deep convolutional neural network, an adaptive prompt model of prompt learning is used, combined with a two-layer prompt method and a pre-trained model VIT (vision Transformer), to form a basic framework for quasi-incremental learning. Through this framework, the image data corresponding to each incremental learning task 1, 2...n is first input into a preset adaptive specific prompt generator to generate a specific prompt vector E corresponding to each image data, and a pre-trained general prompt generator is used to determine the general prompt vector g corresponding to each image data. Thus, by using the VIT neural network as the backbone network, each time a task arrives, a specific task-specified prompt is first generated by the adaptive prompter, and then the trained general task prompt is added, and then they are entered into the VIT network for joint training, and then the classification bias of the model is eliminated through the classifier.

[0057] Specifically, Figure 3As shown, when generating the specific prompt vector corresponding to each image data, the image data corresponding to each incremental learning task is first input into the preset adaptive specific prompt generator, and the image data is transposed ( ), and the transposed image data is horizontally normalized (LN, layer normalize), and then the horizontally normalized image data is input into the multi-layer perception layer (MLP, Multilayer Perceptron) of the preset adaptive specific prompt generator, and the feature representation of the image data is extracted through the multi-layer perception layer to generate a specific prompt vector corresponding to the image data based on the feature representation. It can be understood that the use of the LN layer in this embodiment instead of the traditional batch normalization (Batch Normalization) does not rely on the batch size, which is beneficial to the subsequent stages. In this way, through the adaptive specific prompt generator structure in this embodiment, prompts can be generated through the mode of an independent generator, and the domain-related knowledge corresponding to the target task can be encoded. Specifically, the feedforward network and linear transformation can be used to extract instance-specific information based on the transposed input, and adaptive prompts can be created in time. In this way, by generating specific prompts in the form of an independent generator, this helps the plasticity of the model and can enhance the model's resistance to catastrophic forgetting. When generating the specific prompt vector corresponding to the image data based on the feature representation, the feature representation corresponding to the image data can be input into the linear transformation layer (Linear Transform) of the preset adaptive specific prompt generator, and the linear transformation layer is used to adjust the feature representation based on the preset scaling factor and the preset offset factor, and then a number of adaptive prompts are generated based on the adjusted feature representation, and the adaptive prompts are transposed through the second input transposer of the preset adaptive specific prompt generator to generate the specific prompt vector corresponding to each image data. Specifically, in the linear transformation layer, two parameter factors need to be maintained, gamma (scaling factor): it is multiplied by the standardized data to adjust the size of the data, and To represent; beta (offset factor): It is added to the standardized data to adjust the offset of the data, To represent, the input can be converted into a prompt, as shown in the following formula:

[0058] ;

[0059] in, Refers to the offset factor, which is processed by the linear transformation layer to obtain the corresponding scaling factor and offset factor ,and Middle e finger Figure 2 The image data corresponding to each task shown, p, d The dimension range to which it belongs.

[0060] And it should be pointed out that if Figure 3 As shown, after extracting the feature representation of the image data through the multi-layer perception layer, it is necessary to determine the current training round of the multi-layer perception layer, and freeze the multi-layer perception layer after the current training number of the multi-layer perception layer reaches the preset training number. Through the MLP layer, the deep feature representation of the input data is extracted, and the complex patterns and relationships in the data can be captured through multi-layer nonlinear transformations to find specific hints, so as to better distinguish the characteristics of different classes between similar domains, and freezing the MLP layer after several training stages also helps to alleviate the phenomenon of catastrophic forgetting.

[0061] It should also be pointed out that before the adaptive prompt is transposed by the second input transposer of the preset adaptive specific prompt generator, the adaptive prompt corresponding to each image data can also be determined from a number of adaptive prompts based on the preset parameters of the learning dynamic prompt, so as to transpose the adaptive prompt by the second input transposer of the preset adaptive specific prompt generator. And in the process of determining the adaptive prompt corresponding to each image data from a number of adaptive prompts, the preset parameters can be optimized according to the preset labels of the target transformer layer in the target image classification model using a preset matching loss function, and the adaptive prompt corresponding to each image data can be determined from a number of adaptive prompts based on the optimized preset parameters. In this embodiment, when generating specific prompts, it is mainly based on L2P (Learning To Prompt for Continual Learning). A learnable parameter is maintained in L2P, named as the specified task vector k t , is used to maintain the pre-trained model to make it closer to all training instances D belonging to the target task t t , where the target task refers to the target category of the input data, and the training instance refers to the training data belonging to this category. t , L2P uses a similarity function as shown below:

[0062] ;

[0063] Among them, S C represents the cosine similarity, f b (f p (·)) [CLS] is the last transformer layer of the frozen pre-trained model (i.e. Figure 2The [CLS] tag (without prompt) of the rightmost Transformer layer in , and fb represents the frozen pre-trained model encoder, and fp represents the function that generates the corresponding adaptive prompt representation according to the model input x. That is, the preset matching loss function. In this way, in the model of this embodiment, k t To embed supplementary instructions that are beneficial to prediction, as well as instance-level hints created. The above supplementary instructions refer to the t The possible target categories are obtained to obtain the corresponding specific prompts, so that after the prompt generation is completed, the instance-level prompts that are beneficial to the prediction are supplemented by embedding according to the similarity matching, and then output through the transposed input layer.

[0064] Based on the above technical solution, Figure 3 As shown, in this embodiment, when data enters the adaptive specific prompt generator, it will be subjected to an input transposition through the input transposer, and then it will be horizontally normalized. After being processed by LN, it will be sent to the multi-layer perceptron layer, and the deep feature representation of the input data will be extracted through the MLP layer. Then, the linear transformation layer is used to further encode the input. In this way, through the adaptive specific prompt generator structure in this embodiment, prompts are generated through an independent generator mode, and domain-related knowledge corresponding to the target task is encoded. That is to say, by utilizing a feedforward network and linear transformation to extract instance-specific information based on the transposed input, adaptive prompts can be created in a timely manner, and by generating specific prompts in the form of an independent generator, it is helpful to better predict and create instance-level parameters, which can improve the plasticity of the model and enhance the model's resistance to catastrophic forgetting. That is to say, in this embodiment, the incremental learning model based on the adaptive specific prompt generator and the general prompt generator uses adaptive specific prompts and general prompts to prompt the input, realizing the functions of instance-level specific prompts and general prompts for all inputs. In addition, since there is no need to save past data, only normal incremental training is required, and there are no problems with data privacy and memory costs. On the other hand, since the prompt-based class incremental learning method is improved without pooling, there is no class incremental learning prompt pool, so that different inputs receive instance-level prompts to assist the model in avoiding catastrophic forgetting.

[0065] Step S12: Generate a first input vector based on the specific prompt vector and the general prompt vector, and generate a second input vector based on the specific prompt vector.

[0066] In this embodiment, Figure 2 As shown, after a prompt is generated by a specific prompt generator, it can be used To represent a specific prompt vector that belongs to the input alone, in the specific dual-prompt-based incremental learning algorithm model, for an input that specifies the model depth using a general prompt For example, it should receive two layers of processing, one of which is to get specific prompts through the adaptive specific prompt processor , the first layer is to add the general prompt G through the general prompt. At this time, the first input vector can be generated based on the specific prompt vector and the general prompt vector combined with the corresponding input, which can be specifically expressed by the following formula:

[0067] ;

[0068] in represents the input after two layers of prompt processing, and L represents the number of model inputs.

[0069] If you enter is not an input to the layer at the specified depth of the model, then the second input vector Then the following formula is used:

[0070] ;

[0071] This means that the input at this time only needs to be generated by the adaptive specific prompt generator, and does not need to be attached to the general prompt vector through the general prompt. In this way, by highly decoupling the adaptive specific prompt generator, the general prompt generator and the different modules of the pre-trained model, further improvements are facilitated, and the machine learning model can adapt to the constantly updated data changes through quasi-incremental learning, without the need to retrain from scratch, thereby achieving continuous learning and adapting to new environments. This can well solve the pain points of batch learning models and better adapt to the dynamic changes in the current real environment.

[0072] Step S13: Optimize the target image classification model using the first input vector and the second input vector, and classify the image to be classified in the image classification task using the optimized target image classification model.

[0073] In this embodiment, the first input vector and the second input vector can be used to optimize the target image classification model, and the optimized target image classification model can be used to classify the image to be classified in the image classification task. Figure 2As shown, before optimizing the target image classification model, the image data may be preprocessed. The preprocessing process may include: using the target image classification model to segment the image data into corresponding image blocks, and mapping the image blocks to a preset embedding space, so as to optimize the target image classification model according to the image blocks using the multi-layer perceptron in the target image classification model. The specific image preprocessing process may be to segment the image into blocks and flatten them using Tokenizer, and map these blocks to the embedding space using Projection, so that the Transformer model can process and learn the complex relationships in the image, thereby enabling VIT to process image data using the Transformer architecture. Through the above technical solution, the VIT neural network is used as the backbone network. When each task arrives, a specific task-specified prompt is first generated by the adaptive prompter, and then the trained general task prompt is added, and then enters the VIT network for joint training. After that, the classification bias of the model is eliminated through the classifier. The incremental learning model architecture based on the dual prompts of the adaptive specific prompt generator and the general prompt generator can better decouple different inputs, so that different inputs can obtain instance-level prompts. At the same time, the general prompt knowledge between different inputs can be better captured across neighborhoods, so that the model can better understand the feature connections between different inputs to help the model better remember different knowledge, thereby greatly enhancing the model's ability to combat catastrophic forgetting.

[0074] Specifically, when optimizing the target image classification model, an optimization function can be constructed based on the preset loss function parameters, the preset similarity function, the first input vector, the generator parameters of the preset adaptive specific prompt generator, and the classifier parameters of the target image classification model, and the optimization function is used to optimize the target image classification model based on the first input vector and the second input vector. That is, based on the input containing the prompt vector obtained in the previous step After that, for the loss function of a single layer, it can be expressed as follows:

[0075] ;

[0076] In the above formula, (x, y)∈Dt, represents the cross entropy loss, represents the adaptive specific cue generator parameters, θ represents the classifier parameters, Represents a frozen pre-trained model encoder, Represents the classifier. When the parameters of the VIT backbone network are frozen, only the parameters of the adaptive prompt generator are updated. and the parameters θ of the classifier, where λ is a balance term between the two losses, which is generally fixed to 0.1. That is to say, in the process of optimizing the target image classification model, the generator and the classifier are unfrozen parts, and they will automatically update their parameters during the training phase to achieve better ability to identify input categories. When the backbone network parameters of the visual transformer in the target image classification model are frozen, the generator parameters and the classifier parameters can be updated to update the optimization function based on the updated generator parameters and the classifier parameters. In this way, through the above optimization function, an adaptive specific prompt generator can be trained for each layer, and a general prompt generator can be trained for all inputs at the layer of the specified depth.

[0077] Through the above technical solution, the graphic classification model in this embodiment has better effects and better performance in terms of the number of prompts, the quality of prompts and the action mechanism of prompts by using an adaptively generated prompt and a universal prompt. Specifically, through a model constructed based on an adaptive prompt generator, a VIT frozen layer, a universal prompt generator and a classifier, based on the experimental data verification of the performance of the test set based on the effect of the dual prompt, the specified depth effect result is obtained, and the generation of the universal prompt is set to work only at the depth of the first three layers of the model. For an encoder at a specified depth, it includes three parts: an adaptive specific prompt generator, a frozen pre-trained model layer and a universal prompt generator. When an input arrives, It will be processed by the adaptive specific prompt generator and the general prompt generator respectively. The adaptive specific prompt generator will generate corresponding specific prompts according to different inputs, and the general prompt generator will generate the same general prompt for each input. Then these two prompts will be spliced ​​with the corresponding input, and then sent to the frozen pre-trained model including the attention layer and the MLP layer for corresponding optimization training. When the training level reaches beyond the specified general prompt depth layer, the input will no longer need to attach the general prompt vector, but only need to pass through the corresponding specific prompt generated by the adaptive specific prompt generator as a splicing vector and then be sent to the pre-trained model for optimization. After completing the multi-layer training optimization, the data will be sent to the classifier layer. Through such a model design, the mechanisms of general prompts and adaptive prompt generators can be well combined. At the same time, by freezing the pre-trained model VIT, the resources and training difficulty requirements can be greatly reduced. Compared with other prompt-based incremental learning mechanisms, the adaptive prompt generator generates independent prompts for each input, and the general prompt generator maintains the same knowledge between different inputs. Through this idea, an adaptive incremental learning mechanism based on dual prompts is realized, thereby further improving the anti-forgetting ability of the image classification model and solving the problem of catastrophic forgetting.

[0078] Based on the previous embodiment, it can be seen that the present invention can optimize the image classification model by combining specific prompts with general prompts to further improve the anti-forgetting ability of the image classification model. Next, the optimization process of the above model will be described in detail in this embodiment. Figure 4 A model optimization method based on incremental learning provided by an embodiment of the present invention comprises:

[0079] Step S21, input the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data, and determine the general prompt vector corresponding to each image data through a pre-trained general prompt generator.

[0080] Step S22: Generate a first input vector based on the specific prompt vector and the general prompt vector, and generate a second input vector based on the specific prompt vector.

[0081] Step S23, input the first input vector and the second input vector into the first test layer and the second test layer in the target image classification model respectively to optimize the target image classification model; the model depth where the first test layer is located is the depth where the general prompts and the specific prompts work together, and the model depth where the second test layer is located is the depth where the specific prompts work alone.

[0082] It can be understood in this embodiment that the use of the prompt pool in the prompt pool-based learning can force the input to be associated and explored in the form of a group. From a certain perspective, it is an application of the common logic of different knowledge. In this embodiment, when the adaptive specific generator is used to provide a specific prompt for a group of inputs alone, it only contains the knowledge system of specific knowledge. Therefore, Figure 5 As shown, a general prompt G-Prompt is added in this embodiment, denoted by G. The general prompt is used to find common general knowledge between different inputs. Specifically, a more appropriate specific prompt corresponding to the input image can be obtained by combining the prompt block (PROMPT-BLOCK) and the Transformer encoder layer with the general prompt, so as to output the final result through the classifier. In this way, the knowledge system of the model can be better optimized by optimizing G and using it. It should be pointed out that both in the testing phase and in the training phase, it is necessary to explore the model depth used by the general prompt G. Experiments have shown that using G at different levels will have different effects. This is more obvious in both the testing phase and the training phase. Experiments have shown that generally speaking, the model performance is best when the model depth is about three to four layers. Therefore, the model depth determined based on the use of general prompts can be determined as follows: Figure 2The dual-action depth of general prompts and specific prompts is shown in the figure, and it can be understood that the above model depth can be adjusted according to the actual model training situation, and after the incremental learning in the current stage is completed, the model can also be adjusted according to the test situation of the obtained image classification model, so as to adjust the model depth of the general prompts and specific prompts together when the model is trained next time, so as to further improve the catastrophic forgetting problem of the model.

[0083] Therefore, in a specific embodiment, before determining the universal prompt vector corresponding to each image data, the universal prompt training depth of the target image classification model can be determined according to the preset universal prompt training parameters, and the universal prompt training depth is used as the model depth of the first test layer, and the corresponding universal prompt generator is trained on the first test layer based on the image data of each incremental learning task. In this way, the first input vector and the second input vector can be input into the first test layer and the second test layer of the target image classification model respectively to optimize the target image classification model. The model depth of the first test layer is the depth of the joint action of the universal prompt and the specific prompt, and the model depth of the second test layer is the depth of the action of the specific prompt alone.

[0084] In addition, in this embodiment, after optimizing the target image classification model, the parameters of the general prompt vector in the first test layer can be optimized based on the image data, and the corresponding target prompt vector can be generated according to the optimized general prompt vector, so as to perform model testing on the target image classification model based on the target prompt vector. Figure 2 As shown in the figure, since G can participate in all inputs and extract the common feature optimization logic of the input, in the training of each layer, g is used to participate in the training of each group, where G and g are different expressions of the common prompt, and after each group of training is completed, the following formula is used to update G:

[0085] ;

[0086] That is to say, the usage logic of the general prompt G in this embodiment is divided into two parts: the training phase and the testing phase. During the testing phase, in each layer of the Block, G will participate in the training process of the model in the form of g. For each input of the same layer, the initial g is the same. After that, each input will perform a parameter optimization process on the corresponding g to extract its general knowledge. After the training process of a single layer is completed, g will be updated to G in the form of a formula. In this way, the generated G can have the general knowledge of all inputs. This process will be repeated in each subsequent layer. In the testing phase after the model training is completed, g will not participate in the test. During the testing process, G will participate in the testing process in the form of an invariant vector, serving as a global general prompt to guide the learning process of the model. In this way, on the basis of utilizing the poolless technology, prompts are generated through an adaptive prompter, thus solving the dependence on the prompt pool and avoiding the impact of the fixed size of the prompt pool itself on the model performance, so as to better generate prompts. At the same time, a double-layer prompt mechanism is further proposed to simulate the process of human learning knowledge. On the basis of using an adaptive prompter to generate unique prompts for each task, the concept of universal prompts is proposed, and the common prompt features of each task are extracted using the parameter accumulation method to explore the unified logic within different knowledge, thereby further improving the model's anti-forgetting ability and solving the problem of catastrophic forgetting.

[0087] For more specific processing procedures of the above steps S21 and S22, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be described in detail here.

[0088] Through the above technical scheme, a dual-layer prompt-like incremental learning method based on an adaptive prompter is proposed in this embodiment, which solves the limitation problem of the traditional incremental learning method based on playback relying on the playback memory space and the data privacy problem. When performing incremental learning, it does not need to be troubled by the memory cost limitation and privacy security problems, and can better adapt to the current situation of social development. At the same time, by optimizing the existing prompt learning based on the prompt pool without pooling, using an adaptive prompter to generate task-specific designated prompts to optimize the recognition ability of the corresponding new unique knowledge, and distinguishing between the task-specific designated prompts and the general prompts between different tasks, adding a general prompt mechanism between different tasks to find the commonality of different tasks, so as to improve the efficiency of prompts, and by combining the designated prompts of specific tasks with the common prompts of all tasks, a prompt strategy that is more effective than the prompt learning of the traditional prompt pool is constructed, thereby greatly improving the performance of the model, and thus reducing the situation of catastrophic forgetting to a greater extent.

[0089] Furthermore, an embodiment of the present invention also discloses an electronic device, Figure 6 It is a structural diagram of an electronic device according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of the present invention. The electronic device may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the image classification method based on incremental learning disclosed in any of the aforementioned embodiments. In addition, the electronic device in this embodiment may specifically be an electronic computer.

[0090] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device; the communication interface 24 can create a data transmission channel between the electronic device and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0091] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0092] The operating system 221 is used to manage and control various hardware devices on the electronic device and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the image classification method based on incremental learning performed by the electronic device disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0093] Furthermore, the present invention also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed image classification method based on incremental learning is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0094] Furthermore, the present invention also discloses a computer program product, including a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the aforementioned disclosed method for generating a face image based on phonemes. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.

[0095] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0096] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0097] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0098] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0099] The technical solution provided by the present invention is introduced in detail above. Specific examples are used in this article to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation mode and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An image classification method based on incremental learning, characterized in that: include: Inputting the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each of the image data, and determining a general prompt vector corresponding to each of the image data through a pre-trained general prompt generator; generating a first input vector based on the specific hint vector and the general hint vector, and generating a second input vector based on the specific hint vector; The target image classification model is optimized using the first input vector and the second input vector, and the optimized target image classification model is used to classify the image to be classified in the image classification task.

2. The image classification method based on incremental learning according to claim 1, characterized in that: The step of inputting the image data corresponding to each incremental learning task into a preset adaptive specific prompt generator to generate a specific prompt vector corresponding to each image data includes: Inputting the image data corresponding to each of the incremental learning tasks into the preset adaptive specific prompt generator, transposing the image data using a first input transposer of the preset adaptive specific prompt generator, and horizontally normalizing the transposed image data; The horizontally normalized image data is input into the multi-layer perception layer of the preset adaptive specific prompt generator, the feature representation of the image data is extracted through the multi-layer perception layer, and the specific prompt vector corresponding to the image data is generated based on the feature representation.

3. The image classification method based on incremental learning according to claim 2, characterized in that: After extracting the feature representation of the image data through the multi-layer perception layer, the method further includes: Determine a current training round of the multi-layer perception layer, and freeze the multi-layer perception layer after the current training number of the multi-layer perception layer reaches a preset training number of times.

4. The image classification method based on incremental learning according to claim 2, characterized in that: The generating the specific prompt vector corresponding to the image data based on the feature representation comprises: Inputting the feature representation corresponding to the image data into a linear transformation layer of the preset adaptive specific prompt generator, and using the linear transformation layer to adjust the feature representation based on a preset scaling factor and a preset offset factor; A plurality of adaptive hints are generated based on the adjusted feature representation, and the adaptive hints are transposed through a second input transposer of the preset adaptive specific hint generator to generate the specific hint vector corresponding to each image data.

5. The image classification method based on incremental learning according to claim 4, characterized in that: Before the adaptive hint is transposed by the second input transposer of the preset adaptive specific hint generator, the method further comprises: An adaptive prompt corresponding to each of the image data is determined from a plurality of the adaptive prompts based on preset parameters of the learning dynamic prompt, so as to transpose the adaptive prompt through the second input transposer of the preset adaptive specific prompt generator.

6. The image classification method based on incremental learning according to claim 5, characterized in that: The step of determining the adaptive prompt corresponding to each of the image data from a plurality of the adaptive prompts based on the preset parameters of the learning dynamic prompts includes: The preset parameters are optimized according to the preset labels of the target transformer layer in the target image classification model using a preset matching loss function, and the adaptive prompt corresponding to each image data is determined from a plurality of adaptive prompts based on the optimized preset parameters.

7. The image classification method based on incremental learning according to claim 6, characterized in that: The optimizing the target image classification model by using the first input vector and the second input vector includes: constructing an optimization function based on preset loss function parameters, the preset similarity function, the first input vector, generator parameters of the preset adaptive specific prompt generator, and classifier parameters of the target image classification model; The target image classification model is optimized using the optimization function based on the first input vector and the second input vector.

8. The image classification method based on incremental learning according to claim 7, characterized in that: The process of optimizing the target image classification model based on the first input vector and the second input vector by using the optimization function further includes: When the backbone network parameters of the visual transformer in the target image classification model are frozen, the generator parameters and the classifier parameters are updated to update the optimization function based on the updated generator parameters and the classifier parameters.

9. The image classification method based on incremental learning according to claim 1, characterized in that: The optimizing the target image classification model by using the first input vector and the second input vector includes: The first input vector and the second input vector are respectively input into the first test layer and the second test layer in the target image classification model to optimize the target image classification model; the model depth where the first test layer is located is the depth where general prompts and specific prompts work together, and the model depth where the second test layer is located is the depth where specific prompts work alone.

10. The image classification method based on incremental learning according to claim 9, characterized in that: After optimizing the target image classification model by using the first input vector and the second input vector, the method further includes: Parameters of the general prompt vector in the first test layer are optimized based on the image data, and a corresponding target prompt vector is generated according to the optimized general prompt vector, so as to perform model testing on the target image classification model based on the target prompt vector.

11. The image classification method based on incremental learning according to claim 9, characterized in that: Before determining the universal prompt vector corresponding to each of the image data by the pre-trained universal prompt generator, the method further includes: The general prompt training depth of the target image classification model is determined according to preset general prompt training parameters, and the general prompt training depth is used as the model depth of the first test layer, and the corresponding general prompt generator is trained on the first test layer based on the image data of each incremental learning task.

12. The image classification method based on incremental learning according to any one of claims 1 to 11, characterized in that: Before optimizing the target image classification model using the first input vector and the second input vector, the method further includes: The target image classification model is used to segment the image data into corresponding image blocks, and the image blocks are mapped to a preset embedding space, so that the target image classification model is optimized according to the image blocks using a multi-layer perceptron in the target image classification model.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the image classification method based on incremental learning as claimed in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the image classification method based on incremental learning as claimed in any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the image classification method based on incremental learning described in any one of claims 1 to 12 are implemented.

Images