An image classification method, device, equipment and readable storage medium

Abstract

The specification discloses an image classification method, device and equipment and readable storage medium. In response to a current image classification task, a sample image and a label type corresponding to the current image classification task are obtained. A current prompt network is determined according to each prompt subnetwork matched with the current image classification task. A current image classification model is obtained in combination of the current prompt network and a trained historical image classification model. The current image classification model is trained according to the sample image and the label type. In response to an image classification request, a to-be-classified image is input into the trained current image classification model to obtain the type of the to-be-classified image. It can be seen that the above scheme effectively copes with the catastrophic forgetting problem, enhances the flexibility and accuracy of the model when learning a new task, and avoids the loss of previously obtained knowledge. In addition, the end-to-end training manner makes the learning process of the model more robust and efficient.

Description

TECHNICAL FIELD

[0001] The present specification relates to the technical field of computer technology, and particularly relates to an image classification method and device, equipment and a readable storage medium. BACKGROUND

[0002] With the rapid development of computer vision technology, image classification technology based on artificial intelligence is crucial for understanding complex visual scenes and implementing higher-level visual processing tasks such as object detection, image recognition, and scene analysis. In particular, image classification models based on sustainable learning are widely used in various fields such as industrial production, medical health, and daily life.

[0003] Unlike traditional image classification models that are only trained on fixed data sets and do not update thereafter, image classification models based on sustainable learning can continuously learn and adapt when new data arrives, improving the model's ability to adapt to new environments and new categories. However, during the process of sustainable learning, the image classification model may forget the knowledge of the old image classification task after learning a new image classification task.

[0004] Currently, a large amount of training data can be stored for replay to address this catastrophic forgetting problem, but long-term storage of training data not only incurs huge memory costs, but also may compromise the security of private data. SUMMARY

[0005] The present specification provides an image classification method, device, equipment and readable storage medium to partially solve the above problems existing in the prior art.

[0006] The present specification adopts the following technical solutions:

[0007] The present specification provides an image classification method, comprising:

[0008] In response to a current image classification task, obtaining sample images corresponding to the current image classification task and the labeled types of the sample images;

[0009] Obtaining each prompt subnetwork matched with the current image classification task, and determining a current prompt network according to the each prompt subnetwork;

[0010] Obtaining a current image classification model to be trained according to the current prompt network and a historical image classification model trained, wherein the historical image classification model is trained according to sample images corresponding to a historical image classification task in advance, and the historical image classification task is different from the current image classification task;

[0011] input the sample image corresponding to the current image classification task to the current image classification model to be trained, obtain a predicted type of the sample image output by the image classification model, and optimize the current prompt network with minimization of a difference between the predicted type and the labeled type as an optimization objective, and obtain a trained current image classification model according to the optimized current prompt network;

[0012] In response to an image classification request, input the obtained image to be classified to the trained current image classification model to obtain a type of the image to be classified.

[0013] The current prompt network is determined according to the prompt subnetworks, and specifically includes:

[0014] obtain prompt vectors, and determine a correspondence between the prompt vectors and the prompt subnetworks;

[0015] input the sample image corresponding to the current image classification task to the pre-trained encoder to obtain original features of the sample image;

[0016] determine weights corresponding to the prompt vectors respectively according to the original features of the sample image and the prompt vectors;

[0017] determine weights corresponding to the prompt subnetworks respectively according to the correspondence between the prompt vectors and the prompt subnetworks and the weights corresponding to the prompt vectors respectively;

[0018] determine a current prompt network according to the prompt subnetworks and the weights corresponding to the prompt subnetworks respectively.

[0019] Optionally, the weights corresponding to the prompt vectors respectively are determined according to the original features of the sample image and the prompt vectors, and specifically include:

[0020] determine target features of the sample image through a self-attention mechanism according to the original features of the sample image;

[0021] determine the weights corresponding to the prompt vectors respectively according to similarities between the target features of the sample image and the prompt vectors, wherein a similarity between the target features of the sample image and the prompt vectors is in a direct proportional relationship with the weight corresponding to the prompt vector.

[0022] Optionally, the predicted type of the sample image output by the image classification model is obtained by inputting the sample image corresponding to the current image classification task to the current image classification model to be trained, and specifically includes:

[0023] input a sample image corresponding to the current image classification task to the current image classification model to be trained, and obtain first features of the sample image through each prompt subnetwork included in the current prompt network;

[0024] weight the first features according to weights corresponding to the prompt subnetworks included in the current prompt network, and obtain second features of the sample image according to the weighted first features;

[0025] input the second features to a pre-trained encoder in the current image classification model to obtain third features of the sample image;

[0026] input the third features to a classification network in the current image classification model to obtain a predicted type of the sample image.

[0027] Optionally, the obtaining of the prompt vectors comprises:

[0028] determining a plurality of different reference image classification tasks matched with the current image classification task according to the current image classification task;

[0029] obtaining sample images corresponding to each reference image classification task;

[0030] inputting the sample images corresponding to each reference image classification task into a pre-trained encoder respectively to obtain feature vectors of the sample images corresponding to each reference image classification task as the obtained prompt vectors.

[0031] Optionally, the obtaining of the prompt subnetworks matched with the current image classification task comprises:

[0032] determining a plurality of different reference image classification tasks matched with the current image classification task according to the current image classification task;

[0033] obtaining sample images corresponding to each reference image classification task and labeled types of the sample images corresponding to each reference image classification task;

[0034] obtaining a plurality of layers of neural networks to be trained as prompt subnetworks to be trained;

[0035] for each reference image classification task, optimizing parameters of the prompt subnetworks to be trained according to the sample images corresponding to the reference image classification task and the labeled types of the sample images corresponding to the reference image classification task to obtain a prompt subnetwork corresponding to the reference image classification task;

[0036] the prompt subnetworks corresponding to each reference image classification task are obtained as the prompt subnetworks matched with the current image classification task.

[0037] Optionally, before the inputting the obtained to-be-classified image into the trained current image classification model to obtain the type of the to-be-classified image, the method further comprises:

[0038] obtaining a first test image corresponding to the current image classification task and a labeled type of the first test image, and obtaining a second test image corresponding to the historical image classification task and a labeled type of the second test image;

[0039] testing the trained current image classification model according to the first test image, the labeled type of the first test image, the second test image and the labeled type of the second test image, and determining that a test result of the trained current image classification model satisfies a preset condition.

[0040] The specification provides an image classification device, comprising:

[0041] an obtaining module configured to obtain, in response to a current image classification task, a sample image corresponding to the current image classification task and a labeled type of the sample image;

[0042] a current prompt network determining module configured to obtain each prompt subnetwork matched with the current image classification task, and determine a current prompt network according to the each prompt subnetwork;

[0043] a model constructing module configured to obtain, according to the current prompt network and a trained historical image classification model, a to-be-trained current image classification model; wherein the historical image classification model is trained in advance according to a sample image corresponding to a historical image classification task, and the historical image classification task is different from the current image classification task;

[0044] a training module configured to input the sample image corresponding to the current image classification task into the to-be-trained current image classification model to obtain a predicted type of the sample image output by the image classification model, minimize a difference between the predicted type and the labeled type as an optimization target, optimize the current prompt network, and obtain a trained current image classification model according to the optimized current prompt network;

[0045] a classification module configured to input, in response to an image classification request, a to-be-classified image obtained into the trained current image classification model to obtain a type of the to-be-classified image.

[0046] The specification provides a computer-readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned image classification method.

[0047] The specification provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the above-mentioned image classification method when executing the program.

[0048] The above-mentioned at least one technical solution adopted by the specification can achieve the following beneficial effects:

[0049] In the image classification method provided by the specification, in response to a current image classification task, a sample image and a label type corresponding to the current image classification task are obtained, a current prompt network is determined according to each prompt subnetwork matched with the current image classification task, a current image classification model is obtained in combination of the current prompt network and a trained historical image classification model, and the current image classification model is trained according to the sample image and the label type. Thus, in response to an image classification request, a to-be-classified image is input into the trained current image classification model to obtain the type of the to-be-classified image. As can be seen, the above-mentioned solution effectively deals with the catastrophic forgetting problem, enhances the flexibility and accuracy of the model when learning new tasks, and avoids the loss of previously obtained knowledge. In addition, the end-to-end training manner makes the learning process of the model more robust and efficient. BRIEF DESCRIPTION OF DRAWINGS

[0050] The drawings described herein are used to provide further understanding of the specification, constitute a part of the specification, the illustrative embodiments of the specification and the description thereof are used to explain the specification, and do not constitute an improper limitation on the specification. In the drawings

[0051] In the drawings:

[0052] Figure 1 It is a flowchart of an image classification method in the specification;

[0053] Figure 2 It is a schematic diagram of a current image classification model in the specification;

[0054] Figure 3 It is a flowchart of an image classification method in the specification;

[0055] Figure 4 It is a schematic diagram of an image classification device provided by the specification;

[0056] Figure 5 It is a schematic diagram of an electronic device corresponding to Figure 1 provided by the specification. DETAILED DESCRIPTION

[0057] For the purposes of the present description, the technical solutions and advantages thereof, the following will clearly, completely and concretely describe the technical solutions of the present description in conjunction with specific embodiments of the present description and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present description, rather than all the embodiments. Based on the embodiments in the present description, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present description.

[0058] In addition, it should be noted that all the actions of obtaining signals, information or data in the present description are performed in compliance with the corresponding data protection regulations and policies of the place, and with the authorization given by the corresponding device owner.

[0059] It should be noted that the features in the following embodiments and implementation manners can be combined with each other without conflict.

[0060] Continual Learning is a machine learning paradigm that aims to enable machine learning models to continuously learn based on new data while retaining and utilizing previously learned knowledge, avoiding the phenomenon of catastrophic forgetting, in the process of encountering new tasks. Specifically, in the field of image classification, image classification models based on machine learning algorithms often need to adapt to changing image distributions and new image categories in real-world applications. After learning new image categories and image distributions, the image classification model also needs to avoid forgetting previously learned image categories and image distributions. For example, after learning to classify flower images and grass images, the image classification model can continue to learn to classify cat images and dog images. After the image classification model is trained based on cat images and dog images, the image classification model still needs to retain the ability to distinguish flower images and grass images.

[0061] Currently, to avoid the problem of catastrophic forgetting of image classification models, methods based on rehearsal and methods based on hint-based continual learning can be used. However, the method based on rehearsal relies on long-term storage of historical task training data or generation of model samples, and performs rehearsal on the model based on the stored data and samples to deal with the problem of catastrophic forgetting. The hint-based continual learning method stores the knowledge of historical tasks in the form of hint vectors in a hint pool or library, queries the hint vectors matching the knowledge of new tasks from the hint pool or library when the model learns new tasks, and splices them into the input of the model, thereby ensuring that the model learns new knowledge without forgetting the knowledge.

[0062] However, in the above two schemes, the method based on replay relies on long-term storage of training data, and usually the training data contains a large amount of private data, which has the risk of violating data privacy. The continuous learning method based on prompt cannot be optimized in an end-to-end manner, relies on a secondary, localized optimization process to learn key prompts, and model gradients cannot be backpropagated through key / query indexing selection, that is, the convergence speed of the model to new task data is not fully considered, resulting in slow training speed of the model, which is not suitable for application to practical applications.

[0063] Therefore, the present specification provides an image classification method, constructs a current prompt network based on each prompt subnetwork, and then constructs a current image classification model based on the current prompt network, so as to use an end-to-end training method, optimize the current prompt network in the current image classification model by using a standard backpropagation algorithm, not only improve the training speed of the current image classification model, but also improve the performance of the image classification model on new tasks due to the introduction of the current prompt network which can work with the original model architecture. This not only improves the efficiency of learning, but also helps to maintain the stability of existing knowledge, thereby avoiding catastrophic forgetting. In this way, the current image classification model can accept new information while retaining memories of past tasks, achieving true continuous learning.

[0064] The technical solutions provided by the embodiments of the present specification will be described in detail below with reference to the accompanying drawings.

[0065] Figure 1 A flowchart of an image classification method provided by the present specification.

[0066] S100: In response to a current image classification task, obtaining a sample image corresponding to the current image classification task and a labeled type of the sample image.

[0067] The image classification method provided in the embodiments of the present specification can be executed by an electronic device such as a server for image classification. In addition, in the process of executing the method, the electronic device involved in the model training of the image classification model and the electronic device executing the method can be the same or different, which is not limited in the present specification.

[0068] In practical applications, the image classification model can learn the knowledge of different image classification tasks at different stages, so that the image classification model completed at different stages of learning can adapt to the actual image classification scene where the image type is increasing. Therefore, in the present specification, the image classification knowledge that the image classification model currently needs to learn is taken as the current image classification task, and the image classification knowledge learned by the image classification model at different stages before the current image classification task is taken as each historical image classification task.

[0069] In each image classification task, the categories of the sample images can be two or more, and the number of sample images, the number of categories, the size, and the like are not limited in the present specification.

[0070] Since the image classification method provided in the present specification is based on supervised learning to train the image classification model in the process of sustainable learning of the image classification model, in addition to obtaining the sample images corresponding to the current image classification task as training samples, the annotation types of the sample images corresponding to the current image classification task are also obtained as annotations of the training samples.

[0071] In addition, in an optional embodiment of the present specification, the sample images corresponding to the current image classification task can contain image objects of image types corresponding to other image classification tasks (especially historical image classification tasks) in addition to image objects of image types corresponding to the current image classification task. For example, the current image classification task is to classify whether there is a cat in the sample image, and the historical image classification task is to classify whether there is a dog in the sample image. Then, the sample images corresponding to the current image classification task can have a cat or a dog.

[0072] S102: Obtain each prompt subnetwork matching the current image classification task, and determine the current prompt network according to the prompt subnetworks.

[0073] In the present specification, the prompt subnetwork is a neural network used to assist the current image classification model to learn the knowledge corresponding to the current image classification task. The prompt subnetwork can be a parameter-learnable network composed of one or more layers of neural networks. The network structures of different prompt subnetworks can be the same or different.

[0074] Specifically, in order to avoid the problem of catastrophic forgetting, the current image classification model to be constructed needs to be able to stabilize the knowledge of historical image classification tasks that have been learned in the past and also learn the knowledge of the current image classification task. Therefore, each prompt subnetwork included in the current prompt network needs to match the current image classification task. Each prompt subnetwork that matches the current image classification task can mean that the parameters of each prompt subnetwork have been optimized based on the sample images and annotation types of each reference image classification task, wherein each reference image classification task is similar to the current image classification task.

[0075] For example, the current image classification task is to classify whether there is a white cat in the sample image. Then, the task of classifying whether there is a black cat in the sample image can be regarded as a reference image classification task similar to the current classification task, and the parameters of the untrained prompt subnetwork are optimized based on the task of classifying whether there is a black cat in the sample image, so as to obtain a prompt subnetwork matching the task of classifying whether there is a white cat in the sample image.

[0076] Of course, according to the specific application scenario, the parameters in the prompt subnetwork matched with the current image classification task obtained can also be unoptimized. Since the parameters in the prompt subnetwork are only initialization parameters, the prompt subnetwork does not contain any knowledge of the image classification task, at this time, the so-called prompt subnetwork matched with the current image classification task refers to the neural network that does not interfere with or conflict with the historical image classification task.

[0077] Further, in determining the current prompt network according to the prompt subnetworks, the current prompt network can be determined based on all the prompt subnetworks obtained, or the current prompt network can be determined based on at least one of the prompt subnetworks obtained. The way of constructing the current prompt network based on the prompt subnetworks can be to splice the prompt subnetworks in a certain order, that is, the output of the previous prompt subnetwork is the input of the next prompt subnetwork, or to parallel the prompt subnetworks, that is, each prompt subnetwork has the same input, and the output of the current prompt network is the fusion of the outputs of each prompt subnetwork.

[0078] In this way, the current prompt network constructed can not only more accurately match the learning needs of the current image classification task, but also because the prompt capacity and complexity of the current prompt network can be expanded with the increase of the number of prompt subnetworks, rather than a single prompt length, compared with the traditional single prompt method, it has higher flexibility and expansibility, and this dynamic adjustment capability enables the image classification model to more effectively cope with various learning tasks, thereby improving the overall classification performance and adaptability.

[0079] Optionally, in determining the current prompt network based on the prompt subnetworks, one or more prompt subnetworks matched with the current image classification task can be selected from the prompt subnetworks, and the current prompt network is constructed according to the selected one or more prompt subnetworks.

[0080] S104: obtaining a current image classification model to be trained according to the current prompt network and a historical image classification model trained, wherein the historical image classification model is trained in advance according to sample images corresponding to a historical image classification task, and the historical image classification task is different from the current image classification task.

[0081] To cope with the problem, in the specification, on the basis of the historical image classification model that has been trained, the current prompt network constructed in step S102 is introduced to obtain the current image classification model to be trained. Since the historical image classification model has been supervised trained based on the sample images and their labeled types of the historical image classification task before responding to the current image classification task, that is, the historical image classification model contains the neural network that has learned the knowledge of the historical image classification task, thus, on the basis of the historical image classification model, the current image classification model not only includes one or more prompt subnetworks matched with the current image classification task, but also includes the neural network that has learned the knowledge of the historical image classification task.

[0082] Therefore, in this step, according to the current prompt network and the historical image classification model, the current image classification model to be trained is actually directly embedded with the current prompt network in the specific layers of the historical image classification model, especially the first few layers or the layers related to the image classification task. Specifically, the historical image classification model at least includes a historical prompt network, an encoder and a classification network, and according to the current prompt network, the historical image classification model is updated, which is actually to splice the historical prompt network and the current prompt network, and then combine the encoder and the classification network in the original image classification model. Therefore, the current image classification model at least includes the current prompt network, the historical prompt network, the encoder and the classification network, as shown in Figure 2 The encoder is obtained by pre-training, and the training samples used in the pre-training process of the encoder can be sample images in a general image library. Moreover, whether in the training process of the current image classification model or in the training process of the historical image classification model, the parameters of the encoder will not be updated and optimized.

[0083] In order to enable the image classification model to continuously adapt to the continuously increasing and expanding image classification requirements in the actual application scenarios, in the sustainable learning process of the image classification model, generally, the image classification tasks that the image classification model continuously learns are different from each other, for example, the historical image classification task is to classify whether there is a cat in the image, and the current image classification task is to classify whether there is a dog in the image. Therefore, in the specification, generally, the historical image classification task is different from the current image classification task, and the difference can not only be the different image objects classified, such as the different image objects classified whether there is a cat in the image and whether there is a dog in the image, but also the different number of target types classified, such as the binary classification task of whether there is a cat in the image and the multi-classification task of whether there is a white cat, a black cat or an orange cat in the image.

[0084] In addition, it should be noted that the historical image classification model used to build the current image classification model can be trained based on sample images and their labeled types of one or more historical image classification tasks, that is, the historical image classification task can be one or more, which is not limited in the present specification. Moreover, if the current image classification task is the first training task performed by the image classification model, the historical image classification model has not been trained based on any historical image classification task, at this time, the historical image classification model only includes a pre-trained encoder and a classification network.

[0085] S106: input the sample image corresponding to the current image classification task into the current image classification model to be trained, obtain the predicted type of the sample image output by the image classification model, and optimize the current prompt network with the minimization of the difference between the predicted type and the labeled type as the optimization objective, and obtain the trained current image classification model according to the optimized current prompt network.

[0086] The model structure of the current image classification model actually includes a current prompt network that has not learned the knowledge of the current image classification task, and a historical prompt network and a classification network that have learned the knowledge of the historical image classification task. In order to learn the knowledge of the current image classification task without forgetting the knowledge of the historical image classification task in the current image classification model, only the parameters of the current prompt network are updated and optimized in the training process of the current image classification model, and the parameters of the historical prompt network, the pre-trained encoder and the classification network are frozen and do not participate in the update and optimization. Thus, in the trained current image classification model, the current prompt network trained based on the current image classification task masters the knowledge of the current image classification task, the historical prompt network trained based on the historical image classification task masters the knowledge of the historical image classification task, and the pre-trained encoder has learned the knowledge of general images in advance.

[0087] Thus, the trained current image classification model composed of the optimized current prompt network, the trained historical prompt network and the pre-trained encoder can learn the knowledge of the current image classification task without forgetting the knowledge of the historical image classification task, thereby avoiding the problem of catastrophic forgetting in the process of continuous learning.

[0088] In addition, optionally, when the parameters of the current prompt network are optimized based on the sample images of the current image classification task and their annotation types, the parameters of the classification network in the current image classification model can also be optimized. The classification network can be an untrained fully connected network and a softmax layer introduced when the current image model is constructed, or can be a classification network that has been optimized in the historical image classification model, and the present specification does not limit this. Therefore, S106 can be specifically: inputting the sample images corresponding to the current image classification task into the current image classification model to be trained to obtain the predicted types of the sample images output by the image classification model, taking the minimization of the difference between the predicted types and the annotation types as the optimization goal, optimizing the current prompt network and the classification network in the current image classification model, and obtaining the trained current image classification model according to the optimized current prompt network and the optimized classification network.

[0089] S108: In response to the image classification request, inputting the obtained image to be classified into the trained current image classification model to obtain the type of the image to be classified.

[0090] Since the trained current image classification model is obtained by incremental learning based on the historical image classification model that has been trained and by adding the current prompt network, and only the parameters of the current prompt network are optimized, the parameters of the original historical image classification model that has been trained are frozen and do not change. Therefore, the trained current image classification model can recognize both the image to be classified corresponding to the current image classification task and the image to be classified corresponding to the historical image classification task, that is, after learning the knowledge of the current image classification task, the current image classification model will not forget the learned knowledge of the historical image classification task, thereby avoiding the problem of catastrophic forgetting.

[0091] Thus, when classifying the input image to be classified based on the trained current image classification model, as shown by the solid line and the dashed line in the model structure of Figure 2 The image to be classified will be input into the current prompt network and the historical prompt network, so that whether the image to be classified is an image of the current image classification task or an image of the historical image classification task, the image to be classified can obtain a predicted type with an expected accuracy rate based on the current image classification model.

[0092] Therefore, in the specification, the image to be classified carried in the image classification request can be an image to be classified corresponding to the current image classification task, or an image to be classified corresponding to a historical image classification task before the current image classification task in the incremental training process of the image classification model based on lifelong learning. For example, the current image classification task is to classify leaves and flowers, and the historical image classification task is to classify cats and dogs. The image to be classified carried in the image classification request can be a leaf image or a cat image.

[0093] Of course, the image to be classified carried in the image classification request can also not match any image classification task in the incremental training process of the current image classification model based on lifelong learning. The type of the image to be classified output by the trained current image classification model has a low probability of belonging to any known image type. At this time, it can be determined that the type of the image to be classified is unknown.

[0094] In the image classification method provided in the specification, in response to the current image classification task, the sample image and the labeled type corresponding to the current image classification task are obtained, the current prompt network is determined according to the obtained each prompt subnetwork matching the current image classification task, the current image classification model is obtained by combining the current prompt network and the trained historical image classification model, and the current image classification model is trained according to the sample image and the labeled type. Thus, in response to the image classification request, the image to be classified is input into the trained current image classification model to obtain the type of the image to be classified.

[0095] It can be seen that through the above scheme, the catastrophic forgetting problem is effectively addressed, the flexibility and accuracy of the model in learning new tasks are enhanced, and the loss of previously acquired knowledge is avoided. In addition, the end-to-end training method makes the learning process of the model more robust and efficient.

[0096] In one or more embodiments of the specification, Figure 1 The step S102 shown can be implemented according to the following scheme, as shown in Figure 3

[0097] S200: Obtain each prompt vector and determine the correspondence between each prompt vector and each prompt subnetwork.

[0098] Specifically, the prompt vector is used to query the prompt subnetwork matching the current image classification task. Therefore, there needs to be a correspondence between each prompt subnetwork and each prompt vector. By determining the prompt vector matching the current image classification task, the prompt subnetwork matching the current image classification task can be screened out based on the correspondence between each prompt vector and each prompt subnetwork.

[0099] ​Since each prompt vector needs to be compared with a sample image of the current image classification task to determine which prompt vector is related to the current image classification task, or further determine the matching degree of each prompt vector to the current image classification task, each prompt vector can also be a feature vector of a sample image.

[0100] Based on this, in the present specification, each prompt vector can be a feature vector of a sample image corresponding to each image classification task in the incremental training process of the image classification model in the lifelong learning, and of course, can also be a feature vector of a sample image corresponding to other reference image classification tasks different from each image classification task in the incremental training process, and the present specification does not make any limitation.

[0101] Further, it is also necessary to determine the correspondence between each prompt vector obtained in step S200 and each prompt subnetwork obtained in the aforementioned step S102. Specifically, each obtained prompt subnetwork can be obtained based on the sample images and their labeled types of each reference image classification task, i.e., each obtained prompt subnetwork can learn the knowledge of each reference image classification task in advance, thereby improving the learning efficiency of the current image classification model.

[0102] And based on the foregoing, each prompt vector is also obtained based on the sample images corresponding to each reference image classification model respectively, therefore, in the present specification, the prompt vector and the prompt subnetwork determined based on the same reference image classification task are regarded as the prompt vector and the prompt subnetwork having the correspondence.

[0103] That is, for each prompt vector, the reference image classification task to which the sample image used by the prompt vector belongs is determined as the reference image classification task corresponding to the prompt vector, the prompt subnetwork trained based on the sample images and the labeled types of the reference image classification task corresponding to the prompt vector is determined, and the determined prompt subnetwork is regarded as the prompt subnetwork corresponding to the prompt vector, i.e., the correspondence exists between the two.

[0104] Optionally, each prompt vector can be obtained by the following scheme:

[0105] First, a plurality of sample images corresponding to different reference image classification tasks are obtained.

[0106] The reference image classification tasks can be each image classification task in the incremental training process of the image classification model in the lifelong learning, or other reference image classification tasks different from each image classification task in the incremental training process. The present specification does not make any limitation on the number and specific task content of the reference image classification tasks.

[0107] In the specification, the prompt vector is obtained by inputting a sample image corresponding to a reference image classification task into a pre-trained encoder, so that for each reference image classification task, the obtained sample image corresponding to the reference image classification task can be one or more, and the number of sample images corresponding to different reference image classification tasks can be the same or different. The specification does not limit this.

[0108] In addition, in the specification, the description of different reference image classification tasks can refer to the description of the difference between the historical image classification task and the current image classification task in the foregoing step S104, which will not be repeated here.

[0109] Secondly, the sample images respectively corresponding to the reference image classification tasks are respectively input into the pre-trained encoder to obtain the feature vectors of the sample images respectively corresponding to the reference image classification tasks.

[0110] Specifically, since the encoder is pre-trained based on the sample images in the general image library, the encoder can map the sample images respectively corresponding to the reference images into the vector space to obtain the feature vectors of the sample images corresponding to each reference image.

[0111] Then, the feature vectors of the sample images respectively corresponding to the image classification tasks are obtained as the obtained prompt vectors.

[0112] S202: inputting the sample image corresponding to the current image classification task into the pre-trained encoder to obtain the original feature of the sample image.

[0113] As described above, the pre-trained encoder can be pre-trained based on the sample images in the general image library. The encoder has the function of mapping the sample image of the current image classification task into the vector space and obtaining the feature vector of the sample image, so that the feature vector output by the encoder based on the input sample image corresponding to the current image classification task is the original feature of the sample image corresponding to the current image classification task.

[0114] S204: determining the weight corresponding to each prompt vector according to the original feature of the sample image and the prompt vectors.

[0115] Specifically, the similarity between the original feature of the sample image and each prompt vector can be determined. The higher the similarity, the higher the matching degree of the prompt vector and the original feature of the sample image of the current image classification task, and vice versa. The lower the similarity, the lower the matching degree of the prompt vector and the original feature of the sample image of the current image classification task.

[0116] In addition, if the similarity between a prompt vector and the original features of the sample image is lower than the preset lower limit of similarity, it indicates that the similarity between the prompt vector and the original features of the sample image is too low, and the prompt vector does not match the sample image of the current image classification task. Therefore, the prompt vector can be excluded, and the weight corresponding to the prompt vector is not determined, that is, the prompt subnetwork corresponding to the prompt vector will not be used to construct the current prompt network.

[0117] Subsequently, based on the similarities between the sample image of the current image classification task and each prompt vector, weights of each prompt vector are assigned. The similarity between the sample image of the current image classification task and each prompt vector can be directly used as the weight corresponding to each prompt vector, but in actual applications, the sum of the weights of each prompt vector is usually 1, and directly using the similarity as the weight of each prompt vector may not satisfy the constraint that the sum of the weights is 1. Therefore, the similarity between the sample image of the current image classification task and each prompt vector can be normalized to obtain a normalized result as the weight corresponding to each prompt vector.

[0118] The weight corresponding to each prompt vector is determined by similarity calculation, which not only considers the visual content of the sample image of the current image classification task, but also integrates the context information of the current image classification task, so that the generated weight can more accurately reflect the overall characteristics and learning requirements of the current image classification task. In this way, the weights of each prompt subnetwork in the current prompt network constructed can ensure that the model can select the most suitable prompt subnetwork for learning and adaptation according to the specific characteristics and task requirements of the input image. Through this weight generation mechanism, the model can achieve higher accuracy and efficiency in the continuous learning process.

[0119] Optionally, to further improve the accuracy of the weight corresponding to each prompt vector, the original features can be further processed based on the original features of the sample image. In this specification, a self-attention mechanism processing scheme is adopted for the original features, so that the model can concentrate its computing resources on the most critical image parts for the current image classification task. Specifically, first, according to the original features of the sample image, the target features of the sample image are determined through a self-attention mechanism, and then, according to the similarities between the target features of the sample image and each prompt vector, the weight corresponding to each prompt vector is determined.

[0120] Specifically, the original features of the sample image are further processed using the self-attention mechanism. By introducing the self-attention mechanism, the image content of the sample image can be more deeply understood and analyzed, so that the matching degree between the sample image of the current image classification task and each prompt vector can be more accurately obtained, thereby improving the accuracy of the weight corresponding to each prompt vector. Further, the image can be more accurately classified and understood.

[0121] The core of the self-attention mechanism is to enable the target features of the sample image to better reflect the key areas in the sample image, and to generate more refined query indexes based on the features of these key areas, and then to query the matching degree between each prompt vector and the sample image of the current image classification task based on the more refined query indexes.

[0122] This mechanism achieves this by calculating the relationships between the original features of the sample image, thereby determining which features are important for understanding the overall content and structure of the sample image. In this process, different regions (pixels) are assigned different weights, thereby concentrating their computing resources on those image parts that are most critical for the current image classification task. The target features of the sample image obtained through the self-attention mechanism can more comprehensively and accurately represent the key information of the sample image. This not only includes the basic visual features of the image, but also integrates the evaluation of the importance of the features of different regions of the sample image. Such processing enables the subsequent construction of the current prompt network and the optimization of the update of the current prompt network to more effectively target the key content of the image, thereby improving the classification accuracy and efficiency in the continuous learning process.

[0123] S206: Determine the weight corresponding to each prompt subnetwork based on the correspondence between each prompt vector and each prompt subnetwork and the weight corresponding to each prompt vector.

[0124] In this step, based on the correspondence between each prompt vector and each prompt subnetwork determined in the preceding step S200, for each prompt subnetwork, the weight of the prompt vector corresponding to the prompt subnetwork is taken as the weight corresponding to the prompt subnetwork.

[0125] S208: Determine the current prompt network based on each prompt subnetwork and the weight corresponding to each prompt subnetwork.

[0126] Thus, the current prompt network is composed of each prompt subnetwork and the weight corresponding to each prompt subnetwork.

[0127] It should be noted that the weight corresponding to each prompt subnetwork is not a network parameter of each prompt subnetwork, and the weight corresponding to each prompt subnetwork will be used as a weighting parameter of the output of each prompt subnetwork to weight the vector output by each prompt subnetwork. In general, the weight corresponding to each prompt subnetwork will not be updated during the training of the current prompt network. Of course, according to different actual situations, the weight corresponding to each prompt subnetwork included in the current prompt network can also be updated during the training of the current prompt network, and the present specification does not limit this.

[0128] Based on the scheme shown in Figure 3 , different prompt networks correspond to different characteristics of image classification tasks, and the current prompt network is obtained by combining each prompt subnetwork based on the weight corresponding to each prompt subnetwork, so that the current prompt network can meet the specific needs of the current image classification task. Such a weighted combination not only includes the visual content of the image, but also contains the context information related to the task to which the image belongs, so that the generated current prompt network can more accurately match the current learning needs of the image classification model. In addition, compared with the traditional single prompt method, the generation method of the current prompt network based on the prompt subnetwork has higher flexibility and scalability. Through this method, the prompt capacity of the model can be expanded with the complexity of the task, rather than being fixed to a single prompt length. This dynamic adjustment capability enables the model to more effectively cope with various learning tasks, thereby improving the overall classification performance and adaptability.

[0129] In one or more embodiments of the present specification, based on the scheme shown in Figure 2 , the current prompt network obtained actually contains each prompt subnetwork and the weight corresponding to each prompt subnetwork, so that when step S106 shown in Figure 1 is performed, it is actually realized by the following steps:

[0130] First step: input the sample image into the current prompt network in the image classification model to be trained, and obtain each first feature of the sample image through each prompt subnetwork included in the current prompt network.

[0131] The model structure of the current image classification model based on this scheme can be referred to the solid line part of Figure 2 . In the present specification, the current prompt network is directly embedded in the specific layers of the historical image classification model, especially the first few layers or the layers related to the image classification task. Thus, as Figure 2In the current image classification model shown, the sample image of the current image classification task input into the current image classification model directly enters the current prompt network. Since the current prompt network includes various prompt subnetworks, and the various prompt subnetworks can be arranged side by side in the specification, that is, the various prompt subnetworks have the same input, the outputs of the various prompt subnetworks are fused as the output of the current prompt network. Therefore, the sample image of the current image classification model input into the current prompt network is input into each prompt subnetwork, and each prompt subnetwork can obtain each first feature based on the sample image.

[0132] Second step: weighting each first feature according to the weight corresponding to each prompt subnetwork included in the current prompt network, and obtaining a second feature of the sample image according to the weighted first features.

[0133] Then, the first features are fused. In the specification, the first features are weighted with the weight corresponding to each prompt subnetwork included in the current prompt network as the weighting coefficient, and then the weighted first features are fused to obtain the second feature of the sample image.

[0134] Third step: inputting the second feature into the pre-trained encoder to obtain a third feature of the sample image.

[0135] Fourth step: inputting the third feature into the classification network to obtain a predicted type of the sample image.

[0136] In the training process of the foregoing image classification model, the historical prompt network in the historical image classification model is not actually applied. Therefore, after optimizing the parameters of the current prompt network based on the foregoing scheme, the trained current image classification model can be constructed based on the optimized current prompt network, the trained historical prompt network, the pre-trained encoder, and the classification network.

[0137] Of course, in one or more embodiments of the specification, the parameters of the current prompt network can also be optimized in combination with the historical prompt network in the historical image classification model. The specific scheme is as follows:

[0138] First step: inputting the sample image into the current prompt network in the image classification model to be trained to obtain each first feature of the sample image through each prompt subnetwork included in the current prompt network.

[0139] Second step: weighting each first feature according to the weight corresponding to each prompt subnetwork included in the current prompt network, and obtaining a second feature of the sample image according to the weighted first features.

[0140] Thirdly, input the sample image corresponding to the current image classification task into the history prompt network of the image classification model to be trained, and obtain the history feature of the sample image through the history prompt network.

[0141] The model structure of the current image classification model based on the scheme is shown by solid lines and dashed lines in Figure 2 Unlike the foregoing scheme, in the alternative embodiment, the sample image of the current image classification task input into the image classification model to be trained enters the history prompt network, which has been trained based on the history image classification task in the continuous learning process of the image classification model, that is, the history prompt network has mastered the knowledge of the history image classification task. The history feature of the sample image obtained by inputting the sample image of the current image classification task into the history prompt network actually defines the degree of association between the current image classification task and the history image classification task, and uses the degree of association to assist the training of the current prompt network or assist the classification of the input sample input into the current image classification model.

[0142] Fourthly, input the second feature and the history feature as input into the pre-trained encoder to obtain the third feature of the sample image output by the encoder.

[0143] Fifthly, input the third feature of the sample image into the classification network to obtain the predicted type of the sample image.

[0144] But even if the predicted type of the sample image is obtained by the history prompt network based on the foregoing scheme, in the back propagation, the current prompt network is optimized with the minimization of the difference between the predicted type and the labeled type as the optimization target, and the current image classification model is obtained according to the optimized current prompt network. Figure 1

[0145] In one or more embodiments of the present specification, Figure 1 The acquisition of each prompt subnetwork matched with the current image classification task in step S102 shown in the foregoing scheme can be realized by the following scheme:

[0146] Firstly, acquire sample images and labeled types corresponding to a plurality of different reference image classification tasks.

[0147] Secondly, acquire the prompt subnetwork to be trained.

[0148] Thirdly, for each reference image classification task, optimize the parameters of the prompt subnetwork to be trained according to the sample image and the labeled type corresponding to the reference image classification task to obtain the prompt subnetwork corresponding to the reference image classification task.

[0149] ​Specifically, the prompt subnetworks obtained in step S102 can be obtained by pre-training based on sample images and label types corresponding to a plurality of different reference image classification tasks. The prompt subnetwork to be trained can include one or more neural networks. Generally, the network parameters of the prompt subnetwork to be trained are initialized and not optimized. In addition, the number of neural network layers of the prompt subnetwork to be trained used to obtain the prompt subnetworks of different reference image classification tasks can be the same or different, that is, the number of neural network layers of the prompt subnetworks corresponding to different reference image classification tasks can be the same or different, which is not limited in the present specification.

[0150] In addition, as described above, when the current prompt network is constructed based on the prompt subnetworks, the correspondence between the prompt subnetworks and the prompt vectors needs to be determined to determine the matching degree of the prompt subnetworks and the current image classification task. Therefore, generally, the prompt subnetworks and the prompt vectors are one-to-one corresponding, and the sample images of the reference image classification tasks used by the prompt subnetworks with the correspondence during pre-optimization are the same as the sample images of the reference image classification tasks used by the prompt vectors with the correspondence during determination.

[0151] Step 4: The prompt subnetworks corresponding to each reference image classification task are obtained as the obtained prompt subnetworks.

[0152] In addition, in an optional embodiment of the present specification, after obtaining the trained image classification model in step S104, the trained image classification model can be tested before step S106 is performed to evaluate the performance and accuracy of the trained image classification model. Specifically:

[0153] First, a first test image corresponding to the current image classification task and a label type of the first test image are obtained, and a second test image corresponding to a historical image classification task and a label type of the second test image are obtained.

[0154] The historical image classification task is a model training task that is ranked before the current image classification task in the model training process of the image classification model based on sustainable learning.

[0155] The first test image and the label type thereof can be used to test whether the trained current image classification model masters the knowledge of the current image classification task, and the second test image and the label type thereof can be used to test whether the trained current image classification model does not have the knowledge of the abnormal historical image classification task.

[0156] Then, the trained image classification model is tested according to the first test image, the annotation type of the first test image, the second test image and the annotation type of the second test image, and it is determined that the test result of the trained image classification model meets the preset condition.

[0157] Therefore, the trained current image classification model is tested based on the first test image and the annotation type thereof, and the second test image and the annotation type thereof, and the test result can include the classification performance of the current image classification model on the images of the current image classification task and the classification performance of the current image classification model on the images of the historical image classification task.

[0158] In addition, only when the classification performance of the current image classification model on the images of the current image classification task and the classification performance of the current image classification model on the images of the historical image classification task both meet the preset performance requirement, it is determined that the test result of the trained image classification model meets the preset condition.

[0159] The above is the image classification method provided by one or more embodiments of the present specification. Based on the same idea, the present specification also provides a corresponding image classification device, as shown in Figure 4 .

[0160] Figure 4 An image classification device provided by the present specification includes:

[0161] The acquisition module 300 is configured to acquire sample images corresponding to the current image classification task and annotation types of the sample images in response to the current image classification task.

[0162] The current prompt network determination module 302 is configured to acquire prompt subnetworks matched with the current image classification task, and determine a current prompt network according to the prompt subnetworks.

[0163] The model construction module 304 is configured to obtain a current image classification model to be trained according to the current prompt network and a trained historical image classification model. The historical image classification model is trained in advance according to sample images corresponding to a historical image classification task, and the historical image classification task is different from the current image classification task.

[0164] The training module 306 is configured to input the sample images corresponding to the current image classification task into the current image classification model to be trained, obtain a predicted type of the sample images output by the image classification model, and minimize the difference between the predicted type and the annotation type as an optimization target to optimize the current prompt network, and obtain a trained current image classification model according to the optimized current prompt network.

[0165] The classification module 308 is configured to input the obtained image to be classified into the trained current image classification model to obtain a type of the image to be classified in response to an image classification request.

[0166] Optionally, the current prompt network determination module 302 is specifically configured to: obtain prompt vectors, and determine a correspondence between the prompt vectors and the prompt subnetworks; input sample images corresponding to the current image classification task into a pre-trained encoder to obtain original features of the sample images; determine weights corresponding to the prompt vectors respectively according to the original features of the sample images and the prompt vectors; determine weights corresponding to the prompt subnetworks respectively according to the correspondence between the prompt vectors and the prompt subnetworks and the weights corresponding to the prompt vectors respectively; and determine the current prompt network according to the prompt subnetworks and the weights corresponding to the prompt subnetworks respectively.

[0167] Optionally, the current prompt network determination module 302 is specifically configured to: determine target features of the sample images by a self-attention mechanism according to the original features of the sample images; and determine weights corresponding to the prompt vectors respectively according to similarities between the target features of the sample images and the prompt vectors, wherein a similarity between the target features of the sample images and the prompt vectors is in a positive proportional relationship with the weight corresponding to the prompt vector.

[0168] Optionally, the training module 306 is specifically configured to: input sample images corresponding to the current image classification task into a current image classification model to be trained to obtain first features of the sample images respectively by the prompt subnetworks included in the current prompt network; weight the first features according to the weights corresponding to the prompt subnetworks included in the current prompt network respectively, and obtain second features of the sample images according to the weighted first features; input the second features into a pre-trained encoder in the current image classification model to obtain third features of the sample images; and input the third features into a classification network in the current image classification model to obtain a predicted type of the sample images.

[0169] Optionally, the current prompt network determination module 302 is specifically configured to: determine a plurality of different reference image classification tasks matched with the current image classification task according to the current image classification task; obtain sample images corresponding to the reference image classification tasks respectively; and input the sample images corresponding to the reference image classification tasks respectively into a pre-trained encoder to obtain feature vectors of the sample images corresponding to the reference image classification tasks respectively as the obtained prompt vectors.

[0170] Optionally, the current prompt network determination module 302 is specifically configured to determine, according to the current image classification task, a plurality of different reference image classification tasks matched with the current image classification task; acquire sample images corresponding to each reference image classification task and the labeled types of the sample images corresponding to each reference image classification task; acquire a plurality of layers of neural networks to be trained as prompt subnetworks to be trained; for each reference image classification task, optimize the parameters of the prompt subnetworks to be trained according to the sample images corresponding to the reference image classification task and the labeled types of the sample images corresponding to the reference image classification task, to obtain a prompt subnetwork corresponding to the reference image classification task; and take the prompt subnetworks corresponding to each reference image classification task respectively as the prompt subnetworks acquired and matched with the current image classification task.

[0171] Optionally, the apparatus further comprises:

[0172] The test module 310 is specifically configured to acquire a first test image corresponding to the current image classification task and the labeled type of the first test image, and acquire a second test image corresponding to the historical image classification task and the labeled type of the second test image; and test the trained current image classification model according to the first test image, the labeled type of the first test image, the second test image and the labeled type of the second test image, to determine that the test result of the trained current image classification model satisfies a preset condition.

[0173] The present specification also provides a computer readable storage medium storing a computer program, and the computer program can be used to execute the image classification method shown in the present specification. Figure 1 The present specification also provides a schematic structural diagram of an electronic device shown in the present specification. As shown in the present specification, at the hardware level, the electronic device comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course can also comprise other hardware required by a business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, to implement the image classification method shown in the present specification. Of course, in addition to the software implementation manner, the present specification does not exclude other implementation manners, such as a logic device or a combination of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0174] The present specification also provides a schematic structural diagram of an electronic device shown in the present specification. As shown in the present specification, at the hardware level, the electronic device comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course can also comprise other hardware required by a business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, to implement the image classification method shown in the present specification. Of course, in addition to the software implementation manner, the present specification does not exclude other implementation manners, such as a logic device or a combination of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device. Figure 5 The present specification also provides a schematic structural diagram of an electronic device shown in the present specification. As shown in the present specification, at the hardware level, the electronic device comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course can also comprise other hardware required by a business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, to implement the image classification method shown in the present specification. Of course, in addition to the software implementation manner, the present specification does not exclude other implementation manners, such as a logic device or a combination of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device. Figure 5 The present specification also provides a schematic structural diagram of an electronic device shown in the present specification. As shown in the present specification, at the hardware level, the electronic device comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course can also comprise other hardware required by a business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, to implement the image classification method shown in the present specification. Of course, in addition to the software implementation manner, the present specification does not exclude other implementation manners, such as a logic device or a combination of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device. Figure 1 The present specification also provides a schematic structural diagram of an electronic device shown in the present specification. As shown in the present specification, at the hardware level, the electronic device comprises a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course can also comprise other hardware required by a business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs, to implement the image classification method shown in the present specification. Of course, in addition to the software implementation manner, the present specification does not exclude other implementation manners, such as a logic device or a combination of software and hardware, and so on, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0175] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A digital system is "integrated" on a PLD by the designer programming it, rather than by asking a chip manufacturer to design and fabricate a custom integrated circuit chip. Moreover, instead of manually fabricating integrated circuit chips, this programming is now mostly implemented by "logic compiler" software, which is similar to software compilers used in program development, and the original code to be compiled is written in a specific programming language, which is called a hardware description language (HDL), and there are many such languages, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should be aware that, as long as the method flow is logically programmed in the above-mentioned hardware description languages and programmed into an integrated circuit, a hardware circuit implementing the logical method flow can be easily obtained.

[0176] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to being implemented in pure computer readable program code, the controller can equally well be implemented to perform the same functions using logic gates, switches, an application specific integrated circuit, a programmable logic controller and an embedded microcontroller, etc. by means of a logical programming of the method steps. The controller can thus be considered as a hardware component, and the means comprised therein for performing the various functions can be considered as structures within the hardware component. Alternatively, the means for performing the various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0177] The systems, apparatuses, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0178] For the sake of description, the above apparatuses are described in various units by functions respectively. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the present specification.

[0179] Those skilled in the art will understand that the embodiments of the present specification can be provided as a method, a system or a computer program product. Therefore, the present specification can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0180] The specification is presented with reference to flow diagrams and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks in the flow diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0181] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0182] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow diagrams and / or block diagrams block or blocks. Figure 1 The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks. ​ The flow diagrams and / or block diagrams in the specification can present a method, apparatus or computer program product according to embodiments of the specification. Flow diagrams and / or block diagrams can also present a method, apparatus or computer program product to achieve functions specified in flow diagrams and / or block diagrams block or blocks.

[0183] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0184] The memory can include non-persistent memory and / or storage mechanisms such as, for example, random access memory (RAM), non-volatile memory (NVM), and / or a persistent memory such as, for example, read-only memory (ROM) or flash memory. The memory is an example of computer-readable media.

[0185] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0186] It should also be noted that the terms "comprising", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a list of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0187] Those skilled in the art will appreciate that embodiments of the present specification can be provided as methods, systems or computer program products. Therefore, the present specification can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, the present specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0188] The present specification can be described in the general context of computer-executable instructions, such as program modules, executed by computers. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in both local and remote computer storage media, including storage devices.

[0189] The various embodiments described in this specification are described using a numbering of embodiments approach: these are each individually integrated contributions pertaining to different but related aspects of the description. Each of the various embodiments can stand on its own, and each can be combined with the subject matter of other embodiments to produce further embodiments. Where appropriate, therefore, the contents of the specification can be regarded as being incorporated by reference, including the description, drawings, claims, abstract and the like.

[0190] The above description is embodied in the form of examples only, and is not intended to limit the specification. The specification can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the specification should be included in the scope of the claims of the specification.

Claims

1. An image classification method, characterized in that, include: In response to the current image classification task, obtain the sample image corresponding to the current image classification task and the annotation type of the sample image; Obtain each cue subnet that matches the current image classification task, and determine the current cue network based on each cue subnet; Based on the current prompting network and the trained historical image classification model, a current image classification model to be trained is obtained; wherein, the historical image classification model is pre-trained based on sample images corresponding to the historical image classification task, and the historical image classification task is different from the current image classification task; The sample image corresponding to the current image classification task is input into the current image classification model to be trained, and the predicted type of the sample image output by the image classification model is obtained. The current prompting network is optimized with the goal of minimizing the difference between the predicted type and the labeled type. The trained current image classification model is obtained based on the optimized current prompting network. In response to an image classification request, the acquired image to be classified is input into the trained current image classification model to obtain the type of the image to be classified.

2. The method as described in claim 1, characterized in that, The step of determining the current prompting network based on each prompting subnet specifically includes: Obtain each cue vector and determine the correspondence between each cue vector and each cue subnet; The sample image corresponding to the current image classification task is input into the pre-trained encoder to obtain the original features of the sample image; Based on the original features of the sample image and each cue vector, determine the weight corresponding to each cue vector; Based on the correspondence between each cue vector and each cue subnet, and the weights corresponding to each cue vector, the weights corresponding to each cue subnet are determined. The current prompting network is determined based on each prompting subnet and its corresponding weight.

3. The method as described in claim 2, characterized in that, The step of determining the weights corresponding to each cue vector based on the original features of the sample image and each cue vector specifically includes: Based on the original features of the sample image, the target features of the sample image are determined through a self-attention mechanism; Based on the similarity between the target features of the sample image and each of the prompt vectors, the weights corresponding to each prompt vector are determined, wherein the similarity between the target features of the sample image and the prompt vectors is directly proportional to the weights corresponding to the prompt vectors.

4. The method as described in claim 2, characterized in that, The step of inputting the sample image corresponding to the current image classification task into the current image classification model to be trained, and obtaining the predicted type of the sample image output by the image classification model, specifically includes: The sample image corresponding to the current image classification task is input into the current image classification model to be trained, and the first features of the sample image are obtained through the prompt subnetworks contained in the current prompt network. Based on the weights corresponding to each of the prompt subnets contained in the current prompt network, the first features are weighted, and the second features of the sample image are obtained based on the weighted first features. The second feature is input into the pre-trained encoder in the current image classification model to obtain the third feature of the sample image; The third feature is input into the classification network of the current image classification model to obtain the predicted type of the sample image.

5. The method as described in claim 2, characterized in that, The acquisition of each prompt vector specifically includes: Based on the current image classification task, determine multiple different reference image classification tasks that match the current image classification task; Obtain the sample images corresponding to each reference image classification task; The sample images corresponding to each reference image classification task are input into the pre-trained encoder to obtain the feature vectors of the sample images corresponding to each reference image classification task, which are used as the obtained prompt vectors.

6. The method as described in claim 1, characterized in that, The acquisition of each cue subnet matching the current image classification task specifically includes: Based on the current image classification task, determine multiple different reference image classification tasks that match the current image classification task; Obtain the sample images corresponding to each reference image classification task and the annotation type of the sample images corresponding to each reference image classification task; Obtain several layers of neural networks to be trained as cue subnets to be trained; For each reference image classification task, the parameters of the prompt subnet to be trained are optimized according to the sample images corresponding to the reference image classification task and the annotation type of the sample images corresponding to the reference image classification task, so as to obtain the prompt subnet corresponding to the reference image classification task. The prompt subnets corresponding to each reference image classification task are used as the obtained prompt subnets that match the current image classification task.

7. The method as described in claim 1, characterized in that, Before inputting the acquired image to be classified into the trained current image classification model to obtain the type of the image to be classified, the method further includes: Obtain the first test image and the annotation type of the first test image corresponding to the current image classification task, and obtain the second test image and the annotation type of the second test image corresponding to the historical image classification task; Based on the first test image, the annotation type of the first test image, the second test image, and the annotation type of the second test image, the trained current image classification model is tested to determine whether the test result of the trained current image classification model meets the preset conditions.

8. An image classification device, characterized in that, include: The acquisition module is used to acquire, in response to the current image classification task, the sample image corresponding to the current image classification task and the annotation type of the sample image; The current prompting network determination module is used to obtain each prompting subnet that matches the current image classification task, and determine the current prompting network based on each prompting subnet; The model building module is used to obtain the current image classification model to be trained based on the current prompting network and the trained historical image classification model; wherein, the historical image classification model is pre-trained based on sample images corresponding to the historical image classification task, and the historical image classification task is different from the current image classification task; The training module is used to input the sample image corresponding to the current image classification task into the current image classification model to be trained, obtain the predicted type of the sample image output by the image classification model, optimize the current prompting network with the goal of minimizing the difference between the predicted type and the labeled type, and obtain the trained current image classification model based on the optimized current prompting network. The classification module is used to respond to an image classification request by inputting the acquired image to be classified into the trained current image classification model to obtain the type of the image to be classified.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 7.

Images