Image Classification Method, Device and Image Classification Model Based on Hierarchical Hints

Abstract

The present invention provides an image classification method, apparatus and image classification model based on hierarchical prompts. The method includes: determining a task identifier of an image to be classified; determining target image features based on a task prompt vector corresponding to the current task corresponding to the task identifier, a generalization prompt vector corresponding to the current task, and initial image features of the image to be classified; the task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and virtual image features corresponding to each image category in the historical tasks, the virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks, and the generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning; classifying the image to be classified based on the target image features to obtain an image classification result. The image classification method based on hierarchical prompts provided by the present invention can solve the problem of catastrophic forgetting existing in continuous learning.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to an image classification method, device, and image classification model based on hierarchical prompts. Background Art

[0002] Although deep learning has achieved remarkable results in multiple fields, it still faces a problem that deep learning models need to be trained on independently and identically distributed data. In the current mode, deep learning models cannot handle non-stable data distributions, which leads to the need to continuously retrain deep learning models when new tasks or new categories continuously appear, thus increasing the training cost. More seriously, if the deep learning model trained for previous tasks is fine-tuned to adapt to the current task, catastrophic forgetting will occur, that is, the performance of the deep learning model on previous tasks will drop significantly. To solve this problem, continual learning aims to enable deep learning models to learn new task knowledge while remembering historical task knowledge.

[0003] Recently, prompt learning has received extensive attention because it only needs to fine-tune a small number of learnable prompt parameters. Learnable prompt parameters are used to save task knowledge during the training process and select relevant prompt parameters during the inference process. However, these learned prompt parameters only focus on obtaining discriminative knowledge of the current task and ignore the preservation of past task knowledge, resulting in poor training performance of deep learning models for historical task knowledge. Summary of the Invention

[0004] The present invention provides an image classification method, device, and image classification model based on hierarchical prompts to solve the problem of catastrophic forgetting of historical task knowledge in deep learning models in the prior art.

[0005] The present invention provides an image classification method based on hierarchical prompts, including:

[0006] Determine the task identifier of the image to be classified;

[0007] Based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, determine the target image features; wherein, the task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and the virtual image features corresponding to each image category in the historical tasks, the virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks, the generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning, and the initial image features of the image to be classified are obtained after feature extraction of the image to be classified;

[0008] Classify the image to be classified based on the target image features to obtain an image classification result.

[0009] In some embodiments, before determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, the method further includes:

[0010] Based on a Bayesian neural network, model the data distribution of the image classes of each task to determine the initial class prompt vector corresponding to each task;

[0011] Based on the initial class prompt vector corresponding to each task, obtain the virtual image feature corresponding to each task;

[0012] Combine the initial image features of the sampled images in each task, the task prompt vector corresponding to each task, and the generalization prompt vector corresponding to each task to determine the extended image features of the sampled images in each task;

[0013] Based on the extended image features of the sampled images in each task and the virtual image features corresponding to each task, train the initial class prompt vector corresponding to each task until the target class prompt vector corresponding to each task is obtained.

[0014] In some embodiments, before determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, the method further includes:

[0015] Combine the initial image feature corresponding to the i-th task, the generalization prompt vector corresponding to the i-th task, and the task prompt vector corresponding to the i-th task to obtain the extended image feature corresponding to the i-th task;

[0016] Combine the virtual image features corresponding to the previous i - 1 tasks respectively, the generalization prompt vector corresponding to the i-th task, and the task prompt vector corresponding to the i-th task to obtain the extended virtual image features corresponding to the previous i - 1 tasks respectively; wherein, the virtual image features corresponding to the previous i - 1 tasks respectively are determined based on the target class prompt vectors corresponding to the previous i - 1 tasks respectively;

[0017] Based on the extended image feature corresponding to the i-th task and the extended virtual image features corresponding to the previous i - 1 tasks respectively, train the task prompt vector corresponding to the i-th task.

[0018] In some embodiments, before determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, the method further includes:

[0019] Obtain multiple images in the i-th task, perform data augmentation on the multiple images, and obtain the augmented multiple images;

[0020] Based on the label information corresponding to the augmented multiple images and the target image features corresponding to the augmented multiple images respectively, train the generalization indication vector corresponding to the i-th task through self-supervised learning.

[0021] In some embodiments, the determining the task identifier of the image to be classified includes:

[0022] Construct a query model based on the task prompt vector corresponding to each task and the generalization prompt vector corresponding to each task;

[0023] Based on the query model, determine the task identifier of the image to be classified.

[0024] The present invention provides an image classification model based on hierarchical prompts, which applies the image classification method based on hierarchical prompts as described above, including:

[0025] An embedding layer for extracting features of the image to be classified and outputting the initial image features of the image to be classified;

[0026] A Transformer layer for inputting the task prompt vector corresponding to the current task, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, and outputting the target image features;

[0027] A classifier for classifying the image to be classified based on the target image features to obtain an image classification result;

[0028] Wherein, the Transformer layer is constructed based on a Bayesian distribution alignment module, a general knowledge exploration module, and a cross-task knowledge mining module;

[0029] The Bayesian distribution alignment module is used to determine the category prompt vector corresponding to each task based on the data distribution corresponding to each image category in each task;

[0030] The cross-task knowledge mining module is used to determine the task prompt vector corresponding to the current task based on the initial image features of the image to be classified and the data distribution corresponding to each image category in the historical tasks;

[0031] The general knowledge exploration module is used to train multiple images in the current task through self-supervised learning to obtain a generalization prompt vector corresponding to the current task.

[0032] The present invention also provides an image classification device based on hierarchical prompts, including:

[0033] A first determination module, configured to determine a task identifier of an image to be classified;

[0034] A second determination module, configured to determine a target image feature based on a task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and an initial image feature of the image to be classified; wherein, the task prompt vector corresponding to the current task is determined based on the initial image feature of the image to be classified and virtual image features corresponding to each image category in the historical tasks, the virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks, the generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning, and the initial image feature of the image to be classified is obtained after feature extraction of the image to be classified;

[0035] A classification module, configured to classify the image to be classified based on the target image feature to obtain an image classification result.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the hierarchical prompt-based image classification method as described in any one of the above.

[0037] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the hierarchical prompt-based image classification method as described in any one of the above.

[0038] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the hierarchical prompt-based image classification method as described in any one of the above.

[0039] The hierarchical prompt-based image classification method, device, and image classification model provided by the present invention can well solve the catastrophic forgetting problem of historical task knowledge caused by continuous learning by using task prompt vectors to utilize historical task knowledge and using generalization prompt vectors to learn highly generalizable knowledge, thereby improving the classification accuracy of the image classification model on historical tasks, and at the same time having good generalization ability, and can achieve good results on both large-scale datasets and small-scale datasets. Description of the Drawings

[0040] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0041] Figure 1 is a schematic flowchart of the image classification method based on hierarchical prompts provided by the present invention;

[0042] Figure 2 is a schematic algorithm flowchart of the image classification method based on hierarchical prompts provided by the present invention;

[0043] Figure 3 is a schematic structural diagram of the image classification device based on hierarchical prompts provided by the present invention;

[0044] Figure 4 is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention fall within the protection scope of the present invention.

[0046] The following will describe Figures 1 - 4 the image classification method, device, and image classification model based on hierarchical prompts of the present invention.

[0047] Figure 1 is a schematic flowchart of the image classification method based on hierarchical prompts provided by the present invention. Referring to Figure 1 , the image classification method based on hierarchical prompts provided by the present invention includes: Step 110, Step 120, and Step 130.

[0048] Step 110: Determine the task identifier of the image to be classified;

[0049] Step 120. Determine target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified. The task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and the virtual image features corresponding to each image category in the historical tasks. The virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks. The generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning. The initial image features of the image to be classified are obtained after feature extraction of the image to be classified.

[0050] Step 130. Classify the image to be classified based on the target image features to obtain an image classification result.

[0051] The execution subject of the image classification method based on hierarchical prompts provided by the present invention may be an electronic device, a component in the electronic device, an integrated circuit, or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device may be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The present invention does not make specific limitations.

[0052] Next, taking a computer executing the image classification method based on hierarchical prompts provided by the present invention as an example, the technical solution of the present invention will be described in detail.

[0053] It should be noted that the present invention uses an image classification model to continuously learn image classification tasks. When a new task arrives, the task prompt vector is used to utilize the historical task knowledge and the current task knowledge, and the generalization prompt vector is used to learn highly generalized knowledge, so as to alleviate the catastrophic forgetting problem of the historical task knowledge in the continuous learning process of the image classification model, thereby improving the classification accuracy of the image classification model on historical tasks.

[0054] In continuous learning, the image classification model needs to learn knowledge of non-static data from a series of tasks where T is the total number of tasks, and the tasks may be image classification tasks.

[0055] The i-th task consisting of an image and its corresponding label , and represent the image set and the label set in the i-th task respectively.

[0056] It should be noted that for different tasks, their label sets are disjoint, that is, if i≠j, then

[0057] In addition, each task data can only be obtained once, that is, historical task data cannot be obtained when learning the current task data. In actual implementation, the present invention performs a non-storage class incremental setting, that is, the number of samples stored for historical tasks is zero during the training process, and the task identifier cannot be accessed during the inference process.

[0058] The Hierarchical-Prompts (H-Prompts) in the present invention include category prompt vectors, task prompt vectors, and generalization prompt vectors. Hierarchical prompts can be used to save the knowledge of each category, capture historical task knowledge and current task knowledge, and learn generalization knowledge.

[0059] The training process of the hierarchical prompt will be described one by one below.

[0060] I. Analog prompt vector

[0061] In some embodiments, before step 110, the image classification method based on hierarchical prompts further includes:

[0062] Based on a Bayesian neural network, model the data distribution of the image categories of each task, and determine the initial category prompt vector corresponding to each task;

[0063] Based on the initial category prompt vector corresponding to each task, obtain the virtual image features corresponding to each task;

[0064] Combine the initial image features of the sampled images in each task, the task prompt vector corresponding to each task, and the generalization prompt vector corresponding to each task to determine the extended image features of the sampled images in each task;

[0065] Based on the extended image features of the sampled images in each task and the virtual image features corresponding to each task, train the initial category prompt vector corresponding to each task until the target category prompt vector corresponding to each task is obtained.

[0066] In actual implementation, a Bayesian neural network is used to capture the data distribution of each image category. Using a learnable mean vector and a diagonal covariance matrix To define the multivariate Gaussian distribution The learnable mean vector μ i,m and the diagonal covariance matrix ∑ i,m Defined as the initial category hint vector c to be optimized i,m , thus achieving the i-th task Modeling the data distribution of the m-th image category.

[0067] It can be understood that the i-th task can be any task in the total tasks, and the training process of the target category prompt vector corresponding to the i-th task is the same as the training process of each other task. After the training is completed, the target category prompt vector corresponding to each task can be determined.

[0068] From the image dataset of the i-th task Determine a sample image x in i,m , the initial image feature of the i-th task is defined as f i,m .

[0069] By combining a fixed task cue vector t i and the generalization hint vector g i , the sampled image x of the i-th task i,m The extended image feature is

[0070] At the same time, the image feature f′ i,m Defined as the initial category hint vector c i,m The virtual image features of the i-th task are sampled from .

[0071]

[0072] By combining a fixed task cue vector t i , generalization hint vector g i And the virtual image feature f′ of the i-th task i,m , determine the virtual image feature f′ of the i-th task i,m The corresponding extended virtual image feature is Bayesian neural networks extend virtual image features by aligning them in an adversarial manner and extended image features The distribution of the initial category prompt vector c is trained i,m , thus obtaining the target category prompt vector.

[0073] In particular, using a discriminative classifier To distinguish the sampled image x i,m Extended image features and the extended virtual image features of the i-th task

[0074] Specifically, the discriminative classifier aims to classify the true representation of the extended image features of the sampled image x i,m with the correct label m, and misclassify the virtual representation of the extended virtual image features of the i-th task as the false label

[0075]

[0076] where l(·,·) represents the cross-entropy loss; denotes the loss function of the discriminative classifier for distinguishing between the extended image features and the extended virtual image features

[0077] In addition, the weights of different image classes in the discriminative classifier are constrained to be orthogonal to each other to enhance the discriminability of the classifier.

[0078] Define the class weight matrix as Define the identity matrix as where w i is the weight of the i-th image class in

[0079]

[0080] where l M (·,·) represents the mean squared error loss.

[0081] In addition, the virtual image features f′ of the i-th task sampled from the initial class prompt vector c i,m are optimized to deceive the discriminative classifier i,m that is, to make the discriminative classifier correctly classify the virtual representation of the virtual image features of the i-th task as follows:

[0082]

[0083] where denotes the loss function of the class prompt vector c i,m

[0084] Therefore, the loss function of the Bayesian neural network is:

[0085]

[0086] ​​​Through the initial class prompt vector c i,m and the discriminative classifier through adversarial training between them, the data distribution of the m-th image class in the i-th task can be modeled as: Thus, the knowledge of the corresponding image class is remembered through the target class prompt vector.

[0087] The image classification method based on hierarchical prompts provided by the present invention can reasonably utilize the class prompt vector to save the knowledge of historical tasks, thereby avoiding catastrophic forgetting.

[0088] II. Task Prompt Vector

[0089] In some embodiments, before step 110, the image classification method based on hierarchical prompts further includes:

[0090] Combining the initial image features corresponding to the i-th task, the generalization prompt vector corresponding to the i-th task, and the task prompt vector corresponding to the i-th task to obtain the extended image features corresponding to the i-th task;

[0091] Combining the virtual image features corresponding to the previous i - 1 tasks, the generalization prompt vector corresponding to the i-th task, and the task prompt vector corresponding to the i-th task to obtain the extended virtual image features corresponding to the previous i - 1 tasks respectively; wherein, the virtual image features corresponding to the previous i - 1 tasks respectively are determined based on the target class prompt vectors corresponding to the previous i - 1 tasks respectively;

[0092] Training the task prompt vector corresponding to the i-th task based on the extended image features corresponding to the i-th task and the extended virtual image features corresponding to the previous i - 1 tasks respectively.

[0093] In actual execution, after obtaining the class prompt vector corresponding to the historical task, the present invention proposes cross-task knowledge mining to integrate the knowledge of the current task and the past knowledge replayed by the class prompt vector corresponding to the historical task for training the task prompt vector.

[0094] For the current i-th task, cross-task knowledge mining constrains the task prompt vector t i to learn the knowledge of the initial image features of all image classes in the current task and the knowledge of the virtual image features of all image classes in the previous i - 1 tasks The virtual feature f′ of the u-th class in the v-th task u,v is replayed by its corresponding class prompt vector f′ u,v as follows:

[0095]

[0096] Having obtained the corresponding generalization prompt vector g i and the task prompt vector t i the extended image features can be obtained and the extended virtual image features of all image categories of the previous i - 1 tasks For the current i-th task, optimize the task prompt vector t i and the classifier to classify the extended image features

[0097]

[0098] wherein, is the loss function with respect to the task prompt vector t i and the classifier and is updated using the knowledge of the current task.

[0099] In addition, to learn the knowledge of past tasks, we optimize the task prompt vector t i to correctly classify the extended virtual image features from the previous i - 1 past tasks:

[0100]

[0101] wherein, is the loss function with respect to the task prompt vector t i and is updated using the knowledge of historical tasks; 1 [i>1] is the indicator function, indicating that: if i > 1, then 1 [i>1] = 1, otherwise 1 [i>1] = 0.

[0102] The loss function for cross - task knowledge mining is:

[0103]

[0104] wherein, is the total loss function with respect to the task prompt vector t i and the classifier The image classification method based on hierarchical prompts provided by the present invention can reasonably utilize the category prompt vector to utilize the knowledge of historical tasks, thereby avoiding catastrophic forgetting.

[0105] III. Generalization Prompt Vector

[0106]

[0107] In some embodiments, before step 110, the image classification method based on hierarchical prompts further includes:

[0108] ​Obtain multiple images in the i-th task, perform data enhancement on the multiple images, and obtain multiple images after data enhancement;

[0109] Based on the label information corresponding to the multiple images after data augmentation and the target image features corresponding to the multiple images after data augmentation, the generalization indicator vector corresponding to the i-th task is trained through self-supervised learning.

[0110] In actual implementation, the feature representations learned by the category hint vector and task hint vector obtained through supervised learning may not be optimal for other tasks.

[0111] Therefore, this paper proposes generalized knowledge exploration to learn generalized representations by using pre-training tasks for self-supervised learning.

[0112] Inspired by SupContrast, this paper uses category information for self-supervised training to learn representations with category discriminativeness, thereby promoting the learning of discriminative representations. Therefore, generalized knowledge exploration uses category discrimination as a pre-training task.

[0113] For N images randomly selected in the i-th task, two sets of enhancements are performed on each image to obtain 2N data-enhanced images. The initial image features of each data augmented image are

[0114] The extended image features and adapted representations corresponding to each image are and By using label information, the representations of the same image category are brought closer together, while the representations of different image categories are pulled apart:

[0115]

[0116] in, is the generalization hint vector g i The loss function is updated using self-supervised training; Indicates that it has the same n,i Positive sample images with the same label, d depicts the cosine similarity, and τ is a scalar temperature parameter.

[0117] In actual implementation, the category hint vector, task hint vector, and generalization hint vector in the above embodiment are integrated together for training to form a hierarchical hint. The total loss of the hierarchical hint is:

[0118]

[0119] The image classification method based on hierarchical prompts provided by the present invention can utilize generalization prompt vectors to learn highly generalizable knowledge and improve the generalization ability.

[0120] In some embodiments, step 110 may include:

[0121] Construct a query model based on the task prompt vector corresponding to each task and the generalization prompt vector corresponding to each task;

[0122] Based on the query model, determine the task identifier of the image to be classified.

[0123] In actual execution, during the training process, task prompt vectors and generalization prompt vectors are learned for each task, but the task identifier of each test image cannot be accessed during the inference process. Therefore, the present invention designs a novel task-aware query-key mechanism to select the task identifier of each test image.

[0124] Adopt a pre-trained image classification model, task prompt vector t i and generalization prompt vector g i as the query model q i (·). Through the query model q i (·), until the end of the training of the i-th task, extract all images ' query output

[0125] In addition, use all query outputs Execute K-means to obtain o task-aware cluster centers (keys) These keys contain the fine-grained semantic knowledge of the i-th task. Among them, o represents the number of task-aware keys, and the default setting is the number of image categories for each task. For a test sample x e lacking a task identifier, the query output q i (x e ) can be calculated with the task-aware keys K i under all task identifiers i ∈ {1,..., T}:

[0126]

[0127] where dis represents the Euclidean distance. The task identifier with the smallest distance can be selected as the task identifier of the test sample x e :

[0128]

[0129] After obtaining the task identifier id(x e ) of the test sample x e ), select the generalization prompt vector and the task prompt vector of the id(x e )-th task is implemented for inference together with the pre-trained backbone network.

[0130] In step 110, the image to be classified is input into the trained image classification model, and based on the query model, the task identifier of the image to be classified is determined.

[0131] In step 120, after determining the task identifier of the image to be classified, the current task corresponding to the task identifier is determined, and the i-th task of the m-th image category of the image x to be classified i,m .

[0132] Feature extraction is performed on the image x to be classified i,m to obtain the initial image feature of the image x to be classified i,m Thereafter, the task prompt vector t corresponding to the current task can be determined i , the generalization prompt vector g i corresponding to the current task, and the initial image feature of the image to be classified The task prompt vector t i , the generalization prompt vector g i and the initial image feature of the image to be classified are combined to obtain an extended image feature Thereafter, the target image feature is determined

[0133] Among them, the task prompt vector t i not only contains the discriminative knowledge of the current task, but also should remember the knowledge of historical tasks. The generalization prompt vector g i can contain the discriminative knowledge of the image category.

[0134] In step 130, the classifier can be used to classify the target image feature to obtain the image classification result.

[0135] Figure 2 is a schematic diagram of the algorithm flow of the image classification method based on hierarchical prompts provided by the present invention. Referring to Figure 2 , the Vision Transformers (ViT) is used as the backbone network of the image classification model.

[0136] Given the image x in the m-th category of the i-th task i,m , ViT first divides it into L fixed-size blocks and maps them into sequence features using the pre-trained embedding layer ε where, L represents the sequence length, D represents the feature dimension, and the sequence feature fi,m That is the initial image feature.

[0137] Then the sequence feature f i,m is input into the Transformer layer to obtain a representation For the sake of easy explanation, the class token and position token are omitted.

[0138] To obtain a task-specific adaptive representation, H-Prompts combines the task prompt vector and the generalization prompt vector and combines them with the sequence feature f i,m to concatenate and obtain the extended image feature

[0139]

[0140] where L g and L t represent the lengths of the generalization prompt vector and the task prompt vector respectively.

[0141] For the current i-th task, the key issue of hierarchical prompting is how to infer the task prompt vector t i and the generalization prompt vector g i . The task prompt vector t i should not only contain the discriminative knowledge of the current task but also remember the historical knowledge of past tasks, that is, it should be constructed as follows:

[0142]

[0143] where is the dataset of the current task, is the historical knowledge of all the previous i - 1 tasks:

[0144]

[0145] where k i,m represents the knowledge of the m-th class of the i-th task .

[0146] To obtain the task prompt vector in equation (15), there are two key issues:

[0147] 1) How to obtain the historical knowledge of all the previous i - 1 tasks

[0148] 2) How to transform the historical knowledge of all the previous i - 1 tasks into the task prompt vector t corresponding to the current i-th task i .

[0149] To store historical knowledge in a way that does not require storing historical samples, the present invention proposes Bayesian distribution alignment, modeling the data distribution of each image category in each task with a Bayesian neural network. For the m-th image category in the i-th task, a learnable mean vector and a diagonal covariance matrix are used to define a multivariate Gaussian distribution

[0150] The learnable parameters μ i,m and ∑ i,m are defined as the class hint vector c i,m to be optimized:

[0151]

[0152] In addition, the present invention also proposes cross-task knowledge mining, converting the class hint vector corresponding to historical task knowledge and the knowledge of the current data set into a task hint vector t i , which simultaneously memorizes the knowledge of historical tasks and the knowledge of the current task. To obtain highly generalized knowledge, we propose generalized knowledge exploration, applying self-supervised learning to train and obtain a generalized hint vector g i .

[0153] After learning the task hint vector t i and the generalized hint vector g i , the extended image features are fed into the module to generate an adapted representation

[0154]

[0155] It can be understood that the adapted representation is the target image feature.

[0156] Therefore, the class hint vector c, the task hint vector t, and the generalized hint vector g constitute a hint-based continuous learning paradigm called hierarchical hint.

[0157] The image classification method based on hierarchical hint provided by the present invention can well solve the catastrophic forgetting problem of historical task knowledge caused in continuous learning by using the class hint vector to save historical task knowledge, using the task hint vector to utilize historical task knowledge, and using the generalized hint to learn highly generalized knowledge, thereby improving the classification accuracy of the image classification model on historical tasks, and at the same time having good generalization ability, and can achieve good results on both large-scale data sets and small-scale data sets.

[0158] Reference Figure 2 , the present invention provides an image classification model based on hierarchical prompts, applying the image classification method based on hierarchical prompts as described above, including:

[0159] An embedding layer for extracting features from the image to be classified and outputting the initial image features of the image to be classified;

[0160] A Transformer layer for inputting the task prompt vector corresponding to the current task, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, and outputting the target image features;

[0161] A classifier for classifying the image to be classified based on the target image features to obtain an image classification result;

[0162] Among them, the Transformer layer is constructed based on a Bayesian distribution alignment module, a general knowledge exploration module, and a cross-task knowledge mining module;

[0163] The Bayesian distribution alignment module is used to determine the category prompt vector corresponding to each task based on the data distribution corresponding to each image category in each task;

[0164] The cross-task knowledge mining module is used to determine the task prompt vector corresponding to the current task based on the initial image features of the image to be classified and the data distribution corresponding to each image category in the historical tasks;

[0165] The general knowledge exploration module is used to train multiple images in the current task through self-supervised learning to obtain the generalization prompt vector corresponding to the current task.

[0166] In actual execution, the vision transformer first processes the image to be classified, divides it into several fixed-size blocks, and maps them into initial image features using the pre-trained embedding layer ε.

[0167] Then, the task prompt vector corresponding to the current task, the generalization prompt vector corresponding to the current task, and the initial image features are input into the Transformer layer to obtain the target image features.

[0168] The classifier (not shown in the figure) will obtain the image classification result of the image to be classified based on the target image features output by the Transformer layer.

[0169] Among them, the image classification model further includes a Bayesian distribution alignment module, a cross-task knowledge mining module, and a generalization knowledge exploration module.

[0170] The Bayesian distribution alignment module is used to adopt the category prompt vector corresponding to each task to save historical category knowledge;

[0171] The cross-task knowledge mining module is used to determine the category prompt vector corresponding to the historical task based on the data distribution corresponding to each image category in the historical task, and then determine the task prompt vector corresponding to the current task based on the category prompt vector corresponding to the historical task and the initial image features of the current task, so as to learn the knowledge of the historical task and the current task;

[0172] The generalization knowledge exploration module is used to obtain a generalization prompt vector by self-supervised learning of highly generalizable knowledge.

[0173] In the above embodiments, the training processes of the category prompt vector, the task prompt vector, and the generalization prompt vector have been described accordingly, and will not be elaborated here.

[0174] Next, the image classification device based on hierarchical prompts provided by the present invention will be described. The image classification device based on hierarchical prompts described below can be correspondingly referred to the image classification method based on hierarchical prompts described above.

[0175] The present invention also provides an image classification device based on hierarchical prompts, including:

[0176] Figure 3 is a schematic structural diagram of the image classification device based on hierarchical prompts provided by the present invention. Refer to Figure 3 , the image classification device based on hierarchical prompts provided by the present invention includes:

[0177] The first determination module 310 is used to determine the task identifier of the image to be classified;

[0178] The second determination module 320 is used to determine the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified; wherein, the task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and the virtual image features corresponding to each image category in the historical task, the virtual image features are determined based on the data distribution corresponding to each image category in the historical task, the generalization prompt vector corresponding to the current task is obtained by self-supervised learning and training of multiple images in the current task, and the initial image features of the image to be classified are obtained after feature extraction of the image to be classified;

[0179] The classification module 330 is used to classify the image to be classified based on the target image features to obtain an image classification result.

[0180] The image classification device based on hierarchical prompting provided by the present invention can well solve the catastrophic forgetting problem of historical task knowledge caused in continuous learning by using the category prompt vector to save historical task knowledge, using the task prompt vector to utilize historical task knowledge, and using generalization prompts to learn highly generalizable knowledge. Thus, it can improve the classification accuracy of the image classification model on historical tasks, and at the same time has good generalization ability, and can achieve good results on both large-scale datasets and small-scale datasets.

[0181] In some embodiments, the device further includes:

[0182] A first training module, before determining the target image feature based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image feature of the image to be classified, based on the Bayesian neural network, models the data distribution of the image categories of each task to determine the initial category prompt vector corresponding to each task;

[0183] Based on the initial category prompt vector corresponding to each task, obtain the virtual image feature corresponding to each task;

[0184] Combine the initial image feature of the sampled image in each task, the task prompt vector corresponding to each task, and the generalization prompt vector corresponding to each task to determine the extended image feature of the sampled image in each task;

[0185] Based on the extended image feature of the sampled image in each task and the virtual image feature corresponding to each task, train the initial category prompt vector corresponding to each task until the target category prompt vector corresponding to each task is obtained.

[0186] In some embodiments, the device further includes:

[0187] A second training module, before determining the target image feature based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image feature of the image to be classified, combines the initial image feature corresponding to the i-th task, the generalization prompt vector corresponding to the i-th task, and the task prompt vector corresponding to the i-th task to obtain the extended image feature corresponding to the i-th task;

[0188] Combine the virtual image features corresponding to the first i-1 tasks respectively, the generalization prompt vector corresponding to the i-th task, and the task prompt vector corresponding to the i-th task to obtain the extended virtual image features corresponding to the first i-1 tasks respectively; wherein, the virtual image features corresponding to the first i-1 tasks respectively are determined based on the target category prompt vectors corresponding to the first i-1 tasks respectively.

[0189] Train the task prompt vector corresponding to the i-th task based on the extended image feature corresponding to the i-th task and the extended virtual image features corresponding to the first i-1 tasks respectively.

[0190] In some embodiments, the apparatus further includes:

[0191] A third training module, configured to, before determining the target image feature based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image feature of the image to be classified, obtain multiple images in the i-th task, perform data augmentation on the multiple images to obtain the augmented multiple images.

[0192] Train the generalization indication vector corresponding to the i-th task through self-supervised learning based on the label information corresponding to the augmented multiple images and the target image features corresponding to the augmented multiple images respectively.

[0193] In some embodiments, the first determination module is specifically configured to:

[0194] Construct a query model based on the task prompt vector corresponding to each task and the generalization prompt vector corresponding to each task.

[0195] Determine the task identifier of the image to be classified based on the query model.

[0196] Figure 4 Illustrates a schematic physical structure diagram of an electronic device, as Figure 4 shown. The electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logical instructions in the memory 430 to execute an image classification method based on hierarchical prompts, and the method includes:

[0197] Determine the task identifier of the image to be classified.

[0198] Determine the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified; wherein, the task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and the virtual image features corresponding to each image category in the historical tasks, the virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks, the generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning, and the initial image features of the image to be classified are obtained after feature extraction of the image to be classified;

[0199] Classify the image to be classified based on the target image features to obtain an image classification result.

[0200] In addition, when the logical instructions in the above-mentioned memory 430 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0201] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the hierarchical prompt-based image classification method provided by the above-mentioned various methods. The method includes:

[0202] Determine the task identifier of the image to be classified;

[0203] Determine target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified; wherein, the task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and the virtual image features corresponding to each image category in the historical tasks, the virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks, the generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning, and the initial image features of the image to be classified are obtained after feature extraction of the image to be classified;

[0204] Classify the image to be classified based on the target image features to obtain an image classification result.

[0205] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the image classification method based on hierarchical prompts provided by the above-mentioned various methods. The method includes:

[0206] Determine the task identifier of the image to be classified;

[0207] Determine target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified; wherein, the task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and the virtual image features corresponding to each image category in the historical tasks, the virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks, the generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning, and the initial image features of the image to be classified are obtained after feature extraction of the image to be classified;

[0208] Classify the image to be classified based on the target image features to obtain an image classification result.

[0209] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0210] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0211] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image classification method based on hierarchical prompts, characterized in that, Including: Determine the task identifier of the image to be classified; Based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, determine the target image features; wherein, the task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and the virtual image features corresponding to each image category in the historical tasks, the virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks, the generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning, and the initial image features of the image to be classified are obtained after feature extraction of the image to be classified; Before determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, the method further includes: Based on a Bayesian neural network, model the data distribution of the image categories of each task, and determine the initial category prompt vector corresponding to each task; Based on the initial category prompt vector corresponding to each task, obtain the virtual image features corresponding to each task; Combine the initial image features of the sampled images in each task, the task prompt vector corresponding to each task, and the generalization prompt vector corresponding to each task, and determine the extended image features of the sampled images in each task; Based on the extended image features of the sampled images in each task and the virtual image features corresponding to each task, train the initial category prompt vector corresponding to each task until the target category prompt vector corresponding to each task is obtained; Before determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, the method further includes: Obtain multiple images in the i-th task, perform data augmentation on the multiple images to obtain multiple data-augmented images; Based on the label information corresponding to the multiple data-augmented images and the target image features respectively corresponding to the multiple data-augmented images, train the generalization prompt vector corresponding to the i-th task through self-supervised learning; The determining the task identifier of the image to be classified includes: Based on the task prompt vector corresponding to each task and the generalization prompt vector corresponding to each task, construct a query model; Based on the query model, determine the task identifier of the image to be classified; Based on the target image features, classify the image to be classified to obtain an image classification result.

2. The image classification method based on hierarchical prompting according to claim 1, wherein Before determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, the method further includes: Combine the initial image features corresponding to the i-th task, the generalization prompt vector corresponding to the i-th task, and the task prompt vector corresponding to the i-th task to obtain the extended image features corresponding to the i-th task; Combine the virtual image features corresponding to the previous i-1 tasks respectively, the generalization prompt vector corresponding to the i-th task, and the task prompt vector corresponding to the i-th task to obtain the extended virtual image features corresponding to the previous i-1 tasks respectively; wherein, the virtual image features corresponding to the previous i-1 tasks respectively are determined based on the target category prompt vectors corresponding to the previous i-1 tasks respectively; Train the task prompt vector corresponding to the i-th task based on the extended image features corresponding to the i-th task and the extended virtual image features corresponding to the previous i-1 tasks respectively.

3. An image classification model based on hierarchical prompts, characterized in that, Applied to the hierarchical prompt-based image classification method according to any one of claims 1 or 2, comprising: An embedding layer for extracting features from the image to be classified and outputting the initial image features of the image to be classified; A Transformer layer for inputting the task prompt vector corresponding to the current task, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, and outputting the target image features; A classifier for classifying the image to be classified based on the target image features to obtain an image classification result; Wherein, the Transformer layer is constructed based on a Bayesian distribution alignment module, a general knowledge exploration module, and a cross-task knowledge mining module; The Bayesian distribution alignment module is used to determine the category prompt vector corresponding to each task based on the data distribution corresponding to each image category in each task; The cross-task knowledge mining module is used to determine the task prompt vector corresponding to the current task based on the initial image features of the image to be classified and the data distribution corresponding to each image category in the historical tasks; The general knowledge exploration module is used to train multiple images in the current task through self-supervised learning to obtain the generalization prompt vector corresponding to the current task.

4. An image classification device based on hierarchical prompts, characterized in that, Comprising: A first determination module for determining the task identifier of the image to be classified; A second determination module for determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified; wherein, the task prompt vector corresponding to the current task is determined based on the initial image features of the image to be classified and the virtual image features corresponding to each image category in the historical tasks, the virtual image features are determined based on the data distribution corresponding to each image category in the historical tasks, the generalization prompt vector corresponding to the current task is obtained by training multiple images in the current task through self-supervised learning, and the initial image features of the image to be classified are obtained after feature extraction of the image to be classified; Before determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, it further includes: Based on a Bayesian neural network, model the data distribution of the image categories of each task to determine the initial category prompt vector corresponding to each task; Based on the initial category prompt vector corresponding to each task, obtain the virtual image features corresponding to each task; Combine the initial image features of the sampled images in each task, the task prompt vector corresponding to each task, and the generalization prompt vector corresponding to each task to determine the extended image features of the sampled images in each task; Based on the extended image features of the sampled images in each task and the virtual image features corresponding to each task, train the initial category prompt vector corresponding to each task until the target category prompt vector corresponding to each task is obtained; Before determining the target image features based on the task prompt vector corresponding to the current task corresponding to the task identifier, the generalization prompt vector corresponding to the current task, and the initial image features of the image to be classified, it further includes: Obtain multiple images in the i-th task, perform data augmentation on the multiple images to obtain multiple data-augmented images; Based on the label information corresponding to the multiple data-augmented images and the target image features respectively corresponding to the multiple data-augmented images, train the generalization prompt vector corresponding to the i-th task through self-supervised learning; The determination of the task identifier of the image to be classified includes: Based on the task prompt vector corresponding to each task and the generalization prompt vector corresponding to each task, construct a query model; Based on the query model, determine the task identifier of the image to be classified; A classification module, configured to classify the image to be classified based on the target image features to obtain an image classification result.

5. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the hierarchical prompt-based image classification method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the hierarchical prompt-based image classification method according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the hierarchical prompt-based image classification method according to any one of claims 1 to 3.

Images