Multi-label image classification method for multi-layer prompt information

Abstract

The invention discloses a multi-label image classification method for multi-layer prompt information. The method comprises the following steps: acquiring a test image and a label set corresponding to a training set sample; using the topic model to obtain topic distribution of all samples in the training set; learning a group of prompt blocks representing theme label distribution information in the model, injecting the prompt blocks Prompt Token into the intermediate features, and inputting the prompt blocks Prompt Token into a selected prompt layer Prompt Block; and finally, judging which category the sample belongs to according to an output result of the Vision Transform. Compared with the prior art, the method has the advantages that the accuracy of a multi-label classification task is improved, an attention mechanism is helped to pay attention to a smaller area which is more critical for distinguishing which category an object belongs to, label missing and errors can be made up to a certain degree, the robustness of a model is improved, and meanwhile the labor cost is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and natural language processing, and in particular relates to a multi-label image classification method with multi-layer prompt information. Background Art

[0002] Topic modeling is a core task in the field of text analysis. Its goal is to use machine learning algorithms to allow machines to deeply understand the potential topic structure in a document collection, and then use this topic information to abstract and summarize text data. Topic modeling can perform semantic relationship reasoning and data mining on documents containing a large number of words, thereby revealing hidden topic distribution in the document.

[0003] A topic model is a statistical model used to cluster the implicit semantic structure in a text. It uses unsupervised learning to find potential topics in a document and analyze the correlation between these topics and words. Topic models are mainly used in natural language processing (NLP) and text mining, such as collecting, classifying, and reducing the dimension of text by topic. The working principle of the topic model is based on a probabilistic model, where each topic is regarded as a probability distribution of words in a vocabulary. Each word in an article is obtained through the process of "selecting a topic with a certain probability, and selecting a word from this topic with a certain probability".

[0004] Multi-label classification (MLC) is a key task in the field of computer vision. This task requires machine learning models to fully mine the characteristic information of samples and perform effective relationship reasoning between multiple labels. Through multi-label classification, machine learning models can identify multiple attributes or categories of samples in complex data sets. The research on multi-label classification not only helps to promote the development of machine learning technology, but also has attracted widespread attention in the fields of pattern recognition and information retrieval. This task has a wide range of value in practical applications. For example, in the fields of network content review, image annotation, text classification, etc., it can help machines better understand the multi-dimensional characteristics of samples, improve the accuracy of data processing and analysis, and thus enhance intelligent processing capabilities.

[0005] In multi-label classification tasks, the correctness and completeness of sample labels are one of the key factors affecting model performance. Inaccurate or missing sample label sets will weaken the predictive ability of deep learning models, thereby affecting their performance in practical applications. With the advent of the big data era, the surge in data volume has brought about an explosive growth in the number of samples and the types of labels, which has not only greatly increased the economic cost of manually annotating sample label sets, but also increased the complexity of the labeling work. In the process of manual labeling, due to fatigue, subjective judgment differences, or lack of professional knowledge, incomplete label sets and incorrect labeling are common. The existence of these problems has undoubtedly greatly increased the difficulty of training high-precision multi-label classification deep learning models, and also put forward higher requirements on the generalization ability and reliability of the model. Summary of the invention

[0006] In view of the above, an object of the present invention is to provide a multi-label image classification method with multi-layer prompt information.

[0007] The present invention first uses topic modeling to cluster the topic distribution of samples in a multi-label dataset; then injects the topic distribution into the sample image training model, and uses the cross-attention mechanism to influence the feature extraction process, making it easier for the model to focus on some significant image features corresponding to a certain topic.

[0008] To achieve the above-mentioned purpose, the technical solution steps provided by the present invention are as follows:

[0009] Step 1, obtain the label set corresponding to the test image and training set samples;

[0010] Step 2: Set the number of topics and use the topic model to obtain the topic distribution of all samples in the training set;

[0011] Step 3: Learn a set of prompt blocks representing the topic label distribution information in the Vision Transformer (ViT) model, inject the prompt blocks into the intermediate features, and input them into the selected prompt layer;

[0012] Step 4: Finally, the output results of the ViT model are used to determine which categories the sample belongs to.

[0013] Further:

[0014] In step 1, the image is a color image containing multiple entities in a natural scene, each image corresponds to a label set, which contains at least one label, and the corresponding label set of the samples in the training set is known, while the label set of the samples in the test set is unknown.

[0015] The label set refers to the set of natural language names of the class to which each sample in the training set belongs.

[0016] In step 2, the topic model refers to a model that discovers abstract topics contained in multiple documents. Input the label set corresponding to the training set samples into the topic model, and set the number of topics. Then, the topic model is used to obtain the topic distribution of the training set samples.

[0017] Topic distribution refers to the probability that a sample belongs to several different topics. Usually, the topic distribution of a sample is a vector with a length equal to the set number of topics, and the sum of all elements in the vector is 1.

[0018] The training set samples refer to the image collection, and the topic distribution corresponding to the training set samples refers to the probability value of each training set sample corresponding to each topic.

[0019] In step 3, an auxiliary learning strategy is used to perform auxiliary task label topic classification while carrying out the main task multi-label classification. The specific method is to first select the prompt layer (Prompt Block) for inserting prompts and the number of topics corresponding to the topic information inserted in each prompt layer; for each prompt layer, the corresponding prompt block is connected to the image features output by the previous layer, input into the prompt layer, and the image features and the prompt layer absorb information from each other through the cross-attention mechanism. Finally, the output of the prompt layer is divided into new image features and prompt blocks. The obtained image features continue to enter the next layer of blocks, and the prompt blocks enter the topic classification learning task. Specifically:

[0020] In the ViT model, select some layers of blocks in the middle of the ViT model as prompt blocks, and set the corresponding number of topics;

[0021] For each prompt layer:

[0022] First, initialize a Tensor of size (1, Embed_size) as a prompt block, then connect the prompt block and the image features output by the previous layer of Block, and input them into the prompt layer together;

[0023] Then, the output result of the prompt layer is split into two distributions: the prompt block and the image feature. The prompt block enters the auxiliary task, while the image feature enters the next layer of Block to continue the main task. If the prompt layer is the last layer of Block, the prompt block enters the auxiliary task, while the image feature enters the classifier to perform multi-label classification tasks.

[0024] Finally, the loss of the auxiliary task and the main task is weightedly summed to obtain the loss of the model.

[0025] The auxiliary task topic label classification task uses Cross-Entropy Loss as the loss function, and each prompt layer will generate a corresponding topic label loss A i ; The main task multi-label classification task uses AsymmetricLoss as the loss function, the resulting loss is recorded as B, and the overall loss function of the model is n represents the number of prompt layers.

[0026] The topic label classification task of the auxiliary task uses Cross-Entropy Loss as the loss function. Each prompt layer will generate a corresponding topic label loss A i , the main task multi-label classification task uses AsymmetricLoss as the loss function, the resulting loss is recorded as B, and the overall loss function of the model is i represents the sequence number of the prompt layer, n represents the number of prompt layers, α and βi are artificially set hyperparameters, which are used to weighted sum the losses of the main task and auxiliary tasks to obtain the final total model loss, and βi corresponds to the weight of the topic label loss generated by the i-th prompt layer.

[0027] First, we need to select the prompt block for inserting prompts and the number of topics corresponding to the topic information inserted in each prompt block. The prompt blocks near the bottom correspond to fewer topics, and the number of corresponding prompts gradually increases as the prompt blocks are closer to the top, so that the ViT model can learn topic information with gradually finer granularity in a hierarchical manner.

[0028] The output of the Vision Transformer model is used to determine which category the sample belongs to. The output of the Vision Transformer model is activated by the Sigmoid function, and then a threshold is set, usually 0.5. The label corresponding to the subscript corresponding to the value greater than this threshold is considered to be the label contained in this image.

[0029] In summary, the present invention has more advantages in providing topic information, enriching the hierarchy of label sets and guiding machine attention in the multi-label classification task of the Vision Transformer model, thereby improving the accuracy of the task.

[0030] Compared with the prior art, the multi-label image classification method of the present invention with multi-layer prompt information has the following beneficial effects:

[0031] First, the present invention introduces the topic distribution generated by the topic model and the prediction results of the image classification model in the Block of the Vision Transformer model as the coarse category information of the image, making full use of the information between the sample label sets, providing more information, and improving the accuracy of the multi-label classification task;

[0032] Second, compared with directly using the Vision Transformer model for multi-label classification tasks, the multi-label image classification method with multi-layer prompt information will inject topic information with gradually increasing number of topics on multiple prompt layers, allowing the ViT model to learn topic information with gradually finer granularity in a hierarchical manner, which helps the attention mechanism to focus on smaller areas that are more critical for distinguishing which category an object belongs to.

[0033] Third, as the amount of data increases, the label space becomes larger, and manual labeling becomes expensive and has a higher error rate, injecting topic information into the Block of the Vision Transformer model can compensate for the missing and incorrect labels to a certain extent, improve the robustness of the model, and reduce labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. In the drawings:

[0035] Figure 1 It is a flow chart of the present invention.

[0036] Figure 2 This is a schematic diagram of the effect of using the topic model to generate the topic probability distribution of the training set sample images in the present invention.

[0037] Figure 3 It is a schematic diagram of the effect of how to inject the topic distribution information of the sample into the prompt layer in the present invention. DETAILED DESCRIPTION

[0038] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0039] The inventive concept of the present invention is: in response to the problems of increased label space, increased manual annotation costs and increased error rate caused by the increase in the current data volume, a multi-label image classification method with multi-layer prompt information is proposed, which is applied to correctly output the entity names contained in the picture in the picture scene where there are multiple entities, thereby further improving the accuracy of human-computer interaction and improving the machine's ability to understand pictures.

[0040] In this embodiment, the sample image training model adopts a vision transformer (ViT) model.

[0041] Figure 1 is a flow chart of a multi-label image classification method for multi-layer prompt information provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a multi-label image classification method with multi-layer prompt information, comprising the following steps:

[0042] Step 1: Get the label sets corresponding to the test images and training set samples.

[0043] The image is a color image containing at least one entity in a natural scene. The label set refers to the set of natural language names of the class to which each sample in the training set belongs. Each training set sample corresponds to at least one label, representing the entity in the sample image.

[0044] In this embodiment, reference Figure 2 The input part on the left side of the "topic model" contains sample images and corresponding sample label sets. Usually one image corresponds to multiple labels.

[0045] Step 2: Input the label set of all samples in the training set into the topic model to obtain the topic distribution of all samples in the training set.

[0046] In this embodiment, if Figure 2 As shown in , there are multiple sample images and the label set corresponding to each sample image on the left side of the topic model. The label set corresponding to each sample image is regarded as a document. Multiple documents are input into the topic model, and the topic model generates corresponding results, such as Figure 2 The right side of the topic model shows the distribution of the probability of each image for each topic, and the sum of the probabilities of the same image for each topic is 1.

[0047] In the prior art, topic model algorithms are relatively mature, such as LDA model, BTM (Biterm Topic Model) model and BERTopic model, etc. These models are all suitable for the topic clustering of the present invention.

[0048] Step 3: Learn a set of prompt blocks representing topic tag distribution information in the ViT model, inject the prompt blocks into the intermediate features, and input them into the selected prompt layer.

[0049] In the embodiment, Figure 3 As shown, firstly, the prompt layer and the corresponding number of topics are selected. In this embodiment, the middle first layer Block and the last layer Block are selected as prompt layers (in this example, two Blocks are selected as prompt layers), and the corresponding number of topics is set. The number of topics corresponding to the first prompt layer is 2, and the number of topics corresponding to the second prompt layer is 8.

[0050] For the first prompt layer:

[0051] First, initialize a Tensor of size (1, Embed_size) as a prompt block, then connect the prompt block and the image features output by the previous layer of Block, and input them into the prompt layer together;

[0052] Then, the output results of the prompt layer are split into two distributions: the prompt block and the image feature; the prompt block enters the auxiliary task of topic classification, while the image feature enters the next layer of Block to continue the main task of multi-label classification.

[0053] For the last layer of Block: perform similar operations, and finally add the weighted sum of the losses of the auxiliary task and the main task to get the loss of the model.

[0054] Step 4: Finally, the output results of the ViT model are used to determine which categories the sample belongs to.

[0055] Specifically, the ViT model will eventually output a vector of size (1, Class_num) for an image, where Class_num is a fixed value, generally determined by the fixed attributes of the dataset, representing the size of the label space in the dataset.

[0056] Subsequently, this vector is fed into the activation function Sigmoid and a threshold is set (usually 0.5);

[0057] Finally, the subscripts of each element in the activated vector that is greater than the threshold are output. The labels corresponding to these subscripts are the labels that the ViT model believes the image sample corresponds to.

[0058] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A multi-label image classification method with multi-layer prompt information, characterized in that: The following steps are involved: Step 1, obtain test set samples, obtain training set samples and their corresponding label sets; the samples are images; Step 2: Input the label set corresponding to the training set samples into the topic model, set the number of topics, and use the topic model to obtain the topic distribution of all samples in the training set; Step 3: In the Vision Transformer (ViT) model, learn a set of prompt tokens representing the distribution information of topic tags, inject the prompt blocks into the intermediate features, and input them into the selected prompt layer Prompt Block; Step 4: Use the output of the ViT model to determine which categories the sample belongs to; In step 1, the image is a color image containing at least one entity in a natural scene; the label set refers to the set of natural language names of the class to which each sample in the training set belongs; the number of labels corresponding to each training set sample is at least one, representing the entity in the sample; each image corresponds to a label set, and the label set contains no less than one label; The corresponding label set of the training set samples is known; the label set of the test set samples is unknown; In step 2, for the input of the topic model, the label set corresponding to each sample is regarded as a document, and these documents are input into the topic model; the topic distribution output by the topic model refers to the probability that a sample belongs to several different topics; the topic distribution of a sample is a vector with a length equal to the number of topics set, and the sum of all elements in the vector is 1; In the ViT model in step 3, an auxiliary learning strategy is used to perform auxiliary tasks while carrying out the main task. The main task is a multi-label classification task, and the auxiliary task is a topic label classification task. The steps include: First, select the prompt layer Prompt Block where the prompt block is inserted and the number of topics corresponding to the topic information inserted in each prompt layer; Then, for each hint layer, the corresponding hint block is connected to the image features output by the previous layer; then they are input into the hint layer together, and the image features and hint layer absorb information from each other through the cross attention mechanism; Finally, the output of the prompt layer is split into new image features and prompt blocks, where the image features continue to enter the next layer of blocks and the prompt blocks enter the topic label classification task.

2. The multi-label image classification method with multi-layer prompt information according to claim 1, characterized in that: In step 3, in the ViT model, select some layers of blocks in the middle of the ViT model as prompt blocks, and set the corresponding number of topics; For each prompt layer: First, initialize a Tensor of size (1, Embed_size) as a prompt block, then connect the prompt block and the image features output by the previous layer of Block, and input them into the prompt layer together; Then, the output result of the prompt layer is split into two distributions: the prompt block and the image feature. The prompt block enters the auxiliary task, while the image feature enters the next layer of Block to continue the main task. If the prompt layer is the last layer of Block, the prompt block enters the auxiliary task, while the image feature enters the classifier to perform multi-label classification tasks. Finally, the loss of the auxiliary task and the main task is weightedly summed to obtain the loss of the model.

3. The multi-label image classification method of multi-layer prompt information according to claim 1, characterized in that: In step 2, the topic model is an LDA model, a BTM model, or a BERTopic model.

4. The multi-label image classification method of multi-layer prompt information according to claim 2, characterized in that: In step 3, the topic label classification task of the auxiliary task uses Cross-Entropy Loss as the loss function, and each prompt layer will generate a corresponding topic label loss A i , the main task multi-label classification task uses Asymmetric Loss as the loss function, the resulting loss is recorded as B, and the overall loss function of the model is i represents the sequence number of the prompt layer, n represents the number of prompt layers, α and βi are artificially set hyperparameters, which are used to weighted sum the losses of the main task and auxiliary tasks to obtain the final total model loss, and βi corresponds to the weight of the topic label loss generated by the i-th prompt layer.

5. The multi-label image classification method of multi-layer prompt information according to claim 1, characterized in that: The feature is that in step 4, the output of the ViT model is sent to the activation function; if the label probability value of the output result of the activation function is greater than the preset threshold, it is considered that the sample contains this category.

6. The multi-label image classification method of multi-layer prompt information according to claim 1, characterized in that: The characteristic is that in step 4, the method of judging which categories the sample belongs to by the output result of the ViT model is: First, the ViT model finally outputs a vector of size (1, Class_num) for an image. Class_num is a fixed value determined by the fixed attributes of the dataset and represents the size of the label space in the dataset. Subsequently, this vector is fed into the activation function Sigmoid and a threshold is set; Finally, the subscripts of each element in the activated vector that is greater than the threshold are output. The labels corresponding to these subscripts are the labels that the ViT model believes the image sample corresponds to.

Images