Small-sample fine-grained image classification method based on retrieval and multi-modal large model

Abstract

The invention relates to a small-sample fine-grained image classification method based on retrieval and a multi-modal large model, and belongs to the field of fine-grained image classification, and the method comprises the steps: obtaining all categories of fine-grained images, and setting a category prompt template in combination with the specific features of each category; arranging the fine-grained image data set; performing vectorization model coding on the picture and the category information to obtain an image vector and a text vector; performing retrieval recall on the candidate information; and obtaining a final fine-grained image classification result through a multi-modal large model by using a to-be-classified label-free image sample and a retrieved and recalled image set, text information set and category information set. According to the method provided by the invention, the cost of data set collection can be reduced, and the flexibility and expandability of a fine-grained image classification task are improved.

Description

Technical Field

[0001] The present invention belongs to the field of fine-grained image classification, and in particular relates to a few-sample fine-grained image classification method based on retrieval and multimodal large models. Background Art

[0002] Fine-grained image classification is a basic task in the field of computer vision, which aims to finely divide different subcategories under the same category. Fine-grained image classification faces great difficulties due to the subtle differences between different subcategories and the significant differences between different samples of the same category due to factors such as image angle and posture.

[0003] Traditional fine-grained image classification algorithms are usually based on pre-collected image datasets, where each subclass includes multiple samples, and recognition is achieved by training a neural network model with a fixed classification head.

[0004] The traditional fine-grained image classification model has the following problems:

[0005] (1) Traditional fine-grained image classification requires a large number of training samples based on a dataset with a fixed number of categories, and has poor flexibility and scalability.

[0006] (2) It can only classify subcategories that exist in the training set and cannot incrementally identify new categories that do not exist in the dataset. Summary of the invention

[0007] In view of the above deficiencies in the prior art, the purpose of the present invention is to provide a few-sample fine-grained image classification method based on retrieval and multimodal large models, so as to reduce the cost of data set collection and improve the flexibility and scalability of fine-grained image classification tasks.

[0008] The present invention proposes a few-sample fine-grained image classification method based on retrieval and multimodal large model, comprising:

[0009] S1, constructs a multimodal vector database based on fine-grained images, including:

[0010] Get all categories of fine-grained images and set category prompt templates based on the specific features of each category;

[0011] Organize the fine-grained image dataset D, each sample d includes picture d image and its corresponding category information d class ;

[0012] Combined with the category prompt template, the pictures are d image and category information class Perform vectorized model encoding to obtain image vector and text vector And store in multiple modes

[0013] Vector database L, construct image vector set L i and text vector set L t ;

[0014] S2, retrieve and recall the candidate information, including: encoding the unlabeled images to be classified based on the multimodal vectorization model M, retrieving N candidate image vectors and corresponding N text vectors from the multimodal vector database L as candidate image vectors and text vectors, screening out the recalled image vectors and recalled text vectors from the candidate image vectors and text vectors according to a preset threshold, and obtaining the corresponding recalled image set R i , text information set R t and the category information set R c ;

[0015] S3, the unlabeled image sample s to be classified and its retrieved image set R i , text information set R t and the category information set R c , after the multimodal large model, the final fine-grained image classification result is obtained.

[0016] Furthermore, in S1, a multimodal vectorization model M that has been aligned with multimodal corpora is used, and the text encoder part of the multimodal vectorization model M is denoted as M TE , the image encoder part of the multimodal vectorization model M is denoted as M IE .

[0017] Furthermore, the image d image and category information class Perform vectorized model encoding, including:

[0018] (1) For sample d∈D, image d image After normalization, it passes through the image encoder M IE Convert to vector

[0019] (2) Randomly select one of the category prompt templates and combine it with the category information d class Constructing text information text , through the text encoder M TE Convert to vector

[0020] (3) Perform the above operations (1) to (2) on all samples d in the sample set D, and convert the corresponding and Store in multimodal vector database L, build image vector set L iand text vector set L t .

[0021] Furthermore, vector deduplication is performed to delete repeated image and text vectors in the vector database L.

[0022] Further, in S2, the unlabeled image to be classified is encoded based on the multimodal vectorization model M, including: for the unlabeled image sample to be classified The image encoder part M of the multimodal vectorization model M IE Get Vector

[0023] Furthermore, first, for and image vector set L i , vectorized in the following way Recall the most similar N image vectors and N text vectors as candidate image vectors and text vectors;

[0024] Among them, the most similar N image vectors for:

[0025]

[0026] Where q represents the image vector set L i Any vector in q T represents the transposed vector of the vector;

[0027] The N most similar text vectors for:

[0028]

[0029] Where p represents the image vector set L i Any vector in p T represents the transposed vector of the vector;

[0030] Then perform deduplication to remove repeated samples in the candidate image vector and text vector;

[0031] Finally, the threshold H is set for the candidate image vector and text vector i and H t , the cosine similarity is greater than or equal to the threshold H i The image vector with cosine similarity greater than or equal to the threshold H is determined as the recall image vector; t The text vector of is determined as the recalled text vector.

[0032] Further, obtain the image set R corresponding to the recalled image vector i and text information set R t for:

[0033]

[0034]

[0035] For the recalled pictures;

[0036] For the text of the recall;

[0037] u is the number of images and texts recalled.

[0038] Further, obtain the category information set R corresponding to the recalled image vector c include:

[0039] For the sample d in the recall image set image ∈R i , obtain the corresponding category information, and construct R c for:

[0040]

[0041] in, is the category information corresponding to the recalled image.

[0042] Furthermore, in S3, the output content of the multimodal large model is parsed, redundant explanation content is eliminated, and fine-grained category information corresponding to the image is obtained.

[0043] Furthermore, the multimodal large model uses full parameters or Lora to fine-tune the vertical field of the data.

[0044] The beneficial effects of the present invention are as follows:

[0045] 1. The present invention uses a dataset with a small sample size and constructs a vector database based on a multimodal vectorization model. This method is a general fine-grained image classification methodology with wide versatility.

[0046] 2. For a candidate input image sample, the present invention retrieves similar TopN images and text descriptions from a vector database after vectorization.

[0047] 3. The present invention is based on the generalization ability of the multimodal large model and obtains the classification result by referring to the reference information obtained by retrieval.

[0048] 4. The present invention can reconstruct fine-grained image classification patterns, transforming classification tasks into retrieval tasks, requiring only a small number of samples, making data collection convenient and cost-effective. The present invention changes the status quo of traditional algorithms that have fixed categories and require a large number of training samples.

[0049] 5. The present invention greatly improves the convenience of adding new categories. It only needs to add the corresponding information of the new category to the vector database, which does not affect the original algorithm effect. That is, the new category can be easily added to the vector database without changing the recognition effect of the original algorithm, thereby greatly reducing the cost of data set collection and improving the flexibility and scalability of fine-grained image classification tasks.

[0050] 6. The present invention has low reliance on model training, and open source multimodal vectorization models and multimodal large models can achieve certain effects. The accuracy can also be further improved through further fine-tuning. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are only used to illustrate specific embodiments and are not considered to limit the present invention. In the entire drawings, the same reference symbols represent the same components. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings.

[0052] Figure 1 This is a flow chart of a method for fine-grained image classification with a small number of samples based on retrieval and a multimodal large model according to an embodiment of the present invention;

[0053] Figure 2 A flowchart of constructing a multimodal vector database according to an embodiment of the present invention;

[0054] Figure 3 A flowchart of candidate information retrieval and recall according to an embodiment of the present invention;

[0055] Figure 4 The figure is a flow chart of the multi-modal large model refinement according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work should fall within the scope of protection of the present invention.

[0057] Furthermore, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily obscuring the concepts disclosed in the present invention.

[0058] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. The terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0059] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of methods and systems consistent with some aspects of the present invention as detailed in the appended claims.

[0060] The present invention proposes a few-sample fine-grained image classification method based on retrieval and multimodal large model, which involves retrieval enhancement technology, multimodal large model technology, few-sample classification technology and fine-grained image classification technology.

[0061] The technical terms involved in the present invention are explained below:

[0062] Retrieval Augmented Generation (RAG) is a technique that assists large language models in generating answers by retrieving information from data sources.

[0063] The Multimodal Large Language Model (MLLM) receives multimodal information such as images and audio, and uses a large language model as the brain to generate output results.

[0064] Few-shot classification is a subfield of general classification tasks, which aims to develop and implement classification models when the number of samples in each category is small.

[0065] Fine-grained Visual Classification (FGVC) is a subtask of the general image classification task, which aims to finely distinguish different subcategories belonging to the same category.

[0066] The few-sample fine-grained image classification method based on retrieval and multimodal large model of the present invention mainly includes three processes: multimodal vector database construction, candidate information retrieval and recall, and multimodal large model refinement.

[0067] like Figure 1 As shown, the present invention proposes a few-sample fine-grained image classification method based on retrieval and multimodal large model, comprising the following steps:

[0068] S1, build a multimodal vector database based on fine-grained images.

[0069] Combine the following Figure 2 The process of constructing a multimodal vector database is explained.

[0070] 1) Adopt the multimodal vectorization model M after multimodal corpus alignment, and denote the text encoder part of the multimodal vectorization model M as M TE , the image encoder part of the multimodal vectorization model M is denoted as M IE .

[0071] In an embodiment of the present invention, the multimodal vectorization model M can adopt a CLIP (Contrastive Language Image Pre Training, using text supervision signals to train a visual model with strong migration capabilities) model or a BLIP (Bootstrapping Language Image Pre-training, visual language pre-training model) model, etc.

[0072] 2) Obtain all categories of fine-grained images and set category prompt templates based on the specific features of each category.

[0073] Specifically, the set of all categories of fine-grained images is recorded as C. The prompt template is set in combination with the specific features of each category, for example, [a photo of class {c}, the main features of {c} are...; this is a photo of class {c}, the main features of {c} are...], etc., where c∈C in the template.

[0074] 3) Organize the fine-grained image dataset D, each sample d includes picture d image and its corresponding category information d class .

[0075] 4) Combine the category prompt template and sort the pictures d image and category information class Perform vectorized model encoding to obtain image vector and text vector And store it in the multimodal vector database L to construct the image vector set L i and text vector set L t .

[0076] Specifically, for image d image and category information class Perform vectorized model encoding, including:

[0077] (1) For a sample d∈D, image d image After normalization, it passes through the image encoder M IE Convert to vector

[0078] (2) Randomly select one from the category prompt templates and combine it with the category information d class Constructing text information text , through the text encoder M TE Convert to vector

[0079] (3) Perform the above operations (1) to (2) on all samples d in the sample set D, and convert the corresponding and Store in multimodal vector database L, build image vector set L i and text vector set L t .

[0080] In an embodiment of the present invention, the multimodal vector database L is, for example, a Milvus vector database, a Faiss vector database, etc.

[0081] 5) Perform vector deduplication and delete repeated image and text vectors in the vector database L.

[0082] It should be noted that the multimodal vectorization model can be fine-tuned for specific data vertical fields using related technologies such as CoOp and Adapter.

[0083] S2, retrieve and recall the candidate information, including: encoding the unlabeled images to be classified based on the multimodal vectorization model M, retrieving N candidate image vectors and corresponding N text vectors from the multimodal vector database L as candidate image vectors and text vectors, screening out the recalled image vectors and recalled text vectors from the candidate image vectors and text vectors according to a preset threshold, and obtaining the corresponding recalled image set Ri , text information set R t and the category information set R c .

[0084] Combine the following Figure 3 The process of retrieving and recalling candidate information is described.

[0085] 1) Encode a certain unlabeled image to be classified based on the multimodal vectorization model M, including: The image encoder part M of the multimodal vectorization model M IE Get Vector

[0086] 2) Based on the above multimodal vector database L, Recall the most similar N image vectors and N text vectors. The recall process is based on the cosine similarity algorithm.

[0087] Specifically, for and image vector set L i , vectorized in the following way Recall the most similar N image vectors and N text vectors as candidate image vectors and text vectors.

[0088] Among them, the most similar N image vectors for:

[0089]

[0090] Where q represents the image vector set L i Any vector in q T represents the transposed vector of the vector;

[0091] The N most similar text vectors for:

[0092]

[0093] Where p represents the image vector set L i Any vector in p T represents the transposed vector of the vector;

[0094] 3) Perform deduplication to delete repeated samples in the candidate image vector and text vector;

[0095] 4) Set threshold H for candidate image vectors and text vectors i and H t , the cosine similarity is greater than or equal to the threshold H i The image vector with cosine similarity greater than or equal to the threshold H is determined as the recall image vector;t The text vector of is determined as the recalled text vector.

[0096] 5) Based on the multimodal vector database L, obtain the image set R corresponding to the recalled image vector i and text information set R t for:

[0097]

[0098]

[0099] For the recalled pictures; is the recalled text; u is the number of recalled images and texts.

[0100] Get the category information set R corresponding to the recalled image vector c Included: Sample d in the recall image set image ∈R i , obtain the corresponding category information, and construct R c for:

[0101]

[0102] in, is the category information corresponding to the recalled image.

[0103] S3, the unlabeled image sample s to be classified and its retrieved image set R i , text information set R t and the category information set R c , after the multimodal large model, the final fine-grained image classification result is obtained.

[0104] Combine the following Figure 4 The process of fine sorting of multimodal large models is explained.

[0105] 1) For a certain unlabeled image sample s to be classified and its retrieval recall R i , R c and R t , build prompts for multimodal large models.

[0106] 2) The reference prompt case is as follows:

[0107] For a powerful fine-grained image classifier, given an input image sample and some reference information, it is necessary to carefully think about obtaining the category corresponding to the input image sample.

[0108] Here are some similar images and their corresponding categories:

[0109] 1. The corresponding categories are

[0110] 2. The corresponding categories are

[0111] 3.……

[0112] Below are the possible categories and detailed category description information

[0113] 1.

[0114] 2.

[0115] 3.……

[0116] The input image sample is s. Based on the above reference information, its corresponding sample is given and the reason is explained.

[0117] 3) Analyze the output content of the multimodal large model, remove redundant explanations, and obtain fine-grained category information corresponding to the image.

[0118] In an embodiment of the present invention, the multimodal large model can be fine-tuned for specific data vertical fields using full parameters or Lora and other technologies.

[0119] The present invention's method for fine-grained image classification with a small number of samples based on retrieval and a large multimodal model converts the classification task into a retrieval task. Each category requires only a small number of samples, and the image and text information are encoded through a vectorized model and stored in a vector database. In the inference and prediction stage, after the image to be classified is encoded, N candidate similar images and their corresponding categories are retrieved from the vector database, and the final classification result is obtained through a large multimodal model. Through the above method, each subclass requires only a small number of samples, and new categories can be easily added to the vector database, which greatly reduces the cost of data set collection and improves the flexibility and scalability of fine-grained image classification tasks.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the above embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A few-sample fine-grained image classification method based on retrieval and multimodal large model, characterized in that: include: S1, constructs a multimodal vector database based on fine-grained images, including: Get all categories of fine-grained images and set category prompt templates based on the specific features of each category; Organize the fine-grained image dataset D, each sample d includes picture d image and its corresponding category information d class ; Combined with the category prompt template, the pictures are d image and category information class Perform vectorized model encoding to obtain image vector and text vector And store it in the multimodal vector database L to construct the image vector set L i and text vector set L t ; S2, retrieve and recall the candidate information, including: encoding the unlabeled images to be classified based on the multimodal vectorization model M, retrieving N candidate image vectors and corresponding N text vectors from the multimodal vector database L as candidate image vectors and text vectors, screening out the recalled image vectors and recalled text vectors from the candidate image vectors and text vectors according to a preset threshold, and obtaining the corresponding recalled image set R i , text information set R t and the category information set R c ; S3, the unlabeled image sample s to be classified and its retrieved image set R i , text information set R t and the category information set R c , after the multimodal large model, the final fine-grained image classification result is obtained.

2. According to claim 1, a method for fine-grained image classification with few samples based on retrieval and multimodal large model, characterized in that: In S1, a multimodal vectorization model M that has been aligned with multimodal corpora is used, and the text encoder part of the multimodal vectorization model M is denoted as M TE , the image encoder part of the multimodal vectorization model M is denoted as M IE .

3. The method for fine-grained image classification with few samples based on retrieval and multimodal large model according to claim 2, characterized in that: The pictures d image and category information class Perform vectorized model encoding, including: (1) For sample d∈D, image d image After normalization, it passes through the image encoder M IE Convert to vector (2) Randomly select one of the category prompt templates and combine it with the category information d class Constructing text information text , through the text encoder M TE Convert to vector (3) Perform the above operations (1) to (2) on all samples d in the sample set D, and convert the corresponding and Store in multimodal vector database L and construct image vector set L i and text vector set L t .

4. The method for fine-grained image classification with few samples based on retrieval and multimodal large model according to claim 1 or 3, characterized in that: Perform vector deduplication to delete repeated image and text vectors in the vector database L.

5. The method for fine-grained image classification with few samples based on retrieval and multimodal large model according to claim 1, characterized in that: In S2, the unlabeled image to be classified is encoded based on the multimodal vectorization model M, including: for the unlabeled image sample to be classified The image encoder part M of the multimodal vectorization model M IE Get Vector 6. The method for fine-grained image classification with few samples based on retrieval and multimodal large model according to claim 5, characterized in that: First, for and image vector set L i , vectorized in the following way Recall the most similar N image vectors and N text vectors as candidate image vectors and text vectors; Among them, the most similar N image vectors for: Where q represents the image vector set L i Any vector in q T represents the transposed vector of the vector; The N most similar text vectors for: Where p represents the image vector set L i Any vector in p T represents the transposed vector of the vector; Then perform deduplication to remove repeated samples in the candidate image vector and text vector; Finally, the threshold H is set for the candidate image vector and text vector i and H t , the cosine similarity is greater than or equal to the threshold H i The image vector with cosine similarity greater than or equal to the threshold H is determined as the recall image vector; t The text vector of is determined as the recalled text vector.

7. The method for fine-grained image classification with few samples based on retrieval and multimodal large model according to claim 6, characterized in that: Get the image set R corresponding to the recalled image vector i and text information set R t for: For the recalled pictures; For the text of the recall; u is the number of images and texts recalled.

8. The method for fine-grained image classification with few samples based on retrieval and multimodal large model according to claim 7, characterized in that: Get the category information set R corresponding to the recalled image vector c include: For the sample d in the recall image set image ∈R i , obtain the corresponding category information, and construct R c for: in, is the category information corresponding to the recalled image.

9. The method for fine-grained image classification with few samples based on retrieval and multimodal large model according to claim 1, characterized in that: In S3, the output content of the multimodal large model is parsed, redundant explanation content is eliminated, and fine-grained category information corresponding to the image is obtained.

10. The method for fine-grained image classification with few samples based on retrieval and multimodal large model according to claim 1, characterized in that: The multimodal large model uses full parameters or Lora to fine-tune the vertical field of the data.

Images