Multi-modal large model small sample classification method based on knowledge enhancement

Abstract

The invention discloses a multi-modal large model small sample classification method based on knowledge enhancement, and relates to the field of artificial intelligence and computer vision, and the method comprises the steps: collecting a multi-scene image data set, and constructing base class and new class data into a small sample instruction fine tuning set and a test set which accord with a meta-learning normal form; enhancing the picture and text labels of the fine tuning set, and inputting the multi-modal large model for fine tuning; in the reasoning stage, historical dialogues and instructions in a test set are randomly selected and input into the fine-tuned model together to obtain an initial classification result; then, generating a plurality of attribute descriptions for each image, calculating the similarity of the descriptions among the images, and sorting the descriptions to screen reliable candidate answers; finally, the reliability of the initial answer is judged, and if the initial answer is not reliable, a new candidate answer construction instruction is used for reasoning again. According to the method, the small sample classification capability of the multi-modal large model is improved through a knowledge enhancement method based on two stages of fine adjustment and reasoning, and the method is suitable for general and fine-grained classification tasks.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer vision, and in particular to a multi-modal large model small sample classification method based on knowledge enhancement. Background Art

[0002] With the rapid development of deep learning technology and large-scale datasets, the field of machine vision has made remarkable progress. However, in practical applications, the scarcity of labeled data is still a key bottleneck restricting the performance improvement of deep learning models. In contrast, humans have the ability to quickly learn and recognize new categories from a small number of examples, which inspired the research of few-shot learning. Few-shot learning aims to enable the model to efficiently learn new concepts and categories with only a small amount of labeled data, and enhance its generalization ability in data-scarce environments. Recent research includes designing initial network parameters to quickly adapt to new classes, or building powerful visual embedding networks to learn similarity metrics between samples. However, these methods are still not robust enough when dealing with peripheral samples with the smallest semantic features in each category.

[0003] In order to further improve the classification ability of the model under small sample conditions, some studies have introduced cross-modal information, especially using natural language to enhance the semantic information of images. By constructing text prototypes or using text descriptions to correct classifiers, researchers have achieved certain results in semantic alignment and sample richness. In recent years, the development of large multimodal models has made it possible to align text and images with high quality. These models have accumulated rich semantic knowledge through pre-training on large-scale multimodal datasets, and have demonstrated strong visual understanding and contextual learning capabilities. In natural language processing and visual tasks, large multimodal models can quickly adapt and complete tasks through contextual learning with a small number of examples, thereby greatly reducing the cost of pre-training and fine-tuning.

[0004] However, despite the excellent performance of large multimodal models in tasks such as image caption generation and image question answering, studies have shown that their performance in small sample classification tasks is still insufficient. When faced with unseen image-label pairs, the model tends to imitate human response patterns rather than truly understand the image content, which makes it difficult for the model to learn efficiently from the context. In addition, the model is often affected by position bias when selecting multiple candidate options, giving priority to options that appear earlier. Therefore, it is still challenging for large multimodal models to achieve effective classification of new categories with a small number of examples. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal large model small sample classification method based on knowledge enhancement to address the above-mentioned deficiencies, which is enhanced in the fine-tuning and reasoning stages, providing a more direct and accurate solution for using multimodal large models to perform small sample classification tasks.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A multimodal large model small sample classification method based on knowledge enhancement includes the following steps:

[0008] Step 1: Collect a small sample image classification dataset, organize an instruction fine-tuning set in the form of an N-way K-shot small sample task on the base class, and organize an instruction test set in the form of an N-way K-shot small sample task on the new class;

[0009] Step 2: Apply the strategy to enhance the images and class name labels in the instruction fine-tuning set, and send them to the multimodal large model for fine-tuning;

[0010] Step 3: In the reasoning stage, a plurality of historical dialogues are randomly selected and input into the multimodal large model together with the instructions in the instruction test set to obtain the initial classification answer A1 of the model;

[0011] Step 4, at the same time, for each image in the instruction test instruction, prompt the multimodal large model to generate multiple attribute descriptions;

[0012] Step 5: Calculate the similarity of the attribute descriptions between the image samples obtained in step 4 and sort them, and select N / / 2 reliable candidate answers to form a new candidate answer set C N / / 2 ;

[0013] Step 6: For the initial classification answer A1 and the new candidate answer set C N / / 2 , determine whether A1 is reliable. If it is reliable, directly output the answer as the final classification result;

[0014] Step 7: If A1 is unreliable, use C N / / 2 Reorganize new instructions, input the multimodal large model for re-inference, and obtain the final classification result.

[0015] As a preferred embodiment of the present invention, the specific process of step 1 is as follows:

[0016] Step 1-1: collect small sample image datasets of different granularities from different scenes and fields and their preset classification categories. If the small sample image dataset has no preset classification category, randomly sample all categories in the small sample image dataset at a ratio of 7:3 to obtain the base class set D base and the new class set D novel ;

[0017] Step 1-2, construct instructions in the form of small sample tasks:

[0018] Steps 1-3: randomly sample N categories in the small sample image dataset, sample K samples for each category, and form a support set S; randomly select a category from these N categories and sample 1 sample from the non-support set as the query set Q;

[0019] Fill the support set image path, image label name, and query image path into the instructions described in step 1-2 to form an instruction for a small sample task;

[0020] Step 1-4: A small sample task obtained through steps 1-2 and 1-3 is regarded as a meta-task. base Sample multiple meta-tasks in the dataset, shuffle the order of the meta-tasks obtained by sampling different small sample image datasets and integrate them into one file to obtain the instruction follow fine-tuning set;

[0021] Step 1-5: New class set D in a small sample image dataset novel Multiple meta-tasks are sampled in the test set, and the meta-tasks obtained by sampling different small sample image datasets are shuffled and integrated into one file to obtain the instruction-following test set.

[0022] As a preferred solution of the present invention, the instruction format of the small sample task form in step 1-2 is:

[0023] Image 1: <Image 1> What is this? {Class Name 1}

[0024] Image 2:<Image 2>What is this? {Class Name 2}

[0025] …

[0026] Image N: <Image N> What is this? {Class Name N}

[0027] Query image: <Query image> What is this? Please select an answer from the candidate answer list: [{Class name 1},

[0028] {class name 2},…,{class name N}].

[0029] As a preferred embodiment of the present invention, the specific process of step 2 is as follows:

[0030] Step 2-1: adopt a character perturbation enhancement strategy for the class name label;

[0031] Step 2-2, adopting image enhancement strategy for the image;

[0032] Step 2-3: Input the enhanced fine-tuning instruction into the multimodal large model f to obtain a response; and use LoRA (add Chinese explanation) fine-tuning, compare the obtained response with the true value, calculate the loss, and reversely update the adapter layer of the multimodal large model.

[0033] LoRA, or Low-Rank Adaptation, is an efficient parameter fine-tuning technique that enables rapid customization of large multimodal models by adding a small number of trainable parameters to the model while keeping the original model parameters unchanged.

[0034] As a preferred solution of the present invention, in step 2-3, LoRA fine-tuning is adopted. Specifically, LoRA low-rank decomposition is used to update the adapter layer (i.e., adapter) of the multimodal large model, and the parameters of the remaining visual encoder and language model parts are fixed; a bypass is added next to the adapter, matrix A is initialized with a random Gaussian distribution, matrix B is initialized with a zero matrix, and the adapter parameter W is fixed during training. 0 , only train matrix A and matrix B; after training, update the parameters as follows:

[0035] W 0 +ΔW=W 0 +BA,

[0036] In the formula, ΔW represents the change of the parameter;

[0037] For input x, the forward propagation process of the model is updated as:

[0038] h=W 0 x+ΔWx=W 0 x+BAx.

[0039] Where h represents the updated parameter.

[0040] As a preferred solution of the present invention, the character perturbation enhancement strategy described in step 2-1 includes but is not limited to the following:

[0041] Strategy 1: Split and reassemble, which is to split the tag name according to specific symbols, and then reassemble the split parts in a random order;

[0042] Strategy 2, character reversal, that is, randomly selecting a few characters at the end of the tag name and moving them to the beginning of the tag name;

[0043] Strategy 3, random insertion, that is, randomly extracting these characters from 0 to 9, a to z, A to Z, and inserting them into random positions of the tag name;

[0044] Strategy 4, character shuffling, that is, taking all the characters in the label name, randomly shuffling them and reordering them;

[0045] Strategy 5, character replacement, that is, randomly select some characters in the tag name and replace them with any characters from 0 to 9, a to z, A to Z;

[0046] Strategy 6, reverse order, that is, completely reverse the order of characters in the tag name;

[0047] Strategy 7, character repetition, that is, randomly selecting some substrings in the tag name and repeating the substring 1 to 2 times at the end of the tag name;

[0048] Strategy 8, character deletion, randomly deletes certain characters in the tag name.

[0049] As a preferred solution of the present invention, the image enhancement strategy in step 2-2 includes but is not limited to the following:

[0050] Cropping, randomly crop the sample image;

[0051] Rotation: randomly rotate the sample image to one of the following angles: 90°, 180°, or 270°;

[0052] Flip: flip the sample image horizontally and vertically.

[0053] As a preferred solution of the present invention, the specific process of step 3 is:

[0054] Step 3-1, input the test instructions in the instruction test data set into the multimodal large model;

[0055] Step 3-2: Use the fine-tuning instruction set as a historical dialogue set, randomly select one or two historical dialogues from it, and input them into the historical dialogue interface of "history" of the multimodal large model;

[0056] Step 3-3: Obtain the output of the multimodal large model as the initial classification result A1.

[0057] As a preferred solution of the present invention, the specific process of step 4 is:

[0058] Step 4-1: Perform adaptive attribute selection, specifically: ask the multimodal large model which attributes of the samples will be focused on for the current data set, and then obtain M attributes that the model adaptively selects;

[0059] Step 4-2, automatic prompt generation, specifically: ask the multimodal large model what kind of prompts will be used to prompt the model to generate descriptions of the current data set and the attributes that need to be focused on, and then obtain the prompts automatically generated by the model;

[0060] Step 4-3, generating local attribute descriptions, specifically: sending each image sample in the test instruction to the large model visual end, sending the prompt automatically generated in step 4-2 to the large model text end, and obtaining M attribute descriptions about each image sample;

[0061] Step 4-4, global description generation, specifically: sending the image samples in the test instruction and their corresponding M attribute descriptions to the large model, requiring the model to summarize the global description;

[0062] Step 4-5, attribute description summary, specifically: for each image sample, a total of M+1 attribute descriptions are obtained, which are saved in a dictionary format of "attribute description aspect: attribute description content".

[0063] As a preferred solution of the present invention, in step 4-1, for the fine-grained data set, the prompt used for asking questions is:

[0064] “For the fine-grained {birds / flowers / dogs / …}, which attributes do you care about and briefly explain why”,

[0065] For general domain datasets, the prompts used to ask questions are:

[0066] "Unlike fine-grained datasets, general datasets do not have fixed, specific feature attributes. Therefore, you need to find attribute features that can be widely adapted for general datasets. Which attributes will you focus on?"

[0067] As a preferred embodiment of the present invention, the prompt used in the question in step 4-2 is

[0068] "Please design prompts for these attributes ({attribute 1}, {attribute 2}, {attribute 3}, ...) so that the large model can accurately describe these attribute descriptions."

[0069] As a preferred solution of the present invention, the prompt used in the question in step 4-3 is "<image path>{attribute k prompt}".

[0070] As a preferred embodiment of the present invention, the prompt used in the question in step 4-4 is

[0071] “<Image path>{attribute 1}:{attribute description 1},{attribute 2}:{attribute description 2}…, please combine the above attribute descriptions and give a global, overall description.”

[0072] As a preferred solution of the present invention, the specific process of step 5 is:

[0073] Step 5-1, for each attribute, calculate the text similarity between the attribute description of the query image in the current instruction and the attribute description of the support image in the current instruction to obtain a similarity vector;

[0074] Step 5-2, adding the M+1 similarity vectors obtained in step 5-1 to obtain an aggregated similarity vector;

[0075] Step 5-3: Sort by similarity vector after aggregation, select the top N / / 2 candidate answers with the highest similarity, and form a new candidate answer set C N / / 2 .

[0076] As a preferred solution of the present invention, the specific process of step 6 is:

[0077] If the initial classification answer A1 is in the new candidate answer set C N / / 2 If the classification result is in the range of 0 to 1, it is considered reliable and output as the final classification result.

[0078] As a preferred solution of the present invention, the specific process of step 7 is:

[0079] Step 7-1: If the initial classification answer A1 is not in the new candidate answer set C N / / 2 If the value is medium, it is considered unreliable;

[0080] Step 7-2, reorganize the instructions and change C N / / 2 and the initial classification answer A1 as the new candidate answer set C N / / 2+1 Replace the candidate answer set in the original instruction and delete the candidate answer that is not in C N / / 2+1 Supporting samples of the categories in ;

[0081] Step 7-3: Re-input the newly organized instructions into the multimodal large model, and the obtained response is used as the final classification result.

[0082] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0083] The present invention proposes a multimodal large model small sample classification method based on knowledge enhancement, which specifically includes two parts of enhancement methods;

[0084] The first part is an enhancement method based on instruction fine-tuning. By collecting rich data sets from different fields, the good generalization of the multimodal large model is guaranteed. At the same time, through the graphic enhancement strategy, the multimodal large model can be more focused on the examples provided by the support set in the instruction, so as to analyze and compare the relationship between the query sample and the example. In addition, by fine-tuning the model through meta-learning, the multimodal large model can learn the analysis and comparison method from multiple meta-tasks instead of imitating fixed reasoning paths and answer formats. Compared with the multimodal large model that does not adopt the instruction fine-tuning enhancement strategy, the classification accuracy has been greatly improved.

[0085] The second part is an enhancement method based on reasoning and screening. Through historical conversations, the multimodal large model can quickly understand the form and answer format of the current task, which helps the model understand the task intent more quickly. The attribute-based screening method for candidate answers can use attribute descriptions to calculate text similarity as the prior classification result, filter out unreliable categories, and reduce task complexity. The self-consistency of the multimodal large model can be further verified in re-reasoning. Compared with the multimodal large model that does not adopt the reasoning enhancement strategy, the classification accuracy has been greatly improved, and it is also applicable to the unfine-tuned large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] The present invention will be further described below in conjunction with the accompanying drawings:

[0087] Figure 1 It is an algorithm flow chart of the present invention based on the multimodal large model small sample classification method based on knowledge enhancement;

[0088] Figure 2 is a schematic diagram of an instruction following dataset in the form of a small sample task under the meta-learning paradigm of the present invention;

[0089] Figure 3 It is a schematic diagram of an instruction-enhanced following dataset in the form of a small sample task under the meta-learning paradigm of the present invention;

[0090] Figure 4 is a schematic diagram of a method for generating attribute descriptions in the present invention;

[0091] Figure 5 It is a schematic diagram of the model method of the present invention. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] Embodiment 1:

[0094] Below, the relevant background knowledge involved in the embodiments of the present application is introduced:

[0095] Contextual learning: Contextual learning is also called situational learning. It allows the model to learn and reason in a given context without making any updates to the actual parameters of the model. This means that the model can adjust its behavior based on the input contextual information without changing its pre-trained state. In contextual learning, the core advantage of the model lies in its contextual dependence. It performs reasoning by analyzing the provided contextual information (such as examples and task descriptions) instead of relying on an explicit training process to adjust its parameters. In practical applications, the model reasoned based on the provided context and generated context-related responses. This process relies on the extensive knowledge learned by the model during pre-training, enabling it to generate reasonable outputs based on the current context. This approach not only improves the flexibility of the model, but also reduces the resources and time required to retrain the model for specific tasks.

[0096] Division of base classes and new classes: For a dataset, there is an image set X, a label set C, and an image-text pair represented as (x, y). In a small sample experimental setting, it is usually divided into two parts: one is the base class set D base ={(x,y)|x∈C base ,y∈C base} is used for fine-tuning, and the second is the new class set D novel ={(x,y)|x∈X novel ,y∈C novel} is used to test that the categories of the two sets do not intersect, that is,

[0097] N-way K-shot: The fine-tuning phase starts from the base class set D base Sampling task, new class set D in the test phase novel Sampling task: for each N-way K-shot task, randomly sample N categories, and randomly sample K categories from each category to form a support set The remaining samples constitute the query set Where M is the number of remaining samples in these N categories, and the task of the model is to correctly classify the samples in the query set Q based on the information provided by the support set S.

[0098] Multimodal large model structure: A multimodal large model usually consists of three parts: a visual encoder (or encoder of other modalities), an adapter layer, and a large language model. The visual encoder is used to process visual information and convert images into feature embeddings that the model can understand. The adapter layer converts visual embeddings into dimensions aligned with text embeddings. The large language model is used to process text information, convert text into text embeddings that the model can understand, and output the final response.

[0099] Combination Figure 1As shown, the multimodal large model small sample classification method based on knowledge enhancement of the present invention comprises the following steps:

[0100] 1. Collect a small sample image classification dataset, organize an instruction fine-tuning set and an instruction test set in the form of an N-way K-shot small sample task on the base class and the new class. The specific process is as follows:

[0101] (1) Eight public small-sample image classification datasets are used in this embodiment, including three general-domain datasets, namely CIFAR-FS, Tiered-ImageNet, and Mini-ImageNet, and five fine-grained datasets, namely CUB, Flower102, FGVC, StanfordDogs, and StanfordCars, on birds, flowers, airplanes, dogs, and cars, respectively. The datasets are divided according to the preset classification categories of the collected datasets, where the new class is used for testing and the base class is used for fine-tuning. In order to collect a richer fine-tuning dataset, nine public image datasets, namely GTSRB, Country211, EurosatClip, Resisc45_CLIP, Caltech-101, OxfordPets, Food101, Mnist, and Fer2013, are also used in this example. Since there is no preset classification category for the small-sample base class / new class, the base class and the new class are randomly sampled and divided according to a ratio of 7:3, where only the base class is used for fine-tuning.

[0102] (2) In the new class set D novel Up-sampling is performed to construct instructions in the form of N-way K-shot small sample tasks as shown below to form an instruction test dataset. In this embodiment, the experimental setting is 5-way 1-shot:

[0103] Image 1: <Image 1> What is this? {Class Name 1}

[0104] Image 2:<Image 2>What is this? {Class Name 2}

[0105] …

[0106] Image N: <Image N> What is this? {Class Name N}

[0107] Query image: <Query image> What is this? Please select an answer from the candidate answer list

[0108] [{class name 1}, {class name 2}, …, {class name N}].

[0109] The English expression is used in this embodiment, as shown below:

[0110] Picture 1:<img_path 1> What is this? {classname 1}

[0111] Picture 2:<img_path 2> What is this? {classname 2}

[0112] …

[0113] PictureN:<img_path N> What is this? {classnameN}

[0114] Query Picture:<query_img_path> What is this? Output is one of[{classname1},{classname2},…,{classnameN}].

[0115] (3) Randomly sample N categories in the data set, sample K samples from each category, and form a support set S; randomly select a category from the N categories and sample a sample from a non-support set as the query set Q; fill the support set image path, image label name, and query image path into the instructions described in 3.1 to form an instruction for a small sample task. In this embodiment, the 5-way 1-shot setting is used, that is, randomly sample one sample from each of the five categories to form a support set, and randomly select the remaining samples from one of the five categories as the query set;

[0116] (4) A 5-way 1-shot small sample task obtained by the above two steps is regarded as a meta-task, and is constructed in a base class set D of a dataset. base Sample multiple meta-tasks in the dataset, shuffle the order of the meta-tasks obtained by sampling different datasets and integrate them into one file to obtain the instruction follow-up fine-tuning set, which is stored in a structured json file;

[0117] (5) Follow the same method as above to create a new class set D novel The instruction following test set is obtained by sampling.

[0118] 2. For the instructions in the instruction fine-tuning set, apply the strategy to enhance the image and class name labels, and send them to the multimodal large model for fine-tuning. The specific process is as follows:

[0119] (1) A character perturbation enhancement strategy is adopted for the label. In this embodiment, one of the following methods is randomly adopted for each label name:

[0120] Split and reassemble: split the label name according to specific symbols, and then reassemble the split parts in shuffled order, such as "757-200" becomes "200-757" after splitting and reassembling;

[0121] Character reversal, randomly select a few characters at the end of the tag name and move them to the beginning of the tag name, such as "cellphone" becomes "onecellph" after character reversal;

[0122] Random insertion: randomly select a number between 1 and 10 as the number of inserted characters, and randomly select these characters from 0 to 9, a to z, A to Z, and insert them into random positions of the tag name. For example, "lamp" becomes "lZCa0md4p2" after random insertion.

[0123] Character shuffling: Take out all characters in the label name, randomly shuffle them and reorder them, such as "camera" becomes "aremca" after character shuffling;

[0124] Character replacement, randomly select certain characters in the tag name and replace them with any characters from 0 to 9, a to z, A to Z. For example, "pyramid" becomes "p1raM2D" after character replacement.

[0125] Reverse order, completely flip the order of characters in the tag name, such as "kangaroo" becomes "ooragnak" after reverse order;

[0126] Character repetition, randomly select certain substrings in the tag name, and repeat the substring 1 to 2 times at the end of the tag name, such as "ferry" becomes "ferrryrryrry" after character repetition;

[0127] Character deletion, randomly delete some characters in the tag name, such as "strawberry" becomes "stabrry" after character deletion;

[0128] (2) Adopting an image enhancement strategy for the image. In this embodiment, several of the following methods are randomly adopted for each image:

[0129] Cropping: Randomly crop the sample image. The cropping areas are the upper left, lower left, upper right, lower right, and center. The size is 1 / 2 of the original image size in length and 1 / 2 in width.

[0130] Rotate: randomly rotate the sample image to one of the following angles: 90°, 180°, or 270°;

[0131] Flip: flip the sample image horizontally or vertically;

[0132] (3) In this embodiment, the multimodal large model used is Qwen-VL-Chat-Int4, the visual encoder and large language model are frozen, the Lora low-rank decomposition adapter layer is used, and the learning rate is set to 1×10 -5 , the warm-up ratio is 0.01, the Adam optimizer is used, and the weight decay hyperparameters β1 are 0.1 and β2 are 0.95;

[0133] (4) The process of Lora low-rank decomposition is to add a bypass next to the adapter, initialize matrix A with a random Gaussian distribution, initialize matrix B with a zero matrix, and fix the adapter parameter W during training. 0 , only train matrix A and matrix B; after training, update the parameters as follows:

[0134] W 0 +ΔW=W 0 +BA,

[0135] For input x, the forward propagation process of the model is updated as:

[0136] h=W 0 x+ΔWx=W 0 x+BAx;

[0137] (5) The process of instruction fine-tuning is to pass the instructions in the fine-tuning dataset into the multimodal large model. The model generates output according to the instructions, compares it with the actual true value, and uses LoRa to update the adapter layer parameters to finally obtain the fine-tuned model parameters.

[0138] 3. In the reasoning stage, multiple historical dialogues are randomly selected and input into the multimodal large model together with the instructions in the test set to obtain the model's initial classification answer A1. The specific process is as follows:

[0139] (1) In this embodiment, the multimodal large model used is Qwen-VL-Chat-Int4. The key codes for receiving input and output of the multimodal large model are as follows:

[0140] response,_=model.chat(tokenizer,query=input_text,history=history,append_history=False)

[0141] Input the instructions in the instruction test set as input text input_text into the query query of the multimodal large model;

[0142] (2) At the same time, a historical dialogue is randomly selected from the historical dialogue collection, i.e., the instruction test set, and sent to the “historical dialogue”, i.e., the history interface, of the multimodal large model;

[0143] (3) Obtain the response reponse of the multimodal large model as the initial classification answer A1 of the model.

[0144] 4. For each image in the test instruction, prompt the multimodal large model to generate multiple attribute descriptions, combined with Figure 2 , the specific process is as follows:

[0145] (1) Adaptive attribute selection: ask the multimodal large model which attributes of the sample will focus on for the current data set, and obtain M attributes that the model adaptively selects. In this embodiment, M is set to 4. Take the fine-grained bird data set CUB as an example, and ask questions and obtain responses:

[0146] ---------------------------------------------------------------------

[0147] User: "For the fine-grained {birds / flowers / dogs / …}, which attributes do you care about and briefly explain why? Please select the four attributes you care about most."

[0148] Multimodal large model: “Color distribution: is the most basic visual feature for bird identification, and different bird species usually have unique color patterns;

[0149] Shape and size: provides important information about a bird's morphology, which can indicate its lifestyle and habitat preferences;

[0150] Plumage characteristics: Helps identify species with unique plumage characteristics or variations;

[0151] Behavioral features: can provide additional context for accurate identification”;

[0152] ---------------------------------------------------------------------

[0153] For the fine-grained flower dataset Flower102, the attributes obtained are petal shape and size, color and pattern, calyx shape and size, structure and arrangement;

[0154] For the fine-grained aircraft dataset FGVC, the attributes obtained are airfoil, fuselage design, engine configuration, and tail configuration;

[0155] For the fine-grained dog dataset StanfordDogs, the attributes obtained are shape and size, hair length and type, color and pattern, and head or facial features;

[0156] For the fine-grained car dataset StanfordCars, the attributes obtained are front design, wheel design, body details and decoration, body shape and size;

[0157] For general domain datasets, the prompts for the questions and the attributes of the responses are common:

[0158] User: "Unlike fine-grained datasets, general datasets do not have fixed, specific feature attributes. Therefore, you need to find widely applicable attribute features for general datasets. Which attributes will you focus on?"

[0159] Multimodal Grand Model: “Color and Texture: Provide a detailed analysis of the image’s color scheme, mentioning the dominant colors and any notable contrasts or gradients; Shape: Identify and describe the geometric shapes present in the image; Activity: Analyze any dynamic elements or actions depicted in the image; Scene: Describe the scene or environment captured in the image”.

[0160] (2) Automatic prompt generation: ask the multimodal large model what kind of prompts it will use to generate descriptions of the current data set and the attributes that need to be focused on, and get the prompts automatically generated by the model. The prompts used in the question are "Please design prompts for these attributes ({attribute 1}, {attribute 2}, {attribute 3}, ...) so that the model can generate good image attribute descriptions based on the prompts."

[0161] Taking the fine-grained bird dataset CUB as an example, the prompts and responses are as follows:

[0162] User: “Design hints for the attributes in step 1 so that the model can generate good image attribute descriptions based on the hints.”

[0163] Multimodal large model: “Color distribution: describes the main color and color distribution of various parts of the bird’s body, such as the head, back, abdomen, wings and tail;

[0164] Shape and size: Describe the size and shape of the bird in the picture in detail, including the shape of its head, beak, wings, tail, legs, etc.

[0165] Feather characteristics: A detailed description of the feather type and structure of the bird shown in the image in terms of feather characteristics, including length, texture, and arrangement of feathers;

[0166] Behavioral characteristics: Describe in detail the behavior and posture of the bird shown in the picture from the perspective of behavioral characteristics, such as standing, flying, foraging or other activities.

[0167] (3) Generate local attribute descriptions. Send each image sample in the instruction to the visual end of the large model, and send the automatically generated prompts to the text end of the large model to obtain M attribute descriptions for each image sample. The prompt used for the question is "<image path>{attribute k prompt}".

[0168] Taking the fine-grained bird dataset CUB as an example, the prompts for the questions are as follows:

[0169] User: "<image path> For the object in the picture, please describe the dominant color and color distribution of each part of the bird's body, such as the head, back, abdomen, wings, and tail."

[0170] Fill in the prompts of other attributes in the same way as above. For each attribute, the attribute description of the obtained multimodal large model is as follows:

[0171] Multimodal large model: “Color distribution: This bird has a blue head, green back, red belly, and red breast. The wings are green and black. The tail is black and green.”

[0172] Shape and Size: It is small and bright, it has a blue head, red throat, green back, and black, red and green wings. Its tail is black and its legs are gray. The bird's beak is small and also gray.

[0173] Feather Characteristics: It has colorful feathers that come in blue, green, red, and orange. The feathers are long and soft in texture. They are arranged in such a way that the bird can fly and maneuver in the air.

[0174] Behavioral characteristics: It is perched on a branch with its head turned to the right. It appears to be looking at something in the distance, perhaps scanning for potential threats or scanning for food.

[0175] (4) Global description generation: The image sample and M attribute descriptions are fed into the large model, and the model is required to summarize the global description. The prompt used in the question is "<image path>{attribute 1}:{attribute description 1}, {attribute 2}:{attribute description 2}... Please combine the above attribute descriptions to give a global, overall description."

[0176] Taking the fine-grained bird dataset CUB as an example, the prompts for the questions are as follows:

[0177] User: "<Image path> Color distribution: This bird...; Shape and size: It is...; Feather characteristics: It has colorful feathers...; Behavioral characteristics: It perches on a branch..., please combine the above attribute descriptions to give a global and overall description."

[0178] Multimodal Large Model: "Global Description: The bird is a multi-colored bird with green, blue, red, and orange color patterns. It has a blue head, green back, and red belly. The wings are green with brown stripes. The bird is sitting on a branch. The bird is small and compact with a short tail and short legs. The feathers are long and thin, and the texture looks soft and fuzzy. The behavior of the bird is sitting still with its eyes open, staring at something in front of it."

[0179] (5) Attribute description summary: For each image sample, a total of M+1 attribute descriptions are obtained, which are saved in the dictionary format of "attribute description aspect: attribute description content". The format of the saved json file is as follows:

[0180] <image path>: {

[0181] "<attribute 1>": "<attribute description 1>";

[0182] …

[0183] "<attribute M>": "<attribute description M>";

[0184] "Global Description": "<global description>"

[0185] },

[0186] Taking the fine-grained bird dataset CUB as an example, the format of the saved json file is as follows:

[0187] <image path>: {

[0188] “Color distribution”: “The bird has a blue head…”;

[0189] “Shape and size”: “It is small and bright…”;

[0190] “Feather characteristics”: “It has colorful feathers…”;

[0191] “Behavioral characteristics”: “It perches on a branch…”;

[0192] "Global Description": "This bird is a multi-colored bird..."

[0193] }.

[0194] 5. Calculate the similarity of attribute descriptions between image samples and sort them, and select N / / 2 reliable candidate answers to form a new candidate answer set C N / / 2 , the specific process is as follows, the specific process is as follows:

[0195] (1) For the first attribute, the similarity between the attribute descriptions of the five support samples and the attribute description of the query sample is calculated. The text encoder used in this embodiment is the frozen model SBERT (all-MiniLM-L6-v2) to measure text similarity and obtain the text attribute similarity

[0196] (2) For four attribute descriptions and one global description, we can get S attr1 , S attr2 , S attr3 , S attr4 , S global ;

[0197] (3) Add the obtained similarity vectors to obtain the aggregated text similarity

[0198] S aggr =S attr1 +…+S attr4 +S global ;

[0199] (4) According to S aggr , sort the five candidate answers, sort the top N / / 2 similarities, that is, the two candidate answers with the highest similarity, as the new candidate answer set C N / / 2 .

[0200] 6. For the initial classification answer A1 and the new candidate answer set C N / / 2 , judge whether A1 is reliable. If it is reliable, directly output the answer as the final classification result. The specific process is as follows:

[0201] (1) If the initial classification answer A1 appears in the new candidate answer set C N / / 2 If the value is medium, it is considered reliable;

[0202] (2) Output the initial classification answer A1 as the final answer.

[0203] 7. If A1 is unreliable, use C N / / 2 Reorganize new instructions, input the multimodal large model for re-inference, and obtain the final classification result. The specific process is as follows:

[0204] (1) If the initial classification answer A1 does not appear in the new candidate answer set C N / / 2 If the value is medium, it is considered unreliable;

[0205] (2) Reorganize the instructions and change C N / / 2 and the initial classification answer A1 as the new candidate answer set C N / / 2+1 Replace the candidate answer set in the original instruction and delete the candidate answer that is not in C N / / 2+1 Support samples of the categories in , taking the fine-grained bird dataset CUB as an example, the original test instruction is:

[0206] Picture 1:<img_path 1> What is this? Fox Sparrow.

[0207] Picture2:<img_path 2> What is this? Common Tern.

[0208] Picture 3:<img_path 3> What is this? White crownedsparrow.

[0209] Picture 4:<img_path 4> What is this? Paintedbunting.

[0210] Picture 5:<img_path 5> What is this? Yellow breastedchat.

[0211] Query Picture:<query_img_path> What is this? Output is one of[FoxSparrow,Common Tern,White crownedsparrow,Paintedbunting,Yellowbreastedchat].

[0212] The model’s initial classification answer A1 is “Fox Sparrow”. After the attribute-based reasoning enhancement method, the new candidate answer set C N / / 2 ={"White crowned sparrow","Common Tern"}, since A1 does not appear in C N / / 2 If the N / / 2 and the initial classification answer A1 as the new candidate answer set C N / / 2+1 Replace the candidate answer set in the original instruction and delete the candidate answer set that is not in CN / / 2+1 The new test guide

[0213] Let:

[0214] Picture 1:<img_path 1> What is this? White crownedsparrow.

[0215] Picture2:<img_path 2> What is this? Common Tern.

[0216] Picture 3:<img_path 3> What is this? Fox Sparrow.

[0217] QueryPicture:<query_img_path> What is this? Output is one of[Whitecrownedsparrow,Common Tern,Fox Sparrow].

[0218] (3) Input the new test instruction into the large model to obtain a response. If it is inconsistent with the initial classification answer A1, the sample order in the test instruction can be disrupted and re-input into the model to obtain a response. The majority vote is taken on the response results of multiple rounds, that is, the answer with the most occurrences is selected. In this embodiment, for efficiency considerations, the result of the first round of directly inputting the new test instruction into the large model is regarded as the final classification result.

[0219] Specifically, the present invention is further described below by way of embodiments:

[0220] A. Experimental conditions

[0221] In this embodiment, the experimental environment is GPU NVIDIA GeForce RTX 3080, Python 3.9, torch2.0.0+cu118, torchvision 0.15.1+cu118, and the experimental setting is 5-way 1-shot. The model used in the experiment is Qwen-VL-Chat-Int4 (Qwen-VL for short).

[0222] B. Experimental results evaluation criteria

[0223] The classification accuracy is used as the evaluation criterion. The higher the accuracy, the better the model performance.

[0224] Since the output of a large, untuned multimodal model is usually long and contains redundant information, it cannot be directly matched and compared with the true value. Therefore, three metrics are designed to calculate the classification accuracy:

[0225] Acc: Apply regular expressions to filter non-alphanumeric symbols in model responses and true values, and strictly match model output with true values. If they match exactly, they are considered correct.

[0226] Acc ocour : Apply a regular expression to filter non-alphanumeric symbols in the model response and the true value. If the true value appears anywhere in the model response, it is considered correct.

[0227] Acc clip : CLIP-L 14 is used to map the model's response to the candidate answer set, that is, the text similarity between the model response and the candidate answer is calculated, and then the result is strictly matched with the true value.

[0228] The experimental results of this embodiment use Acc by default clip evaluation criteria.

[0229] C. Experimental comparative analysis

[0230] On three general datasets, CIFAR-FS (CIFAR for short), Tiered-ImageNet (Tiered for short), and Mini-ImageNet (MINI for short), and five fine-grained datasets, CUB, Flower102 (Flower for short), FGVC, StanfordDogs (Dogs for short), and StanfordCars (Cars for short), the performance of the unfine-tuned multimodal large model Qwen-VL on these eight datasets is given, and the performance and improvement of the multimodal large model Qwen-VL to which the method of the present invention is applied are also given.

[0231] Table 1 Performance comparison of small sample image classification methods in four datasets with general domains

[0232] method CIFAR Tiered MINI CUB BAVARDAGE 87.35 85.20 84.80 90.42 EASY3×ResNet12 87.16 84.29 84.04 90.56 PEMnE-BMS★ 88.44 86.07 85.54 94.78 PTMAP-SF-SOT <![CDATA[ 89.94 ]]> - 85.59 <![CDATA[ 95.80 ]]> P>M>F 84.30 - 95.30 - CAML 70.80 <![CDATA[ 95.40 ]]> <![CDATA[ 96.20 ]]> 91.80 Qwen-VL 37.04 57.14 39.38 45.00 Method of the present invention (fine-tuning enhancement) 95.02 98.06 98.24 96.40

[0233] Table 2 Performance comparison of fine-grained domain small sample image classification methods on four fine-grained datasets

[0234]

[0235]

[0236] This example was experimented on 3 coarse-grained and 5 fine-grained datasets, and the results were compared with the most advanced methods. The comparison results in Table 1 and Table 2 with the current cutting-edge small sample image classification methods show that the small sample image classification effect of the method of the present invention exceeds that of other compared methods, which illustrates the effectiveness of the method in general and fine-grained fields.

[0237] In order to verify the main contribution of the model proposed in this example, a series of ablation experiments were conducted. The main focus was on the influence of instruction fine-tuning and label and image enhancement modules. The results of these ablation studies provide an explanation for the basic role of each method in the final performance of the model, as shown in Table 3.

[0238] Table 3 The impact of each module on the small sample image classification method on eight datasets

[0239]

[0240] This embodiment also specifically gives the average benefits of different modules for model small sample classification, as shown in Table 4.

[0241] Table 4. Average benefits of each module for small sample image classification on eight datasets

[0242] Instruction fine-tuning Fine-tuning enhancements Reasoning Enhancement Average performance Average revenue × × × 43.12 - √ × × 96.39 +53.27 √ √ × 97.29 +54.17 × × √ 68.81 +25.69 √ × √ 96.71 +53.59 √ √ √ 97.45 +54.33

[0243] As shown in Tables 3 and 4, when no fine-tuning strategy and enhancement module are used, the performance is poor. Except for the Tiered dataset, the accuracy of the model on the other seven datasets does not exceed 50%; when instruction fine-tuning is used, the accuracy is significantly improved, but not optimal; when instruction fine-tuning and fine-tuning enhancement are used at the same time, better results can be obtained. For the unfine-tuned multimodal large model, the model performance is significantly improved by directly applying the inference enhancement module; when instruction fine-tuning and fine-tuning enhancement are combined, the model performance reaches the optimal level.

[0244] At the same time, in order to verify the reliability of the generated attribute description, this embodiment directly calculates the text similarity between a certain attribute description of the query sample and the corresponding attribute description of the support sample, and takes the option with the greatest similarity as the classification result to verify the validity of the attribute description. This embodiment reports the accuracy of 4 attribute descriptions and 1 global description in Table 5, and reports the average accuracy of the above five attribute descriptions and the accuracy of the top 1 to top 4 of the aggregated attribute descriptions in Table 6.

[0245] Table 5 Accuracy of attribute descriptions generated by the method of the present invention

[0246]

[0247]

[0248] Table 6 Accuracy of description after aggregation by the method of the present invention

[0249] Dataset 5 descriptions average Aggregate top1 Aggregate top2 Aggregation top3 Aggregation top4 CIFAR 65.03 77.40 89.92 95.70 98.26 Tiered 73.76 82.66 95.12 98.78 99.84 MINI 66.75 76.58 89.34 95.56 98.76 CUB 47.58 66.62 84.82 93.46 97.56 Flowers 55.03 72.64 87.76 94.72 98.32 FGVC 34.94 43.86 66.10 80.98 92.20 Dogs 35.82 50.50 71.38 83.42 92.98 Cars 61.14 88.74 95.46 98.00 99.40

[0250] As shown in the table above, using attribute descriptions alone to assist classification is not reliable, especially in several fine-grained datasets, because fine-grained datasets require more discriminative attribute descriptions. Compared with the accuracy of individual attribute descriptions, the aggregated description exceeds the average accuracy of five individual descriptions in terms of top1 accuracy. This is reasonable, based on the fact that the true value images in the query image and the support image are not necessarily the most similar in some attributes, but there is a greater probability that they are not the lowest in similarity. Therefore, the aggregated attributes have a good accuracy, and even the top2 accuracy on most datasets has reached more than 84%, and even more than 93% for top3. For the comprehensive consideration of efficiency and reducing task complexity, the range of reliable candidate answers selected for the 5-way 1-shot experimental setting in this embodiment is top2.

[0251] The above embodiments are only for illustrating the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A multimodal large model small sample classification method based on knowledge enhancement, characterized in that: The steps include: Step 1: Collect a small sample image classification dataset, organize an instruction fine-tuning set in the form of an N-way K-shot small sample task on the base class, and organize an instruction test set in the form of an N-way K-shot small sample task on the new class; Step 2: Apply the strategy to enhance the images and class name labels in the instruction fine-tuning set, and send them to the multimodal large model for fine-tuning; Step 3: In the reasoning stage, a plurality of historical dialogues are randomly selected and input into the multimodal large model together with the instructions in the instruction test set to obtain the initial classification answer A1 of the model; Step 4, at the same time, for each image in the instruction test instruction, prompt the multimodal large model to generate multiple attribute descriptions; Step 5: Calculate the similarity of the attribute descriptions between the image samples obtained in step 4 and sort them, and select N / / 2 reliable candidate answers to form a new candidate answer set C N / / 2 ; Step 6: For the initial classification answer A1 and the new candidate answer set C N / / 2 , determine whether A1 is reliable. If it is reliable, directly output the answer as the final classification result; Step 7: If A1 is unreliable, use C N / / 2 Reorganize new instructions, input the multimodal large model for re-inference, and obtain the final classification result.

2. The multimodal large model small sample classification method based on knowledge enhancement according to claim 1 is characterized in that: The specific process of step 1 is as follows: Step 1-1: collect small sample image datasets of different granularities from different scenes and fields and their preset classification categories. If the small sample image dataset has no preset classification category, randomly sample all categories in the small sample image dataset at a ratio of 7:3 to obtain the base class set D base and the new class set D novel ; Step 1-2, construct instructions in the form of small sample tasks: Steps 1-3: randomly sample N categories in the small sample image dataset, sample K samples for each category to form a support set S; randomly select a category from these N categories and sample 1 sample from the non-support set as the query set Q; Fill the support set image path, image label name, and query image path into the instructions described in step 1-2 to form an instruction for a small sample task; Step 1-4: A small sample task obtained through steps 1-2 and 1-3 is regarded as a meta-task. base Sample multiple meta-tasks in the dataset, shuffle the order of the meta-tasks obtained by sampling different small sample image datasets and integrate them into one file to obtain the instruction follow fine-tuning set; Step 1-5: New class set D in a small sample image dataset novel Multiple meta-tasks are sampled in the test set, and the meta-tasks obtained by sampling different small sample image datasets are shuffled and integrated into one file to obtain the instruction-following test set.

3. The multimodal large model small sample classification method based on knowledge enhancement according to claim 2 is characterized in that: The instruction format of the small sample task in step 1-2 is: Image 1: <Image 1> What is this? {Class Name 1} Image 2:<Image 2>What is this? {Class Name 2} … Image N: <Image N> What is this? {Class Name N} Query image: <Query image> What is this? Please select an answer from the candidate answer list: [{class name 1}, {class name 2}, …, {class name N}].

4. The multimodal large model small sample classification method based on knowledge enhancement according to claim 1 is characterized in that: The specific process of step 2 is: Step 2-1: adopt a character perturbation enhancement strategy for the class name label; Step 2-2, adopting image enhancement strategy for the image; Step 2-3, input the enhanced fine-tuning instruction into the multimodal large model f to obtain a response; And LoRA fine-tuning is used to compare the obtained response with the true value, calculate the loss, and reversely update the adapter layer of the multimodal large model.

5. The multimodal large model small sample classification method based on knowledge enhancement according to claim 4 is characterized in that: In the step 2-3, LoRA fine-tuning is used. Specifically, LoRA low-rank decomposition is used to update the adapter of the multimodal large model, and the parameters of the remaining visual encoder and language model parts are fixed; a bypass is added next to the adapter, matrix A is initialized with a random Gaussian distribution, matrix B is initialized with a zero matrix, and the adapter parameter W is fixed during training. 0 , only train matrix A and matrix B; after training, update the parameters as follows: IN 0 +ΔW=W 0 +BA, In the formula, ΔW represents the change of the parameter; For input x, the forward propagation process of the model is updated as: h=W 0 x+ΔWx=W 0 x+BAx。 6. The multimodal large model small sample classification method based on knowledge enhancement according to claim 1 is characterized in that: The specific process of step 3 is as follows: Step 3-1, input the test instructions in the instruction test data set into the multimodal large model; Step 3-2: Use the fine-tuning instruction set as a historical dialogue set, randomly select one or two historical dialogues from it, and input them into the historical dialogue interface of "history" of the multimodal large model; Step 3-3: Obtain the output of the multimodal large model as the initial classification result A1.

7. The multimodal large model small sample classification method based on knowledge enhancement according to claim 1 is characterized in that: The specific process of step 4 is as follows: Step 4-1: Perform adaptive attribute selection, specifically: ask the multimodal large model which attributes of the samples will be focused on for the current data set, and then obtain M attributes that the model adaptively selects; Step 4-2, automatic prompt generation, specifically: ask the multimodal large model what kind of prompts will be used to prompt the model to generate descriptions of the current data set and the attributes that need to be focused on, and then obtain the prompts automatically generated by the model; Step 4-3, generating local attribute descriptions, specifically: sending each image sample in the test instruction to the large model visual end, sending the prompt automatically generated in step 4-2 to the large model text end, and obtaining M attribute descriptions about each image sample; Step 4-4, global description generation, specifically: sending the image samples in the test instruction and their corresponding M attribute descriptions to the large model, requiring the model to summarize the global description; Step 4-5, attribute description summary, specifically: for each image sample, a total of M+1 attribute descriptions are obtained, and they are saved in the dictionary format of "attribute description aspect: attribute description content".

8. The multimodal large model small sample classification method based on knowledge enhancement according to claim 1 is characterized in that: The specific process of step 5 is as follows: Step 5-1, for each attribute, calculate the text similarity between the attribute description of the query image in the current instruction and the attribute description of the support image in the current instruction to obtain a similarity vector; Step 5-2, adding the M+1 similarity vectors obtained in step 5-1 to obtain an aggregated similarity vector; Step 5-3: Sort by similarity vector after aggregation, select the top N / / 2 candidate answers with the highest similarity, and form a new candidate answer set C N / / 2 .

9. The multimodal large model small sample classification method based on knowledge enhancement according to claim 1 is characterized in that: The specific process of step 6 is: if the initial classification answer A1 is in the new candidate answer set C N / / 2 If the classification result is in the range of 0 to 1, it is considered reliable and output as the final classification result.

10. The multimodal large model small sample classification method based on knowledge enhancement according to claim 1 is characterized in that: The specific process of step 7 is as follows: Step 7-1: If the initial classification answer A1 is not in the new candidate answer set C N / / 2 If the value is medium, it is considered unreliable; Step 7-2, reorganize the instructions and change C N / / 2 and the initial classification answer A1 as the new candidate answer set C N / / 2+1 Replace the candidate answer set in the original instruction and delete the candidate answer that is not in C N / / 2+1 Supporting samples of the categories in ; Step 7-3: Re-input the newly organized instructions into the multimodal large model, and the obtained response is used as the final classification result.

Images