A knowledge enhancement-based multi-modal large model small sample classification method

Abstract

The application discloses a kind of based on knowledge enhancement's multi-modal big model small sample classification method, it is related to artificial intelligence and computer vision field, method includes: collection multi-scene image dataset, base class, new class data is constructed as the small sample instruction fine-tuning set and test set in line with meta-learning paradigm;The picture of fine-tuning set and text label are enhanced, and input multi-modal big model and fine-tune;In inference stage, randomly select instruction in historical dialogue and test set, together input the model after fine-tuning and obtain initial classification result;Afterwards, generate multiple attribute descriptions for each image, calculate the similarity between the descriptions of the images and sort them to filter reliable candidate answers;Finally, determine the reliability of the initial answer, if not reliable, then use new candidate answers to construct instructions and re-infer.The application improves the small sample classification ability of multi-modal big model through the knowledge enhancement method based on the two-stage fine-tuning and inference, and is suitable for general and fine-grained classification tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and computer vision, and particularly relates to a multi-modal large model small sample classification method based on knowledge enhancement. BACKGROUND

[0002] With the rapid development of deep learning technology and large-scale datasets, significant progress has been made in the field of machine vision. However, in practical applications, the scarcity of labeled data remains a key bottleneck that hinders the performance improvement of deep learning models. In contrast, humans have the ability to quickly learn and recognize new categories from a small number of examples, which has inspired the study of small sample learning. Small sample learning aims to enable models to efficiently learn new concepts and categories with only a small amount of labeled data, enhancing their generalization ability in data-scarce environments. Recent research includes designing initial network parameters to quickly adapt to new classes, or constructing powerful visual embedding networks to learn similarity measures between samples. However, these methods are still not robust enough when dealing with peripheral samples in each class with minimal semantic features.

[0003] To further enhance the classification ability of models under small sample conditions, some research has introduced cross-modal information, particularly using natural language to enhance the semantic information of images. By constructing text prototypes or using text descriptions to correct the classifier, researchers have made some progress in semantic alignment and sample richness. In recent years, the development of multi-modal large models has provided the possibility for high-quality alignment between text and images. These models are pre-trained on large-scale multi-modal datasets, accumulating rich semantic knowledge and exhibiting strong visual understanding and context learning capabilities. In natural language processing and visual tasks, multi-modal large models can quickly adapt and complete tasks through context learning under the condition of a small number of examples, thereby greatly reducing the cost of pre-training and fine-tuning.

[0004] However, despite the excellent performance of multi-modal large models in image caption generation, image question answering, and other tasks, research has shown that their performance in small sample classification tasks is still insufficient. When faced with unseen image-label pairs, models tend to imitate human response patterns rather than truly understanding image content, making it difficult for models to efficiently learn from context. In addition, models are often affected by position bias when selecting multiple options, preferring options that appear earlier. Therefore, multi-modal large models still face challenges in effectively classifying new categories through a small number of examples. SUMMARY

[0005] The present application aims to provide a multi-modal large model small sample classification method based on knowledge enhancement to address the above deficiencies. The method provides a more direct and accurate solution for performing small sample classification tasks using multi-modal large models by enhancing both the fine-tuning and inference stages.

[0006] To achieve the above object, the present application adopts the following technical solutions:

[0007] A multi-modal large model small sample classification method based on knowledge enhancement, comprising the following steps:

[0008] Step 1, collect small sample image classification data sets, organize instruction fine-tuning sets in the form of N-way K-shot small sample tasks on base classes, and organize instruction test sets in the form of N-way K-shot small sample tasks on new classes;

[0009] Step 2, apply instruction application strategy to enhance pictures and class name labels in the instruction fine-tuning set, and input them into the multi-modal large model fine-tuning;

[0010] Step 3, in the inference stage, randomly select multiple historical dialogues, input them into the multi-modal large model together with the instructions in the instruction test set, and obtain the initial classification answer A1 of the model;

[0011] Step 4, at the same time, for each image in the instruction test instruction, prompt the multi-modal large model to generate multiple attribute descriptions;

[0012] Step 5, calculate the attribute description similarity between the image samples obtained in step 4 and sort them, and select N / / 2 reliable candidate answer groups to form a new candidate answer set C N / / 2 ;

[0013] Step 6, for the initial classification answer A1 and the new candidate answer set C N / / 2 , judge whether A1 is reliable, and if so, directly output the answer as the final classification result;

[0014] Step 7, if A1 is not reliable, use C N / / 2 to reorganize new instructions and input them into the multi-modal large model for re-inference to obtain the final classification result.

[0015] As a preferred scheme of the present application, the specific process of step 1 is:

[0016] Step 1-1, collect small sample image data sets of different granularities from different scenes and fields and their preset classification categories, if the small sample image data sets have no preset classification categories, randomly sample all categories in the small sample image data sets according to a ratio of 7:3 to obtain a base class set D base and a new class set D novel ;

[0017] Step 1-2, construct instructions in the form of small sample tasks:

[0018] Step 1-3, randomly sample N classes in a small sample image dataset, and sample K samples in each class to form a support set S; randomly select a class from the N classes and sample 1 sample in the non-support set as the query set Q;

[0019] Fill the support set picture path, picture label name and query picture path into the instruction in step 1-2 to form an instruction of a small sample task;

[0020] Step 1-4, the small sample task obtained through steps 1-2 and 1-3 is regarded as a meta task, and a plurality of meta tasks are sampled in a base class set D base of a small sample image dataset, and the meta tasks sampled from different small sample image datasets are shuffled and integrated in a file to obtain an instruction following the fine-tuning set;

[0021] Step 1-5, a plurality of meta tasks are sampled in a new class set D novel of a small sample image dataset, and the meta tasks sampled from different small sample image datasets are shuffled and integrated in a file to obtain an instruction following the test set.

[0022] As a preferred scheme of the present application, the instruction format of the small sample task in step 1-2 is:

[0023] Picture 1: <Picture 1> What is this? {Class name 1}

[0024] Picture 2: <Picture 2> What is this? {Class name 2}

[0025] …

[0026] Picture N: <Picture N> What is this? {Class name N}

[0027] Query picture: <Query picture> What is this? Please select an answer from the candidate answer list: [{Class name 1},

[0028] {Class name 2}, …, {Class name N}].

[0029] As a preferred scheme of the present application, the specific process of step 2 is:

[0030] Step 2-1, adopt a character perturbation enhancement strategy for the class name label;

[0031] Step 2-2, adopt an image enhancement strategy for the image;

[0032] Step 2-3, input the enhanced fine-tuning instruction into the multi-modal large model f to obtain a response; and adopt LoRA (increase Chinese explanation) fine-tuning, compare the obtained response with the true value, calculate the loss, and update the adapter layer of the multi-modal large model in reverse.

[0033] LoRA, Low-Rank Adaptation, is an efficient parameter fine-tuning technique. It realizes the rapid customization of multi-modal large models by adding a small number of trainable parameters to the model while keeping the original model parameters unchanged.

[0034] As a preferred scheme of the present application, in step 2-3, LoRA fine-tuning is adopted, specifically, LoRA low-rank decomposition is used to update the adapter layer (i.e. adapter) of the multi-modal large model, and the parameters of the remaining visual encoder and language model part are fixed; a bypass is added beside the adapter, matrix A is initialized with a random Gaussian distribution, and matrix B is initialized with a zero matrix, and the adapter parameters W are fixed during training 0 , only matrix A and matrix B are trained; after training, the parameters are updated as:

[0035] W 0 +ΔW=W 0 +BA,

[0036] In the formula, ΔW represents the change amount of the parameter;

[0037] For input x, the forward propagation process of the model is updated as:

[0038] h=W 0 x+ΔWx=W 0 x+BAx.

[0039] In the formula, h represents the updated parameter.

[0040] As a preferred scheme of the present application, the enhancement strategy of character perturbation in step 2-1 includes but is not limited to the following:

[0041] Strategy 1, splitting and recombining, that is, splitting the label name according to a specific symbol, and then recombining the split parts in random order;

[0042] Strategy 2, character inversion, that is, randomly selecting several characters at the end of the label name and moving them to the beginning of the label name;

[0043] Strategy 3, random insertion, that is, randomly extracting characters from 0-9, a-z, A-Z, and inserting them into a random position in the label name;

[0044] Strategy 4, character mixed arrangement, that is, taking out all the characters in the label name, randomly shuffling and reordering them;

[0045] Strategy 5, character replacement, that is, randomly selecting some characters in the label name and randomly replacing them with any character in 0-9, a-z, A-Z;

[0046] Strategy 6, reverse order arrangement, that is, completely reversing the order of characters in the label name;

[0047] Strategy 7, character repetition, that is, randomly selecting some sub-strings in the label name and repeating the sub-strings at the end of the label name 1-2 times;

[0048] Strategy 8, character deletion, that is, randomly deleting some characters in the label name.

[0049] As a preferred scheme of the present application, the image enhancement strategy in step 2-2 includes but is not limited to the following:

[0050] cropping, randomly cropping the sample image;

[0051] rotation, randomly rotating the sample image by one of 90°, 180°, and 270°;

[0052] flipping, horizontally flipping and vertically flipping the sample image.

[0053] As a preferred scheme of the present application, the specific process of step 3 is as follows:

[0054] Step 3-1, input the test instructions in the instruction test data set into the multi-modal large model;

[0055] Step 3-2, taking the instruction fine-tuning set as a historical dialogue set, randomly selecting one to two historical dialogues therefrom, and inputting the historical dialogues into the "history" historical dialogue interface of the multi-modal large model;

[0056] Step 3-3, obtaining the output of the multi-modal large model as an initial classification result A1.

[0057] As a preferred scheme of the present application, the specific process of step 4 is as follows:

[0058] Step 4-1, adaptive attribute selection, specifically: asking the multi-modal large model which attributes of the sample will be focused on for the current data set, and then obtaining M attributes selected by the model in adaptation;

[0059] Step 4-2, automatic prompt generation, specifically: asking the multi-modal large model what prompt will be used to prompt the model to generate descriptions of the attributes for the current data set and the attributes that need to be focused on, and then obtaining the prompt automatically generated by the model;

[0060] Step 4-3, local attribute description generation, specifically: inputting each image sample in the test instruction into the visual end of the large model, and inputting the prompt automatically generated in step 4-2 into the text end of the large model, to obtain M attribute descriptions about each image sample;

[0061] Step 4-4, global description generation, specifically: put the image samples in the test instructions and their corresponding M attribute descriptions into the large model, and require the model to summarize the global description;

[0062] Step 4-5, attribute description induction, specifically: for each image sample, a total of M+1 attribute descriptions are obtained, which are saved in the form of a dictionary of "attribute description aspect: attribute description content".

[0063] As a preferred scheme of the present application, the prompt used for questioning in step 4-1 for the fine-grained dataset is:

[0064] "Which attributes of the fine-grained {bird / flower / dog / …} would you care about and briefly explain why",

[0065] For the general domain dataset, the prompt used for questioning is:

[0066] "Different from the fine-grained dataset, the general dataset does not have fixed and specific attribute features. Therefore, you need to find attribute feature aspects that can be widely adapted for the general dataset, which attributes would you pay attention to".

[0067] As a preferred scheme of the present application, the prompt used for questioning in step 4-2 is

[0068] "Please design prompts for these attributes ({attribute 1}, {attribute 2}, {attribute 3}, …), and the large model can accurately describe these attributes".

[0069] As a preferred scheme of the present application, the prompt used for questioning in step 4-3 is "<picture path> {attribute k prompt}".

[0070] As a preferred scheme of the present application, the prompt used for questioning in step 4-4 is

[0071] "<picture path> {attribute 1}: {attribute description 1}, {attribute 2}: {attribute description 2}…, please integrate the above attribute descriptions and give a global and overall description".

[0072] As a preferred scheme of the present application, the specific process of step 5 is:

[0073] Step 5-1, for each attribute, calculate the text similarity between the attribute description of the query image in the current instruction and the attribute description of the support image in the current instruction, and obtain a similarity vector;

[0074] Step 5-2, add the M+1 similarity vectors obtained in step 5-1 to obtain an aggregated similarity vector;

[0075] Step 5-3, sorting according to the similarity vector after aggregation, screening out the top N / / 2 similar candidate answers to form a new candidate answer set C N / / 2 .

[0076] As a preferred scheme of the application, the specific process of step 6 is:

[0077] If the initial classification answer A1 is in the new candidate answer set C N / / 2 , it is considered reliable, and is output as the final classification result.

[0078] As a preferred scheme of the application, the specific process of step 7 is:

[0079] Step 7-1, if the initial classification answer A1 is not in the new candidate answer set C N / / 2 , it is considered unreliable.

[0080] Step 7-2, reorganize the instructions, and replace the candidate answer set in the original instructions with C N / / 2 and the initial classification answer A1 as a new candidate answer set C N / / 2+1 , and delete the support samples of the categories not in C N / / 2+1 .

[0081] Step 7-3, re-input the newly organized instructions into the multi-modal large model, and the obtained response is taken as the final classification result.

[0082] Compared with the prior art, the application has the following technical effects:

[0083] The application proposes a multi-modal large model small sample classification method based on knowledge enhancement, which specifically includes two parts of enhancement methods.

[0084] The first part is an instruction fine-tuning enhancement method, which collects rich data sets in different fields to ensure good generalization of the multi-modal large model. At the same time, through the image-text enhancement strategy, the multi-modal large model can focus more on the examples provided by the support set in the instruction, so as to analyze and compare the relationship between the query sample and the example. In addition, through the meta-learning way to fine-tune the model, the multi-modal large model can learn to analyze and compare from multiple meta-tasks instead of imitating a fixed reasoning path and answer format. Compared with the multi-modal large model without instruction fine-tuning enhancement strategy, the classification accuracy is greatly improved.

[0085] The second part is an enhanced method based on reasoning screening. Through historical dialogues, a multi-modal large model can quickly understand the form and answer format of the current task, which helps the model to understand the task intent faster. The method of screening candidate answers based on attributes can use attribute description to calculate the text similarity as a prior classification result, filter out unreliable categories, and reduce the task complexity. In the re-reasoning, the self-consistency of the multi-modal large model can be further verified. Compared with the multi-modal large model without using reasoning enhancement strategy, the classification accuracy is greatly improved, and it is also suitable for large models that are not fine-tuned. BRIEF DESCRIPTION OF DRAWINGS

[0086] The application will be further described below with reference to the drawings:

[0087] Figure 1 is the algorithm flowchart of the multi-modal large model small sample classification method based on knowledge enhancement of the application;

[0088] Figure 2 is the instruction following data set diagram of the small sample task form under the meta-learning paradigm of the application;

[0089] Figure 3 is the instruction following data set diagram of the small sample task form under the meta-learning paradigm of the application;

[0090] Figure 4 is the method diagram of generating attribute description in the application;

[0091] Figure 5 is the model method diagram of the application. DETAILED DESCRIPTION

[0092] The technical solutions of the embodiments of the application will be described below in detail with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all embodiments. Based on the embodiments of the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the application.

[0093] Embodiment one:

[0094] Next, the related background knowledge related to the embodiments of the application will be introduced:

[0095] Contextual Learning: Contextual learning, also known as contextual learning, allows the model to learn and reason in a given context without any updates to the actual parameters of the model. This means that the model can adjust its behavior based on the input context information without changing its pre-training state. In contextual learning, the core advantage of the model lies in its context dependency. It reasons by analyzing the provided context information (such as examples and task descriptions) rather than relying on an explicit training process to adjust its parameters. In practical applications, the model reasons based on the provided context to generate responses related to the context. This process relies on the extensive knowledge learned by the model during pre-training, enabling it to generate reasonable outputs based on the current context. This approach not only improves the flexibility of the model but also reduces the resources and time required to retrain the model for specific tasks.

[0096] Division of base class and new class: For a data set, there is an image set X and a label set C, a graph-text pair is represented as (x, y), in the small sample experimental setting, it is usually divided into two parts, one is the base class set D base = {(x, y) | x ∈ C base , y ∈ C base} for fine-tuning, and the other is the new class set D novel = {(x, y) | x ∈ X novel , y ∈ C novel} for testing, the classes of the two sets are disjoint, that is

[0097] N-way K-shot: The fine-tuning stage samples tasks from the base class set D base , and the testing stage samples tasks from the new class set D novel For each N-way K-shot task, N classes are randomly sampled, and K samples are randomly sampled for each class to form the support set The remaining samples form the query set where M is the number of remaining samples of the N classes, the task of the model is to correctly classify the samples in the query set Q based on the information provided by the support set S.

[0098] Multi-modal large model structure: A multi-modal large model usually consists of a visual encoder (or an encoder for other modalities), an adapter layer, and a large language model. The visual encoder is used to process visual information and convert images into feature embeddings that the model can understand. The adapter layer converts the visual embeddings into a dimension that aligns with the text embeddings. The large language model is used to process text information and convert it into text embeddings that the model can understand, and output the final response.

[0099] Combination Figure 1As shown, the present application is based on a knowledge-enhanced multi-modal large model small sample classification method, which includes the following steps:

[0100] 1. Collect small sample wood image classification data set, organize N-way K-shot small sample task form instruction fine-tuning set and instruction test set on base class and new class, the specific process is as follows:

[0101] (1) Eight public small sample image classification data sets are used in this embodiment, including CIFAR-FS, Tiered-ImageNet, Mini-ImageNet three general field data sets and CUB, Flower102, FGVC, StanfordDogs, StanfordCars five fine-grained data sets about birds, flowers, aircraft, dogs and cars respectively, and are divided according to the preset classification categories of the collected data sets, wherein the new class is used for testing, and the base class is used for fine-tuning; In order to collect more abundant fine-tuning data set, GTSRB, Country211, EurosatClip, Resisc45_CLIP, Caltech-101, OxfordPets, Food101, Mnist, Fer2013 and other nine public image data sets are also used. Since there is no preset small sample base class / new class classification category, the base class and the new class are divided by random sampling in the ratio of 7:3, wherein only the base class is used for fine-tuning.

[0102] (2) Sample on the new class set D novel , construct the instruction in the form of N-way K-shot small sample task as shown below, form the instruction test data set, in this embodiment, the experimental setting is 5-way 1-shot:

[0103] Picture 1: <Picture 1> What is this? {Class name 1}

[0104] Picture 2: <Picture 2> What is this? {Class name 2}

[0105] …

[0106] Picture N: <Picture N> What is this? {Class name N}

[0107] Query picture: <Query picture> What is this? Please choose an answer from the list of candidate answers

[0108] [{Class name 1}, {Class name 2}, …, {Class name N}].

[0109] The expression used in this embodiment is English, as shown below:

[0110] Picture 1:<img_path 1>What is this?{classname 1}

[0111] Picture 2:<img_path 2>What is this?{classname 2}

[0112] …

[0113] PictureN:<img_path N>What is this?{classnameN}

[0114] Query Picture:<query_img_path>What is this?Output is one of[{classname 1},{classname2},…,{classnameN}].

[0115] (3) Randomly sample N categories in the dataset, sample K samples in each category to form the support set S; randomly select one category from the N categories and sample 1 sample in the non-support set as the query set Q; fill the support set picture path, picture label name, query picture path into the instructions described in 3.1 to form an instruction of a small sample task. In this embodiment, a 5-way 1-shot setting is applied, that is, 1 sample in each of 5 categories is randomly sampled to form a support set, and the remaining samples in one of the 5 categories are randomly selected as a query set;

[0116] (4) A 5-way 1-shot small sample task obtained from the above two steps is regarded as a meta task, and multiple meta tasks are sampled in the base class set D base of a dataset, and the meta tasks sampled from different datasets are shuffled and integrated into a file to obtain an instruction following the fine-tuning set, which is stored in a json file in a structured manner;

[0117] (5) The instructions following the test set are sampled in the new class set D novel in the same way as described above.

[0118] 2. For the instructions in the instruction fine-tuning set, the picture and class name label are enhanced and input into the multi-modal large model for fine-tuning. The specific process is as follows:

[0119] (1) Adopt a character perturbation enhancement strategy for the label. In this embodiment, one of the following methods is randomly adopted for each label name:

[0120] Split and recombine, split the tag name according to a certain symbol, and then recombine the split parts in a random order, such as "757-200" after split and recombination becomes "200-757";

[0121] Character reversal, randomly select a few characters at the end of the tag name and move them to the beginning of the tag name, such as "cellphone" after character reversal becomes "onecellph";

[0122] Random insertion, randomly select a number between 1 and 10 as the number of insertion characters, and randomly select characters from 0-9, a-z, A-Z, and insert them into random positions in the tag name, such as "lamp" after random insertion becomes "lZCa0md4p2";

[0123] Character mixing, take all the characters in the tag name, randomly shuffle and reorder, such as "camera" after character mixing becomes "aremca";

[0124] Character replacement, randomly select some characters in the tag name and randomly replace them with any character from 0-9, a-z, A-Z, such as "pyramid" after character replacement becomes "p1raM2D";

[0125] Reverse order, completely reverse the order of characters in the tag name, such as "kangaroo" after reverse order becomes "ooragnak";

[0126] Character repetition, randomly select some sub-strings in the tag name and repeat the sub-strings at the end of the tag name 1 to 2 times, such as "ferry" after character repetition becomes "ferrryrryrry";

[0127] Character deletion, randomly delete some characters in the tag name, such as "strawberry" after character deletion becomes "stabrry";

[0128] (2) Take image enhancement strategy for images, in this embodiment, randomly take several of the following ways for each image:

[0129] Crop, randomly crop the sample image, the crop area can be top left, bottom left, top right, bottom right, center, and the size can be 1 / 2 length and 1 / 2 width of the original image size;

[0130] Rotation, randomly rotate the sample image by 90°, 180°, or 270°;

[0131] Flip, flip the sample image, either horizontally or vertically;

[0132] (3) In this embodiment, the multi-modal large model used is Qwen-VL-Chat-Int4, the visual encoder and large language model are frozen, the Lora low-rank decomposition adapter layer is used, the learning rate is set to 1x10 -5 , the warm-up ratio is 0.01, the Adam optimizer is used, the weight decay hyperparameter β1 is 0.1, and β2 is 0.95;

[0133] (4) The process of Lora low-rank decomposition is to add a bypass to the adapter, initialize matrix A with a random Gaussian distribution, initialize matrix B with a zero matrix, and fix the adapter parameters W 0 only train matrix A and matrix B; after training, the parameters are updated as:

[0134] W 0 +ΔW=W 0 +BA,

[0135] For input x, the forward propagation process of the model is updated as:

[0136] h=W 0 x+ΔWx=W 0 x+BAx;

[0137] (5) The process of instruction fine-tuning is to input the instructions in the instruction fine-tuning dataset into the multi-modal large model, the model generates output according to the instructions, compares with the actual true value, updates the adapter layer parameters with lora, and finally obtains the fine-tuned model parameters.

[0138] 3. In the inference stage, a plurality of historical dialogues are randomly selected and input into the multi-modal large model together with the instructions in the test set to obtain the initial classification answer A1 of the model, the specific process is as follows:

[0139] (1) In this embodiment, the multi-modal large model used is Qwen-VL-Chat-Int4, and the key code for receiving input and output of the multi-modal large model is as follows:

[0140] response,_=model.chat(tokenizer,query=input_text,history=history,append_history=False)

[0141] The instructions in the instruction test set are input as input text input_text into the query query of the multi-modal large model;

[0142] (2) At the same time, a historical dialogue in the historical dialogue set, i.e., the instruction test set, is randomly selected and sent to the interface of the multi-modal large model "history";

[0143] (3) The response reponse of the multi-modal large model is obtained as the initial classification answer A1 of the model.

[0144] 4. For each image in the test instruction, prompt the multi-modal large model to generate multiple attribute descriptions, combined with Figure 2 , the specific process is as follows:

[0145] (1) Adaptive attribute selection, ask the multi-modal large model which attributes of the current data set will focus on, and get M attributes selected by the model, in this embodiment, M is set to 4, taking the fine-grained bird data set CUB as an example, ask and get the response as follows:

[0146]

[0147] For the fine-grained flower data set Flower102, the attributes obtained are petal shape and size, color and pattern, calyx shape and size, structure and arrangement;

[0148] For the fine-grained aircraft data set FGVC, the attributes obtained are airfoil, fuselage design, engine configuration, and tail configuration;

[0149] For the fine-grained dog data set StanfordDogs, the attributes obtained are shape and size, hair length and type, color and pattern, head or facial features;

[0150] For the fine-grained car data set StanfordCars, the attributes obtained are front design, wheel design, body details and decoration, body shape and size;

[0151] For the general domain data set, the prompt and the response attribute obtained by asking are general:

[0152] User: "Unlike fine-grained data sets, general data sets do not have fixed, specific feature attributes. Therefore, you need to find attribute features that can be widely adapted for general data sets. Which attributes will you focus on?"

[0153] Multi-modal large model: "Color and texture: provide a detailed analysis of the color scheme of the image, mention the main colors and any notable contrasts or gradients; Shape: identify and describe the geometric shapes present in the image; Activity: analyze any dynamic elements or actions depicted in the image; Scene: describe the scene or environment captured in the image."

[0154] (2) Automatic prompt generation, the multimodal large model asks what kind of prompt to use to prompt the model to generate descriptions of the current data set and the attributes that need to be focused on. The model automatically generates the prompt, and the prompt used in the question is "Please design a prompt for these attributes ({attribute 1}, {attribute 2}, {attribute 3}, …) so that the model can generate good image attribute descriptions based on the prompt."

[0155] Taking the fine-grained bird dataset CUB as an example, the prompt for the question and the response obtained are as follows:

[0156] User: "Design a prompt for the attributes in step 1 so that the model can generate good image attribute descriptions based on the prompt."

[0157] Multimodal large model: "Color distribution: Describe the dominant colors and color distribution of each part of the bird's body, such as head, back, abdomen, wings, and tail."

[0158] Shape and size: Describe the size and shape of the bird in the picture in detail, including the shape of its head, beak, wings, tail, and legs.

[0159] Feather characteristics: Describe the feather type and structure of the bird shown in the picture from the perspective of feather characteristics, including the length, texture, and arrangement of the feathers.

[0160] Behavioral characteristics: Describe the behavior and posture of the bird shown in the picture from the perspective of behavioral characteristics, such as standing, flying, foraging, or other activities."

[0161] (3) Local attribute description generation, send each image sample in the instruction to the large model visual end, and send the automatically generated prompt to the large model text end, to obtain M attribute descriptions for each image sample. The prompt used in the question is "<picture path> {attribute k prompt}",

[0162] Taking the fine-grained bird dataset CUB as an example, the prompt for the question is as follows:

[0163] User: "<picture path> For the object in the picture, please describe the dominant colors and color distribution of each part of the bird's body, such as head, back, abdomen, wings, and tail."

[0164] Fill in the prompts for other attributes in the same way. For each attribute, the multimodal large model's attribute description is as follows:

[0165] Multimodal large model: "Color distribution: This bird has a blue head, green back, red belly, and red chest. The wings are green and black. The tail is black and green."

[0166] Shape and size: It is small and bright, it has a blue head, a red throat, a green back, and black, red, and green wings. Its tail is black, and its legs are gray. The bird's beak is small and also gray.

[0167] Feather characteristics: It has colorful feathers, with blue, green, red, and orange. The feathers are long and soft in texture. Their arrangement allows the bird to fly and maneuver in the air.

[0168] Behavioral characteristics: It perches on a branch with its head to the right. It seems to be looking at something in the distance, possibly scanning for potential threats or food.

[0169] (4) Global description generation, input image samples and M attribute descriptions into a large model, require the model to summarize the global description, the prompt for the question is "<picture path> {attribute 1}: {attribute description 1}, {attribute 2}: {attribute description 2} …, please integrate the above attribute descriptions and give a global and overall description",

[0170] Taking the fine-grained bird dataset CUB as an example, the prompt for the question is as follows:

[0171] User: "<picture path> Color distribution: this bird …; Shape and size: it is …; Feather characteristics: it has colorful feathers …; Behavioral characteristics: it perches on a branch …, please integrate the above attribute descriptions and give a global and overall description."

[0172] Multimodal large model: "Global description: this bird is a multi-colored bird with green, blue, red, and orange color patterns. It has a blue head, a green back, and a red belly. The wings are green with brown stripes. The bird is perched on a branch. This bird is small and compact in size, with a short tail and short legs. The feathers are long and fine in texture, looking soft and fluffy. The behavior of this bird is to sit still, with its eyes open, staring at something in front of it."

[0173] (5) Attribute description induction, for each image sample, a total of M+1 attribute descriptions are obtained, which are saved in the format of "attribute description aspect: attribute description content" dictionary, the saved json file format is as follows:

[0174] <picture path>: {

[0175] "<attribute 1>": "<attribute description 1>",

[0176] ...

[0177] "<attribute M>": "<attribute description M>",

[0178] "Global description": "<global description>"

[0179] },

[0180] Take the fine-grained bird dataset CUB as an example, the format of the saved json file is as follows:

[0181] <image_path>: {

[0182] "color distribution": "This bird has a blue head…";

[0183] "shape and size": "It is small and bright…";

[0184] "feather features": "It has colorful feathers…";

[0185] "behavior characteristics": "It inhabits in the branches…";

[0186] "global description": "This bird is a multi-colored bird…"

[0187] }。

[0188] 5、Calculate the attribute description similarity between image samples and sort them out, and select N / / 2 reliable candidate answers to form a new candidate answer set C N / / 2 , The specific process is as follows:

[0189] (1) For the first attribute, calculate the similarity of the attribute description of the five support samples and the attribute description of the query sample. In this embodiment, the text encoder used is the frozen model SBERT (all-MiniLM-L6-v2) to measure the text similarity, and the text attribute similarity is obtained

[0190] (2) For the four attribute descriptions and one global description, S attr1 , S attr2 , S attr3 , S attr4 , S global can be obtained.

[0191] (3) Add the obtained similarity vectors to obtain the aggregated text similarity

[0192] S aggr =S attr1 +…+S attr4 +S global ;

[0193] (4) According to S aggr , rank the five candidate answers, and the top N / / 2 similar candidate answers, that is, the two candidate answers with the highest similarity, are selected as the new candidate answer set C N / / 2 .

[0194] 6、For the initial classification answer A1 and the new candidate answer set C N / / 2 , judge whether A1 is reliable, if reliable, output the answer as the final classification result directly, the specific process is as follows:

[0195] (1) If the initial classification answer A1 appears in the new candidate answer set C N / / 2 , it is considered reliable;

[0196] (2) Output the initial classification answer A1 as the final answer.

[0197] 7、If A1 is not reliable, use C N / / 2 to reorganize new instructions and input multimodal large model for re-inference to get the final classification result, the specific process is as follows:

[0198] (1) If the initial classification answer A1 does not appear in the new candidate answer set C N / / 2 , it is considered unreliable;

[0199] (2) Reorganize the instructions, take C N / / 2 and the initial classification answer A1 as the new candidate answer set C N / / 2+1 to replace the candidate answer set in the original instruction, and delete the support samples of the categories not in C N / / 2+1 , taking the fine-grained bird data set CUB as an example, the original test instruction is:

[0200] Picture 1:<img_path 1>What is this?Fox Sparrow.

[0201] Picture2:<img_path 2>What is this?Common Tern.

[0202] Picture 3:<img_path 3>What is this?White crownedsparrow.

[0203] Picture 4:<img_path 4>What is this?Paintedbunting.

[0204] Picture 5:<img_path 5>What is this?Yellow breastedchat.

[0205] Query Picture: <query_img_path> What is this?Output is one of [Fox Sparrow, Common Tern, White crowned sparrow, Painted bunting, Yellowbreasted chat].

[0206] The initial classification answer A1 of the model is "Fox Sparrow", and after the reasoning enhancement method based on attribute screening, the new candidate answer set C N / / 2 is { "White crowned sparrow", "Common Tern"}. Since A1 does not appear in C N / / 2 , it is considered unreliable, and C N / / 2 and the initial classification answer A1 are replaced as the new candidate answer set C N / / 2+1 in the original instruction, and the support samples of the categories not in CN / / 2+1 are deleted, and the new test instruction

[0207] is:

[0208] Picture 1: <img_path 1> What is this?White crowned sparrow.

[0209] Picture2: <img_path 2> What is this?Common Tern.

[0210] Picture 3: <img_path 3> What is this?Fox Sparrow.

[0211] Query Picture: <query_img_path> What is this?Output is one of [White crowned sparrow, Common Tern, Fox Sparrow].

[0212] (3) Input the new test instruction into the large model to obtain the response. If it is inconsistent with the initial classification answer A1, the sample order in the test instruction can be shuffled to input the model to obtain the response again. The majority vote of the response results of multiple rounds is selected, that is, the answer with the most occurrences is selected. In this embodiment, for efficiency, the result of the first round of the large model after inputting the new test instruction is regarded as the final classification result.

[0213] Specifically, the present application will be further described by the following embodiments:

[0214] A, Experimental conditions

[0215] In this embodiment, the experimental environment is GPU NVIDIA GeForce RTX 3080, Python 3.9, torch2.0.0+cu118, torchvision 0.15.1+cu118, and the experimental setting is 5-way 1-shot. The model used in the experiment is Qwen-VL-Chat-Int4 (abbreviated as Qwen-VL).

[0216] B, Evaluation criteria for experimental results

[0217] The classification accuracy is used as the evaluation criterion, and the higher the accuracy, the better the model performance.

[0218] Since the output of the un-fine-tuned multi-modal large model is usually long and contains redundant information that cannot be directly matched and compared with the true value, three measurement methods are designed to calculate the classification accuracy:

[0219] Acc: Regular expression is used to filter non-alphanumeric symbols in the model response and the true value, and the model output is strictly matched with the true value. If it is completely matched, it is considered correct.

[0220] Acc ocour : Regular expression is used to filter non-alphanumeric symbols in the model response and the true value. If the true value appears anywhere in the model response, it is considered correct.

[0221] Acc clip : The response of the model is mapped to the candidate answer set using CLIP-L 14, i.e. calculating the text similarity between the model response and the candidate answers, and then strictly matching the result with the true value.

[0222] The experimental results of this embodiment use Acc clip as the evaluation criterion by default.

[0223] C, Comparative analysis of experiments

[0224] The performance of the multi-modal large model Qwen-VL without fine-tuning on the eight data sets is given, and the performance of the multi-modal large model Qwen-VL to which the method of the application is applied and the improvement are given.

[0225] Table 1 Comparison of performance of general-purpose small sample image classification methods on four data sets

[0226] Method CIFAR Tiered MINI CUB BAVARDAGE 87.35 85.20 84.80 90.42 EASY3xResNet12 87.16 84.29 84.04 90.56 PEMnE-BMS 88.44 86.07 85.54 94.78 PTMAP-SF-SOT 89.94 ]]> ​ - 85.59 95.80 ]]> ​ P>M>F 84.30 - 95.30 - CAML 70.80 95.40 ]]> ​ 96.20 ]]> ​ 91.80 Qwen-VL 37.04 57.14 39.38 45.00 Inventive method (fine-tuning augmentation) 95.02 98.06 98.24 96.40

[0227] Table 2 Comparison of performance of fine-grained small sample image classification methods on four fine-grained data sets

[0228]

[0229]

[0230] The present embodiment performs experiments on 3 coarse-grained and 5 fine-grained data sets, and compares the results with the previous most advanced method. The comparison results of the present embodiment with the current leading small sample image classification method in Table 1 and Table 2 show that the small sample image classification effect of the method of the present embodiment is better than that of other compared methods, which shows the effectiveness of the method of the present embodiment in the general-purpose field and the fine-grained field.

[0231] In order to verify the main contribution of the model proposed in the present embodiment, a series of ablation experiments are performed. The influence of instruction fine-tuning and label, image enhancement module is mainly concerned. The results of these ablation studies give an explanation of the basic role of each method in the final performance of the model, as shown in Table 3.

[0232] Table 3 Influence of each module on small sample image classification method on eight data sets

[0233]

[0234] The present embodiment also specifically gives the average revenue of different modules to the small sample classification of the model, as shown in Table 4.

[0235] Table 4 Average revenue of each module to small sample image classification on eight data sets

[0236] Instruction fine-tuning Fine-tuning augmentation Inference augmentation Average performance Average gain × × × 43.12 - √ × × 96.39 +53.27 √ √ × 97.29 +54.17 × × √ 68.81 +25.69 √ × √ 96.71 +53.59 √ √ √ 97.45 +54.33

[0237] As shown in Table 3, Table 4, when no fine-tuning strategy and enhancement module are used, the performance is poor, and the accuracy of the model on the remaining seven data sets is less than 50% except for the Tiered data set; when the instruction fine-tuning is used, the accuracy is significantly improved, but it is not optimal; when the instruction fine-tuning and fine-tuning enhancement are used at the same time, better results can be obtained. For the un-tuned multi-modal large model, directly applying the inference enhancement module, the model performance is significantly improved; combined with instruction fine-tuning and fine-tuning enhancement, the model performance is optimal.

[0238] At the same time, in order to verify the reliability of the generated attribute description, the text similarity between the attribute description of the query sample and the corresponding attribute description of the support sample is directly calculated, and the option with the maximum similarity is taken as the classification result to test the effectiveness of the attribute description. The accuracy of the four attribute descriptions and the global description is reported in Table 5, and the average accuracy of the above five attribute descriptions, the top1 to top4 accuracy of the aggregated attribute description is reported in Table 6.

[0239] Table 5 Accuracy of attribute description generated by the method of the present application

[0240]

[0241]

[0242] Table 6 Accuracy of the aggregated description of the method of the present application

[0243] Dataset 5 description average Aggregate top1 Aggregate top2 Aggregate top3 Aggregate top4 CIFAR 65.03 77.40 89.92 95.70 98.26 Tiered 73.76 82.66 95.12 98.78 99.84 MINI 66.75 76.58 89.34 95.56 98.76 CUB 47.58 66.62 84.82 93.46 97.56 Flowers 55.03 72.64 87.76 94.72 98.32 FGVC 34.94 43.86 66.10 80.98 92.20 Dogs 35.82 50.50 71.38 83.42 92.98 Cars 61.14 88.74 95.46 98.00 99.40

[0244] As shown in the above table, using attribute description alone to assist classification is not reliable, especially on several fine-grained data sets, because fine-grained data sets require more discriminative attribute descriptions. Compared with the accuracy of the single attribute description, the aggregated description exceeds the average accuracy of the five single descriptions in top1 accuracy. This is reasonable, based on the true value picture in the query picture and the support picture, it is not necessarily the most similar in some attributes, but it has a greater probability of being the least similar. Therefore, the aggregated attribute has good accuracy, and even the top2 accuracy on most data sets is more than 84%, and the top3 accuracy is even more than 93%. For the purpose of efficiency and reducing the complexity of the task, the range of reliable candidate answers selected by the embodiment for the 5-way 1-shot experimental setting is top2.

[0245] The above embodiments only illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made on the basis of the technical idea of the present application and technical scheme falls within the protection scope of the present application.

Claims

1. A knowledge enhancement-based multi-modal large model small sample classification method, characterized in that, Comprising the following steps: Step 1, collect a small sample image classification data set, organize instruction fine-tuning set in the form of N-way K-shot small sample task on base class, organize instruction test set in the form of N-way K-shot small sample task on new class; Step 2, apply instruction application strategy to enhance the picture and class name label in the instruction fine-tuning set, and input into the multi-modal large model fine-tuning; Step 3, in the inference stage, randomly select multiple historical dialogues, input the multi-modal large model together with the instruction in the instruction test set, and obtain the initial classification answer A1 of the model; Step 4, at the same time, for each image in the instruction test instruction, prompt the multi-modal large model to generate multiple attribute descriptions; Step 5, calculate the attribute description similarity between the image samples obtained in step 4 and sort them, and screen out N / / 2 reliable candidate answers to form a new candidate answer set C N / / 2 ; Step 6, for the initial classification answer A1 and the new candidate answer set C N / / 2 , determine whether A1 is reliable, and if so, directly output the answer as the final classification result; Step 7, if A1 is not reliable, use C N / / 2 Reorganize new instructions, input multimodal large model for re-inference, and get final classification result; N is the number of categories sampled in the small sample image data set.

2. The knowledge enhancement-based multi-modal large model small sample classification method according to claim 1, characterized in that, The specific process of step 1 is: Step 1-1, collect small sample image datasets of different granularities from different scenes and fields and their preset classification categories, if the small sample image datasets have no preset classification categories, then randomly sample all categories in the small sample image datasets according to a ratio of 7:3 to obtain a base class set D base and a new class set D novel ; Step 1-2, construct instruction in the form of small sample task: Step 1-3, randomly sample N categories in the small sample image data set, sample K samples in each category to form a support set S; randomly select a category from the N categories and sample 1 sample in the non-support set as a query set Q; Fill the support set picture path, picture label name, query picture path into the instruction in step 1-2 to form an instruction of a small sample task; Step 1-4, the task of a small sample obtained through the step 1-2 and the step 1-3 is regarded as a meta task, and a base class set D of a small sample picture data set base A plurality of meta tasks are sampled in the middle of the set D, the meta tasks sampled from different small sample picture data sets are shuffled and integrated in a file, and a command fine-tuning set is obtained. Step 1-5, sampling multiple meta-tasks in a new class set D of a small sample picture dataset novel sampling multiple meta-tasks in a new class set D of a small sample picture dataset, shuffling the order of the meta-tasks sampled from different small sample picture datasets and integrating them in one file to obtain an instruction test set.

3. The knowledge enhancement-based multi-modal large model small sample classification method according to claim 2, characterized in that, The instruction format of the small sample task in step 1-2 is: Picture 1: <Picture 1> What is this? {Class name 1} Picture 2: <Picture 2> What is this? {Class name 2} … Picture N: <Picture N> What is this? {Class name N} Query picture: <Query picture> What is this? Please select an answer from the candidate answer list: [{Class name 1}, {Class name 2}, …, {Class name N}].

4. The knowledge enhancement-based multi-modal large model small sample classification method according to claim 1, characterized in that, The specific process of step 2 is: Step 2-1, adopt character perturbation enhancement strategy for class name label; Step 2-2, adopt image enhancement strategy for image; Step 2-3, input the enhanced fine-tuning instruction into the multi-modal large model f to obtain the response; And adopt LoRA fine-tuning, compare the obtained response with the true value, calculate the loss, and update the adapter layer of the multi-modal large model in reverse.

5. The knowledge enhancement-based multi-modal large model small sample classification method according to claim 4, characterized in that, In steps 2-3, LoRA fine-tuning is adopted, specifically, LoRA low-rank decomposition is used to update the adapter of the multi-modal large model, and the parameters of the remaining visual encoder and language model part are fixed; a bypass is added beside the adapter, matrix A is initialized with a random Gaussian distribution, and matrix B is initialized with a zero matrix, and the adapter parameters W are fixed during training 0 Only matrix A and matrix B are trained; after training is completed, the parameters are updated as follows: W 0 + ΔW = W 0 + BA, In the formula, ΔW represents the change amount of the parameter; For input x, the forward propagation process of the model is updated as: h = W 0 x + ΔWx = W 0 x + BAx.

6. The knowledge enhancement based multi-modal large model small sample classification method according to claim 1, characterized in that, The specific process of step 3 is: Step 3-1, input the test instruction in the instruction test data set into the multi-modal large model; Step 3-2, select one or two historical dialogues from the instruction fine-tuning set as the historical dialogue set, and input into the "history" historical dialogue interface of the multi-modal large model; Step 3-3, obtain the output of the multi-modal large model as the initial classification result A1.

7. The knowledge enhancement based multi-modal large model small sample classification method according to claim 1, characterized in that, The specific process of step 4 is: Step 4-1, adaptive attribute selection, specifically: ask the multi-modal large model, which attributes of the sample will be focused on for the current data set, and then obtain M attributes selected by the model adaptively; Step 4-2, automatic prompt generation, specifically: ask the multi-modal large model, for the current data set and the attributes that need to be focused on, what kind of prompt will be used to prompt the model to generate descriptions of these attributes, and then get the model automatically generated prompt; Step 4-3, local attribute description generation, specifically: send each image sample in the test instruction to the large model visual end, and send the automatically generated prompt in step 4-2 to the large model text end, to get M attribute descriptions about each image sample; Step 4-4, global description generation, specifically: send the image samples in the test instruction and their corresponding M attribute descriptions to the large model, and require the model to summarize the global description; Step 4-5, attribute description induction, specifically: for each image sample, a total of M+1 attribute descriptions are obtained, which are saved in the dictionary format of "attribute description aspect: attribute description content".

8. The knowledge enhancement based multi-modal large model small sample classification method according to claim 1, characterized in that, The specific process of step 5 is: Step 5-1, for each attribute, calculate the text similarity of the attribute description of the query image in the current instruction and the attribute description of the support image in the current instruction, to get a similarity vector; Step 5-2, add the M+1 similarity vectors obtained in step 5-1 to get an aggregated similarity vector; Step 5-3, ranking according to the similarity vectors after the aggregation, screening out the top N / / 2 high similarity candidate answers to form a new candidate answer set C N / / 2 .

9. The knowledge enhancement based multi-modal large model small sample classification method according to claim 1, characterized in that, The specific process of step 6 is: if the initial classification answer A1 is in the new candidate answer set C N / / 2 , it is considered reliable, and is output as the final classification result.

10. The knowledge enhancement based multi-modal large model small sample classification method according to claim 1, characterized in that, The specific process of step 7 is: Step 7-1, if the initial classification answer Al is not in the new candidate answer set C N / / 2 then it is considered unreliable; Step 7-2, reorganize the instructions, replace the candidate answer set C N / / 2 and the initial classification answer Al with the new candidate answer set C N / / 2+1 Replace the candidate answer set in the original instruction with C N / / 2+1 and delete the support samples of the categories not in C Step 7-3, re-input the newly organized instruction into the multi-modal large model, and get the response as the final classification result.

Images