Image question and answer data acquisition method based on visual language large model
By optimizing the image question-answering data acquisition method of the visual language large model, and using image and prompt embedding to generate question-answer data pairs, combined with cross-entropy loss and instruction evaluator, the problems of low generation performance and high computational burden in the existing technology are solved, and efficient and accurate multi-turn visual dialogue data generation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CENT SOUTH UNIV
- Filing Date
- 2024-01-29
- Publication Date
- 2026-07-24
AI Technical Summary
Existing large visual language models rely on image descriptions when generating multimodal instruction following data, which limits the range of instruction content, resulting in low generation performance, low efficiency, and insufficient accuracy. Furthermore, existing methods require running the large language model twice, increasing the computational burden.
By acquiring image and prompt embeddings, question-answer data pairs are generated using a large language model. The model is then optimized using a cross-entropy loss function and an instruction evaluator. By combining an image encoder and a large language model, the instruction evaluator is further optimized to achieve multi-turn visual dialogue and self-evaluation, reducing computational burden and improving generation efficiency and accuracy.
It improves the performance and efficiency of visual language large models in generating multimodal question-answering data, reduces the computational burden, enhances the accuracy of generation, and is able to generate richer multi-turn dialogue data.
Smart Images
Figure CN117972053B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method for acquiring image question-answering data based on a large visual language model. Background Technology
[0002] Large Vision-Language Models (LVLMs) have made significant progress in the field of multimodal visual language. The core idea is to integrate a visual encoder into Large Language Models (LLMs). Leveraging the superior language understanding capabilities of LLMs, LVLMs can perform tasks such as visual dialogue, visual localization, inference segmentation, image generation, and arbitrary modal dialogue. One of the core technologies for achieving these tasks is visual instruction fine-tuning, which is mainly achieved by creating question-and-answer samples that follow instructions to fine-tune LVLMs.
[0003] Current research primarily highlights the importance of multimodal visual language instruction following data. However, manually annotating large-scale, high-quality multimodal instruction following data presents significant challenges. Generating instruction following samples using machines like GPT-4 is costly, and GPT-4's instruction data generation relies on image descriptions, making it unable to create question-and-answer data not included in the descriptions.
[0004] Current research utilizes LVLMs to generate large-scale, high-quality multimodal instruction-following data. These methods primarily focus on generating single-turn visual dialogue datasets, limiting the variety and complexity of the generated instructions. Furthermore, unlike LLMs, available LVLMs lack comprehensive functionality; existing LVLM methods often generate inappropriate responses or incorrect information, including fantasies. To address these issues, the Visual Instruction Generation and Correction (VIGC) method establishes a visual instruction correction module to regenerate responses. However, this introduces a computational burden because it requires running the large language model twice.
[0005] In summary, most current methods for fine-tuning large-scale visual language models heavily rely on instruction data, and the construction of most instruction data depends heavily on image descriptions. However, image descriptions limit the scope of instruction content. Existing models suffer from low performance and inefficiency in image-based instruction generation. Summary of the Invention
[0006] The purpose of this invention is to provide a method for acquiring image question-answering data based on a large visual language model that improves performance, increases efficiency, and enhances accuracy.
[0007] The image question-answering data acquisition method based on a large visual language model provided by this invention includes the following steps:
[0008] S1. Obtain the image dataset and encode the images in the image dataset to obtain the image embeddings;
[0009] S2. Encode the set prompt data according to the set format to obtain the corresponding prompt embedding;
[0010] S3. Using the image embedding obtained in step S1 and the prompt embedding obtained in step S2, the question-answer data pairs and corresponding prediction probability vectors are obtained through the large language model;
[0011] S4. Using the question-and-answer data pairs and corresponding prediction probability vectors obtained in step S3, obtain the evaluation score of the question-and-answer data pairs through the instruction evaluator;
[0012] S5. Using the question-and-answer data pairs and real question-and-answer data pairs obtained in step S3, optimize the large language model by calculating the cross-entropy loss function;
[0013] S6. Optimize the instruction evaluator using the evaluation scores of the question-answer data pairs obtained in step S4 and the cross-entropy loss function calculated in step S5;
[0014] S7. Using an image dataset without question-answering data annotation, obtain the image question-answering data of the annotation results through the large language model optimized in step S5 and the instruction evaluator optimized in step S6;
[0015] Step S1, which involves obtaining an image dataset and encoding the images in the dataset to obtain image embeddings, specifically includes:
[0016] The image dataset consists of raw images I and visual instruction data T;
[0017] An image encoder model is used to encode the original image I to obtain the corresponding image embedding X. I ;
[0018] The visual instruction data T consists of real question-and-answer data pairs, used to calculate the cross-entropy loss function;
[0019] Step S2, which involves encoding the set prompt data according to a set format to obtain the corresponding prompt embedding, specifically includes:
[0020] Use the following configuration prompt data:
[0021] “Generate multiple question-answer pairs based on the image.Requirement:The answer to each question must be found within the image, and the format is as follows:Question:…[\n]Answer:…[\n]”;
[0022] The specified prompt data is encoded according to the set format to obtain the prompt embedded X. prompt ;
[0023] Step S3, using the image embedding obtained in step S1 and the prompt embedding obtained in step S2, employs a large language model to obtain question-answer data pairs and corresponding prediction probability vectors, specifically including:
[0024] For image embedding X I And prompts embedded X prompt The data is spliced together using a large language model Φ. L Perform large-scale language analysis to obtain predicted question-answer data pairs and corresponding predicted probability vectors;
[0025] The following formula represents the predicted probability vector v corresponding to the hint in each question-and-answer data pair in the output:
[0026] [X C ,v]=Φ L ([X I ,X prompt ])
[0027] Among them, X C Dialogue data representing the generated questions and their corresponding answers;
[0028] Step S4, which uses the question-and-answer data pairs and corresponding predicted probability vectors obtained in step S3, to obtain evaluation scores for the question-and-answer data pairs through an instruction evaluator, specifically includes:
[0029] Select the predicted probability vector v, and obtain the corresponding evaluation scores for the question-and-answer data pairs through the instruction evaluator Ψ. The evaluation process is represented by the following formula:
[0030] s = Ψ(v)
[0031] Where s represents the output of the instruction evaluator as the evaluation score of the question-answer data pair;
[0032] The instruction evaluator Ψ is a fully connected linear layer, directly added to the large language model Φ. LAfter the output layer, the large language model Φ L The output of is used as the input to the instruction evaluator Ψ;
[0033] A fully connected linear layer is a single layer of a fully connected network.
[0034] Step S5, which involves using the question-and-answer data pairs and real question-and-answer data pairs obtained in step S3 to optimize the large language model by calculating the cross-entropy loss function, specifically includes:
[0035] The cross-entropy loss function is calculated using the predicted question-and-answer data pairs obtained in step S3 and the actual question-and-answer data pairs.
[0036] Among them, the real question-and-answer data pair is the visual instruction data T obtained in step S1;
[0037] During model training, the loss is calculated based on the output question-and-answer data and the real question-and-answer data, and the model is optimized based on the loss results.
[0038] The following formula represents the generation of target dialogue data X. C The probability of:
[0039]
[0040] Where p(X) C |X I ,X prompt ) represents the product of the probabilities of all cues corresponding to the dialogue data generated by the large language model; p θ (·) represents the probability of one of the prompts generated by the large language model; x i The model provides real-time prediction prompts during execution; L represents the length of the output question-and-answer dialogue; θ represents the trainable parameters; X C,＜i x represents i All previous dialogue prompts;
[0041] Using the generated target dialogue data X C The probability p(X) C |X I ,X prompt ), determine the labels of the predicted data l m Simultaneously, generate target dialogue data X. C The probability p(X) C |X I ,X prompt It also serves as input to the instruction evaluator;
[0042] By calculating the labels of dialogue data prompts predicted by a large language model and the labels of actual dialogue data prompts... nThe cross-entropy loss between the two sides is used to complete the optimization of the model;
[0043] The cross-entropy loss is calculated using the following formula:
[0044]
[0045] Among them, L MT-VCG Represents cross-entropy loss; This represents the probability of the true label corresponding to the i-th token predicted by the large language model; represents the true label corresponding to the i-th Token; L represents the cross-entropy loss value between the model prediction result and the true result;
[0046] During the training of the large language model, the cross-entropy loss L corresponding to each prediction result is calculated. MT-VCG And backpropagation, updating the large language model Φ L The parameters are adjusted until the set conditions are met to obtain the optimal model;
[0047] Step S6, which involves optimizing the instruction evaluator using the evaluation scores of the question-answer data pairs obtained in step S4 and the cross-entropy loss function calculated in step S5, specifically includes:
[0048] The loss of the instruction evaluator is expressed by the following formula:
[0049] L VISE =L1(tanh(L MT-VCG ),1-s)
[0050] Among them, L VISE L1 represents the loss value of the instruction evaluator; L1 represents the absolute value loss function, calculated as follows:
[0051] L1=|L MT-VCG -(1-s)|
[0052] tanh(L MT-VCG ) represents the autoregressive loss value of each question-answer pair after activation using the tanh(·) function;
[0053] Based on the calculated L VISE To achieve backpropagation processing;
[0054] Calculate L VISE The derivative of the evaluation score s is used to update the parameters of the evaluator, thereby reducing the loss L. VISE The value;
[0055] Step S7 involves using an image dataset labeled without question-answering data. Through the optimized large language model from step S5 and the optimized instruction evaluator from step S6, image question-answering data with annotation results is obtained. Specifically, this includes:
[0056] An image question-answering data acquisition model is constructed using the optimized large language model in step S5 and the optimized instruction evaluator in step S6.
[0057] The image dataset without question-answering data annotation is input into the optimized large language model in step S5, and the dialogue data and corresponding probability vectors are output simultaneously; the output probability vectors are input into the optimized instruction evaluator in step S6, and the score of the dialogue data is output simultaneously.
[0058] By constructing an image question-answering data acquisition model, and using an image dataset without question-answering data annotation, the image question-answering data acquisition and processing is completed, and the labeled image question-answering data is obtained.
[0059] The labeled image question-answering data includes dialogue data output by the optimized large language model and scores output by the optimized instruction evaluator;
[0060] The method of the present invention further includes the following steps:
[0061] 1) Obtain the open-source LLaVA multimodal large language model and perform fine-tuning on the open-source LLaVA multimodal large language model;
[0062] 2) Optimize the open-source LLaVA multimodal large language model after fine-tuning in step 1) to obtain the final LLaVA multimodal large language model;
[0063] 3) Using the final LLaVA multimodal large language model obtained in step 2), a question-answering test is conducted on images to verify the effectiveness of the image question-answering data acquisition model;
[0064] Step 1) involves obtaining the open-source LLaVA multimodal language model and fine-tuning it, specifically including:
[0065] Fine-tuning includes the fully connected layer between the image encoder and the large language model in the open-source LLaVA multimodal large language model, and the large language model within the open-source LLaVA multimodal large language model;
[0066] Fine-tuning parameters includes all parameters in the fully connected layers and the large language model;
[0067] The image encoder in the open-source LLaVA multimodal large language model is used for image encoding processing;
[0068] The image encoder and the large language model are connected through a fully connected layer network;
[0069] During the fine-tuning process, the input data of the open-source LLaVA multimodal large language model are the prompt codes corresponding to the image and the prompt codes corresponding to the question, and the output data is the answer to the corresponding question;
[0070] Step 2) involves optimizing the open-source LLaVA multimodal language model after fine-tuning in Step 1) to obtain the final LLaVA multimodal language model. This specifically includes:
[0071] Optimizations include the fully connected layer between the image encoder and the large language model in the open-source LLaVA multimodal large language model, and the large language model within the open-source LLaVA multimodal large language model.
[0072] The optimization parameters include all parameters in the fully connected layer and the large language model;
[0073] Using the output data from step 1) and the image question-and-answer data from the annotation results obtained in step S7, calculate the loss function and optimize the open-source LLaVA multimodal large language model based on the loss function;
[0074] The formula for calculating the loss function is shown below:
[0075]
[0076] Among them, L CA-VIT This represents the loss value weighted by the confidence score; n represents the number of dialogues in a multi-turn conversation; s i s represents the confidence score for each dialogue. i ∈s;L i The cross-entropy loss between the LLM output and the supervised answer is represented by the following formula:
[0077]
[0078] in, This represents the probability of the true label corresponding to the j-th prompt predicted by the large language model; This represents the actual label corresponding to the j-th prompt;
[0079] Based on the calculated L CA-VIT Implement backpropagation processing; calculate L CA-VIT For the evaluation score y m The derivative value is calculated, and the parameters of the open-source LLaVA multimodal large language model are updated by updating the calculated derivative value, thereby reducing the loss L. CA-VIT The value;
[0080] Step 3) involves using the final LLaVA multimodal large language model obtained in step 2) to perform question-answering tests on images, thus verifying the effectiveness of the image question-answering data acquisition model. Specifically, this includes:
[0081] The final LLaVA multimodal large language model was used to perform question answering tests on images, verifying the effectiveness of the image question answering data obtained by the model through image question answering data acquisition.
[0082] If the image question-answering data obtained from the labeled results can correctly answer questions in actual dialogues using the final LLaVA multimodal large language model, then the image question-answering data obtained from the labeled results model is valid; if the image question-answering data obtained from the labeled results cannot correctly answer questions in actual dialogues using the final LLaVA multimodal large language model, then the image question-answering data obtained from the labeled results model is invalid.
[0083] The multi-turn visual dialogue and self-evaluation large model method provided by this invention, by combining an image encoder and a large language model, can realize multi-turn dialogue data based on image content as input, and the output data can be further used to fine-tune the multimodal question answering large model, thereby experimenting with multimodal question answering tasks; the method of this invention improves performance, increases efficiency, and enhances accuracy. Attached Figure Description
[0084] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0085] like Figure 1 The diagram shown illustrates the method flow of this invention: This method for acquiring image question-answering data based on a large visual language model, provided by this invention, includes the following steps:
[0086] S1. Obtain the image dataset and encode the images in the dataset to obtain image embeddings; specifically including:
[0087] The image dataset consists of raw images I and visual instruction data T;
[0088] In the method of this invention, the original image I includes the Visual Genome image dataset and the LLaVA_instruct_80k image dataset; the visual instruction data T includes the multi-turn dialogue description text dataset corresponding to the LLaVA_instruct_80K image dataset;
[0089] The LLaVA_instruct_80k image dataset and the corresponding multi-turn dialogue description text dataset are used to train a multi-turn visual dialogue and self-evaluation large language model.
[0090] The Visual Genome image dataset was used to evaluate trained multi-turn visual dialogue and self-evaluating large language models.
[0091] An image encoder model is used to encode the original image I to obtain the corresponding image embedding X. I ;
[0092] In the method of this invention, the CLIP image encoder ViT-L / 14 is selected; ViT-L / 14 represents the publicly disclosed ViT-L pre-trained image encoder model;
[0093] The visual instruction data T consists of real question-and-answer data pairs, used to calculate the cross-entropy loss function;
[0094] S2. Encode the specified prompt data according to the set format to obtain the corresponding prompt embedding; specifically including:
[0095] Use the following configuration prompt data:
[0096] “Generate multiple question-answer pairs based on the image.Requirement:The answer to each question must be found within the image, and the format is as follows:Question:…[\n]Answer:…[\n]”;
[0097] The specified prompt data is encoded according to the set format to obtain the prompt embedded X. prompt ;
[0098] In the method of this invention, a pre-trained token embedding model in the LLaMA-2 model is selected, and the prompt data is encoded.
[0099] S3. Using the image embedding obtained in step S1 and the prompt embedding obtained in step S2, a large language model is used to obtain question-answer data pairs and corresponding predicted probability vectors; specifically including:
[0100] For image embedding X I And prompts embedded X prompt The data is spliced together using a large language model Φ. LPerform large-scale language analysis to obtain predicted question-answer data pairs and corresponding predicted probability vectors;
[0101] In the method of this invention, the large language model disclosed in LLaMA-2 is selected as the large language model Φ. L ;
[0102] The following formula represents the predicted probability vector v corresponding to the hint in each question-and-answer data pair in the output:
[0103] [X C ,v]=Φ L ([X I ,X prompt ])
[0104] Among them, X C Dialogue data representing the generated questions and their corresponding answers;
[0105] S4. Using the question-and-answer data pairs and corresponding predicted probability vectors obtained in step S3, the evaluation score of the question-and-answer data pairs is obtained through the instruction evaluator; specifically including:
[0106] Select the predicted probability vector v, and obtain the corresponding evaluation scores for the question-and-answer data pairs through the instruction evaluator Ψ. The evaluation process is represented by the following formula:
[0107] s = Ψ(v)
[0108] Where s represents the output of the instruction evaluator as the evaluation score of the question-answer data pair;
[0109] The instruction evaluator Ψ is a fully connected linear layer, directly added to the large language model Φ. L After the output layer, the large language model Φ L The output of is used as the input to the instruction evaluator Ψ;
[0110] A fully connected linear layer is a single layer of a fully connected network.
[0111] In the method of the present invention, the input length of the instruction evaluator is 2048. If the length of v is less than 2048, zero-padding alignment is required.
[0112] S5. Using the question-and-answer data pairs and real question-and-answer data pairs obtained in step S3, optimize the large language model by calculating the cross-entropy loss function; specifically including:
[0113] The cross-entropy loss function is calculated using the predicted question-and-answer data pairs obtained in step S3 and the actual question-and-answer data pairs.
[0114] Among them, the real question-and-answer data pair is the visual instruction data T obtained in step S1;
[0115] During model training, the loss is calculated based on the output question-and-answer data and the real question-and-answer data, and the model is optimized based on the loss results.
[0116] The following formula represents the generation of target dialogue data X. C The probability of:
[0117]
[0118] Where p(X) C |X I ,X prompt ) represents the product of the probabilities of all cues corresponding to the dialogue data generated by the large language model; p θ (·) represents the probability of one of the prompts generated by the large language model; x i The model provides real-time prediction prompts during execution; L represents the length of the output question-and-answer dialogue; θ represents the trainable parameters; X C,＜i x represents i All previous dialogue prompts;
[0119] Using the generated target dialogue data X C The probability p(X) C |X I ,X prompt ), determine the labels of the predicted data l m Simultaneously, generate target dialogue data X. C The probability p(X) C |X I ,X prompt It also serves as input to the instruction evaluator;
[0120] By calculating the labels of dialogue data prompts predicted by a large language model and the labels of actual dialogue data prompts... n The cross-entropy loss between the two sides is used to complete the optimization of the model;
[0121] The cross-entropy loss is calculated using the following formula:
[0122]
[0123] Among them, L MT-VCG Represents cross-entropy loss; This represents the probability of the true label corresponding to the i-th token predicted by the large language model; represents the true label corresponding to the i-th Token; L represents the cross-entropy loss value between the model prediction result and the true result;
[0124] During the training of the large language model, the cross-entropy loss L corresponding to each prediction result is calculated. MT-VCG And backpropagation, updating the large language model Φ L The parameters are adjusted until the set conditions are met to obtain the optimal model;
[0125] S6. Optimize the instruction evaluator using the evaluation scores of the question-and-answer data pairs obtained in step S4 and the cross-entropy loss function calculated in step S5; specifically including:
[0126] The loss of the instruction evaluator is expressed by the following formula:
[0127] L VISE =L1(tanh(L MT-VCG ),1-s)
[0128] Among them, L VISE L1 represents the loss value of the instruction evaluator; L1 represents the absolute value loss function, calculated as follows:
[0129] L1=|L MT-VCG -(1-s)|
[0130] tanh(L MT-VCG ) represents the autoregressive loss value of each question-answer pair after activation using the tanh(·) function;
[0131] Based on the calculated L VISE To achieve backpropagation processing;
[0132] Calculate L VISE The derivative of the evaluation score s is used to update the parameters of the evaluator, thereby reducing the loss L. VISE The value;
[0133] S7. Using an image dataset without question-answering data annotation, obtain the image question-answering data of the annotation results through the large language model optimized in step S5 and the instruction evaluator optimized in step S6; specifically including:
[0134] An image question-answering data acquisition model is constructed using the optimized large language model in step S5 and the optimized instruction evaluator in step S6.
[0135] The image dataset without question-answering data annotation is input into the optimized large language model in step S5, and the dialogue data and corresponding probability vectors are output simultaneously; the output probability vectors are input into the optimized instruction evaluator in step S6, and the score of the dialogue data is output simultaneously.
[0136] By constructing an image question-answering data acquisition model, and using an image dataset without question-answering data annotation, the image question-answering data acquisition and processing is completed, and the labeled image question-answering data is obtained.
[0137] The labeled image question-answering data includes dialogue data output by the optimized large language model and scores output by the optimized instruction evaluator;
[0138] The method of the present invention further includes the following steps:
[0139] 1) Obtain the open-source LLaVA multimodal language model and fine-tune it; specifically including:
[0140] Fine-tuning includes the fully connected layer between the image encoder and the large language model in the open-source LLaVA multimodal large language model, and the large language model within the open-source LLaVA multimodal large language model;
[0141] Fine-tuning parameters includes all parameters in the fully connected layers and the large language model;
[0142] The image encoder in the open-source LLaVA multimodal large language model is used for image encoding processing;
[0143] The image encoder and the large language model are connected through a fully connected layer network;
[0144] During the fine-tuning process, the input data of the open-source LLaVA multimodal large language model are the prompt codes corresponding to the image and the prompt codes corresponding to the question, and the output data is the answer to the corresponding question;
[0145] 2) Optimize the open-source LLaVA multimodal language model after fine-tuning in step 1) to obtain the final LLaVA multimodal language model; specifically including:
[0146] Optimizations include the fully connected layer between the image encoder and the large language model in the open-source LLaVA multimodal large language model, and the large language model within the open-source LLaVA multimodal large language model.
[0147] The optimization parameters include all parameters in the fully connected layer and the large language model;
[0148] Using the output data from step 1) and the image question-and-answer data from the annotation results obtained in step S7, calculate the loss function and optimize the open-source LLaVA multimodal large language model based on the loss function;
[0149] The formula for calculating the loss function is shown below:
[0150]
[0151] Among them, L CA-VIT This represents the loss value weighted by the confidence score; n represents the number of dialogues in a multi-turn conversation; s i s represents the confidence score for each dialogue. i ∈s;L i The cross-entropy loss between the LLM output and the supervised answer is represented by the following formula:
[0152]
[0153] in, This represents the probability of the true label corresponding to the j-th prompt predicted by the large language model; This represents the actual label corresponding to the j-th prompt;
[0154] Based on the calculated L CA-VIT Implement backpropagation processing; calculate L CA-VIT For the evaluation score y m The derivative value is calculated, and the parameters of the open-source LLaVA multimodal large language model are updated by updating the calculated derivative value, thereby reducing the loss L. CA-VIT The value;
[0155] 3) Using the final LLaVA multimodal large language model obtained in step 2), conduct question-answering tests on images to verify the effectiveness of the image question-answering data acquisition model; specifically including:
[0156] The final LLaVA multimodal large language model was used to perform question answering tests on images, verifying the effectiveness of the image question answering data obtained by the model through image question answering data acquisition.
[0157] If the image question-answering data obtained from the labeled results can correctly answer questions in actual dialogues using the final LLaVA multimodal large language model, then the image question-answering data obtained from the labeled results model is valid; if the image question-answering data obtained from the labeled results cannot correctly answer questions in actual dialogues using the final LLaVA multimodal large language model, then the image question-answering data obtained from the labeled results model is invalid.
[0158] In the method of this invention, after generating multi-turn dialogue data on the Virtual Geneme (VG) dataset, the LLaVA model is fine-tuned.
[0159] The performance of different methods was evaluated using the MMBench, SEED-Bench, and MM-Vet datasets.
[0160] The MMBench dataset is used to evaluate the model's abilities in attribute reasoning (AR), coarse-grained perception (CP), fine-grained perception (CrossInstance) (FP-C), fine-grained perception (Single Instance) (FP-S), logical reasoning (LR), and relational reasoning (RR).
[0161] SEED-Bench is used to evaluate a model’s capabilities in Scene Understanding (SC), Instance Identity (IId), Instance Attributes (IA), Instance Location (IL), Instance Counting (IC), Spatial Relations (SR), Instance Interaction (IIt), Visual Reasoning (VR), and Text Recognition (TR).
[0162] The MM-Vet dataset is used to evaluate a model’s recognition (Rec), knowledge (Know), optical character recognition (OCR), spatial awareness (Spat), language generation (Gen), and mathematical (Mathematics) abilities.
[0163] The experimental results are shown below:
[0164] Table 1 shows a comparison of test performance on the MMBench development and test sets using the VG dataset and without the VG dataset, respectively.
[0165] In the method of this invention, after generating multi-turn dialogue data on the Virtual Geneme (VG) dataset, the LLaVA model is fine-tuned.
[0166] The performance of different methods was evaluated using the MMBench, SEED-Bench, and MM-Vet datasets.
[0167] The MMBench dataset is used to evaluate the model's abilities in attribute reasoning (AR), coarse-grained perception (CP), fine-grained perception (CrossInstance) (FP-C), fine-grained perception (Single Instance) (FP-S), logical reasoning (LR), and relational reasoning (RR).
[0168] SEED-Bench is used to evaluate a model’s capabilities in Scene Understanding (SC), Instance Identity (IId), Instance Attributes (IA), Instance Location (IL), Instance Counting (IC), Spatial Relations (SR), Instance Interaction (IIt), Visual Reasoning (VR), and Text Recognition (TR).
[0169] The MM-Vet dataset is used to evaluate a model’s recognition (Rec), knowledge (Know), optical character recognition (OCR), spatial awareness (Spat), language generation (Gen), and mathematical (Mathematics) abilities.
[0170] The experimental results are shown below:
[0171] Table 1 shows a comparison of test performance on the MMBench development and test sets using the VG dataset and without the VG dataset, respectively.
[0172] Table 1
[0173]
[0174] Table 2 is a comparative table of test performance on the SEED-Bench development set and test set, showing the model fine-tuned using the VG dataset and without the VG dataset, respectively.
[0175] Table 2
[0176]
[0177] Table 3 is a comparative table of test performance on the MM-Vet development set and test set, showing the model fine-tuned using the VG dataset and the model without using the VG dataset.
[0178] Table 3
[0179] LLaVA 27.1 32.8 18.5 20 21.1 26.8 3.8 +VG 29.1 34.9 20.1 22.4 24.2 22.9 3.8
[0180] Tables 1, 2, and 3 demonstrate that the overall score of LLaVA+VG outperforms the LLaVA model on all three datasets, achieving improvements of 4.08% and 8.06% on the MM-Bench development and test sets, respectively, 0.85% on the SEED-Bench dataset, and 2.0% on the MM-Vet dataset. These results prove the effectiveness of the method of this invention.
[0181] This invention is supported by the Hunan Provincial Natural Science Foundation (Youth Fund) for Hierarchical Open Set Target Extraction and Retrieval for Natural Scene Images (2022JJ40632) and the National Natural Science Foundation of China (Youth Fund) for Research on Key Technologies of Open Set Target Extraction and Retrieval for Open World Scenes (62202499).
Claims
1. A method for acquiring image question-answering data based on a large visual language model, comprising the following steps: S1. Obtain the image dataset and encode the images in the image dataset to obtain image embeddings; S2. Encode the set prompt data according to the set format to obtain the corresponding prompt embedding; S3. Using the image embedding obtained in step S1 and the prompt embedding obtained in step S2, the question-answer data pairs and corresponding prediction probability vectors are obtained through the large language model; S4. Using the question-and-answer data pairs and corresponding prediction probability vectors obtained in step S3, obtain the evaluation score of the question-and-answer data pairs through the instruction evaluator; S5. Using the question-and-answer data pairs and real question-and-answer data pairs obtained in step S3, optimize the large language model by calculating the cross-entropy loss function; specifically including: The cross-entropy loss function is calculated using the predicted question-and-answer data pairs obtained in step S3 and the actual question-and-answer data pairs. Among them, the real question-and-answer data is the visual instruction data obtained in step S1. ; During model training, the loss is calculated based on the output question-and-answer data and the real question-and-answer data, and the model is optimized based on the loss results. The following formula represents the generation of target dialogue data. The probability of: in, This represents the product of the probabilities of all prompts corresponding to the dialogue data generated by the large language model; This represents the probability of one of the prompts generated by the large language model; This indicates a prompt for real-time predictions during model execution; Indicates the length of the output question-and-answer dialogue; Indicates trainable parameters; express All previous dialogue prompts; Image embedding; For the purpose of embedding; Using generated target dialogue data probability Determine the labels of the predicted data Simultaneously, generate target dialogue data. probability It also serves as input to the instruction evaluator; By calculating the labels of dialogue data prompts predicted by a large language model and the labels of actual dialogue data prompts... The cross-entropy loss between the two sides is used to complete the optimization of the model; The cross-entropy loss is calculated using the following formula: in, Represents cross-entropy loss; The first one represents the prediction of the large language model. The probability of a token corresponding to a real tag; Indicates the first The real tag corresponding to each token; This represents the cross-entropy loss value between the model's predicted results and the actual results; During the training of the large language model, the cross-entropy loss corresponding to each prediction result is calculated. And backpropagation to update the large language model The parameters are adjusted until the set conditions are met to obtain the optimal model; S6. Optimize the instruction evaluator using the evaluation scores of the question-answer data pairs obtained in step S4 and the cross-entropy loss function calculated in step S5; specifically including: The loss of the instruction evaluator is expressed by the following formula: in, This represents the loss value of the instruction evaluator; The absolute value loss function is calculated as follows: This indicates that each pair of question-and-answer data in the output is used The autoregressive loss value after function activation; Based on calculations To achieve backpropagation processing; calculate For the evaluation score The derivative value is calculated, and the parameters of the evaluator are updated by updating the calculated derivative value, thereby reducing the loss. The value; S7. Using an image dataset without question-answering data annotation, obtain the image question-answering data of the annotation results through the large language model optimized in step S5 and the instruction evaluator optimized in step S6.
2. The image question-answering data acquisition method based on a large visual language model according to claim 1, characterized in that... Step S1, which involves obtaining an image dataset and encoding the images in the dataset to obtain image embeddings, specifically includes: The image dataset includes the original images. and visual instruction data ; An image encoder model is used for the original image. Perform encoding processing to obtain the corresponding image embedding. ; Visual instruction data These are real question-and-answer data pairs used to calculate the cross-entropy loss function.
3. The image question-answering data acquisition method based on a large visual language model according to claim 2, characterized in that... Step S3, using the image embedding obtained in step S1 and the prompt embedding obtained in step S2, employs a large language model to obtain question-answer data pairs and corresponding prediction probability vectors, specifically including: For image embedding and prompt embedding The data is then processed using a large language model. Perform large-scale language analysis to obtain predicted question-answer data pairs and corresponding predicted probability vectors; The following formula represents the predicted probability vector corresponding to the prompt in each question-and-answer data pair output. : in, Dialogue data representing the generated questions and their corresponding answers.
4. The image question-answering data acquisition method based on a large visual language model according to claim 3, characterized in that... Step S4, which uses the question-and-answer data pairs and corresponding predicted probability vectors obtained in step S3, to obtain evaluation scores for the question-and-answer data pairs through an instruction evaluator, specifically includes: Select the prediction probability vector Through the instruction evaluator The evaluation process involves obtaining the corresponding assessment scores for the question-and-answer data and using the following formula to represent the evaluation process: in, This indicates that the output of the instruction evaluator is the evaluation score of the question-answer data pair; Instruction evaluator It is a fully connected linear layer, directly added to the large language model. After the output layer, the large language model The output of the instruction evaluator is used as the instruction evaluator. Input; A fully connected linear layer is a single fully connected network.
5. The image question-answering data acquisition method based on a large visual language model according to claim 4, characterized in that... Step S7 involves using an image dataset labeled without question-answering data. Through the optimized large language model from step S5 and the optimized instruction evaluator from step S6, image question-answering data with annotation results is obtained. Specifically, this includes: An image question-answering data acquisition model is constructed using the optimized large language model in step S5 and the optimized instruction evaluator in step S6. The image dataset without question-answering data annotation is input into the optimized large language model in step S5, and the dialogue data and corresponding probability vectors are output simultaneously; the output probability vectors are input into the optimized instruction evaluator in step S6, and the score of the dialogue data is output simultaneously. By constructing an image question-answering data acquisition model, and using an image dataset without question-answering data annotation, the image question-answering data acquisition and processing is completed, and the labeled image question-answering data is obtained. The labeled image question-answering data includes dialogue data output by the optimized large language model and scores output by the optimized instruction evaluator.