Large language model guided panoramic image description enhanced visual question and answer method and system

Abstract

The invention discloses a panoramic image description enhanced visual question answering method and system guided by a large language model, and belongs to the field of multi-modal information processing. In order to solve the problem that an existing method lacks accurate description of key details of an image, so that an answer system generates an answer mistakenly, the invention designs a system comprising a multi-level image information acquisition module, a constraint-enhanced panoramic image description generation module and a visual question-answering module. And the multi-level image information acquisition module is used for extracting rich visual information from the image by effectively utilizing the visual language model. And the constraint-enhanced panoramic image description generation module is used for generating detailed panoramic image description in combination with the inference capability of the large language model, and introducing a context constraint example and a constraint instruction to relieve the illusion problem possibly generated by the large language model. And the visual question and answer module selects a question and answer sample as a context learning sample from three dimensions of image description, question description and panoramic image description, and effectively guides the application of the large language model in question answering.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal information processing, and in particular relates to a panoramic image description enhanced visual question answering method and system guided by a large language model. Background Art

[0002] Knowledge-based Visual Question Answering (KB-VQA) is a challenging task that requires leveraging knowledge beyond image content to answer questions, and its applications cover various fields such as healthcare, transportation, and education.

[0003] Existing KB-VQA methods convert multimodal tasks into natural language processing tasks, obtain the descriptions corresponding to the images through the Vision-Language Model (VLM), and then use these image descriptions as prompts to help the Large Language Model (LLM) generate answers. However, the image description models used in existing KB-VQA methods mainly focus on global alignment in the pre-training stage, emphasizing the overall match between the entire image and the sentence, ignoring key information such as sentence keywords and specific objects in the image, and these descriptions usually lack the detailed details of the image-specific elements required in question answering, causing the large language model to make mistakes when answering questions. Although some KB-VQA methods try to incorporate fine-grained visual information such as labels, this fine-grained visual information is semantically incomplete, sparse, and contains errors, which usually introduces a lot of irrelevant noise to the model. Summary of the invention

[0004] In view of the problem that the existing KB-VQA method lacks accurate description of key details and objects in the image, which causes the question-answering system to generate incorrect answers, the present invention provides a panoramic image description enhanced visual question-answering method and system guided by a large language model.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A large language model-guided panoramic image description-enhanced visual question-answering system, including a multi-level image information acquisition module, a constraint-enhanced panoramic image description generation module, and a visual question-answering module;

[0007] The multi-level image information acquisition module: generates the global and detailed text representation required for visual input through the visual language model; although VLMs are good at generating image descriptions, they usually produce descriptions that lack visual details and thus cannot faithfully represent the content of the image. The multi-level image information acquisition module focuses on generating more comprehensive and detailed image descriptions through the following two steps.

[0008] The constraint-enhanced panoramic image description generation module uses context constraint examples as references to guide the synthesis of panoramic image descriptions and reduce the hallucination problem when generating descriptions;

[0009] The visual question answering module: uses the powerful reasoning of a large language model to extract accurate answers;

[0010] Furthermore, the multi-level image information acquisition module includes two image information acquisition modules.

[0011] A large language model guided panoramic image description enhanced visual question answering method, comprising the following steps:

[0012] Step 1: transform the image v i Input into the visual language model BLIP2, and use the generated general image description as the global image description, denoted as c g ;

[0013] c g =BLIP2(v i )

[0014] Detailed image description: In order to obtain finer image details, we further use the dense image description model GRiT and OCR to describe the image v i Processing is performed to generate K different detailed descriptions, expressed as

[0015] Step 2: Use context-constrained examples as references to guide the synthesis of panoramic image descriptions and reduce the hallucination problem when generating descriptions;

[0016] Step 3: Use the powerful reasoning ability of the large language model and question answering (QA) examples as prompts to extract accurate answers.

[0017] Further, using the context constraint example as a reference, specifically: based on the multi-level image information acquisition module, LLM is used to convert the global image description c g and detailed image description Refine and consolidate the information in the document into a more specific and cohesive description;

[0018]

[0019] Among them, L 1 represents a context constraint example, L 2 Represents a constraint instruction.

[0020] Furthermore, the context constraint example specifically includes: using a standard In-Context Learning (ICL) example as a key constraint to promote the model to generate answers that meet the requirements, ensuring the consistency of its output and being faithful to the image content;

[0021] We carefully selected 10 standard examples written by humans as context constraints; these examples include global image descriptions, detailed image descriptions, and standard image descriptions written by humans; the standard examples follow two main principles: ① cover the visual details required in the prompt; ② describe the main objects, such as the global image description.

[0022] Furthermore, the constraint instructions for image description generation. In order to enhance the ability of LLM to remain consistent with the authenticity of the image when generating image descriptions with more details, we formulated prompt instructions to stimulate the model's ability in inductive summarization. To reduce the potential risk of LLM generating incorrect content, we further instruct the model to "avoid inferences beyond the image content and use concise wording." The following is an example of a prompt for panoramic image description synthesis: "Please merge the global description and the detailed description to generate a more specific description. The global description provides a description of the overall scene, while the detailed description supplements the details of the global description. Avoid inferences beyond the image content and use concise wording."

[0023] After converting images to text, the KB-VQA task naturally turns into a text-based question answering task. We used the following prompt: “Given the provided context, answer the question”. In addition, we believe that answers to similar examples are likely to be similar, which can further motivate LLM to generate more consistent answers. To achieve this goal, we link ICL examples together to build prompts. Each example contains a context generated by our model, a question, and an answer. Given that LLM sometimes produces irrelevant text when generating answers, we explicitly specify in the prompt that “the answer length should not exceed two words”.

[0024] Furthermore, the QA example specifically includes: using CLIP to calculate the similarity between the test sample and the training sample; CLIP includes a visual encoder and a text encoder, which are used to process images and texts respectively;

[0025] The question and panoramic image description are fed into CLIP’s text encoder, the image is fed into CLIP’s visual encoder, and then the dot product between the image and the text is calculated to evaluate the similarity between the two.

[0026]

[0027] Where τ is a learnable temperature parameter, E represents the first M examples; question qi and the panoramic image description c p Connect to form t i ; s(,) represents the dot product.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] This paper proposes an innovative and efficient visual question answering method. This method uses LLM to generate image descriptions containing key details, so as to better stimulate the question answering ability of LLM. In addition, by introducing constrained examples and instructions, the hallucination problem that may occur in the generation process is effectively alleviated. The overall experimental results on the OK-VQA and VQAv2 datasets show that our model significantly outperforms existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 Schematic diagram of the large language model-guided panoramic image description-enhanced visual question answering method. DETAILED DESCRIPTION

[0032] In order to gain a deeper understanding of the present invention, we will provide a comprehensive and detailed description of the present invention. However, the present invention has multiple implementations and is not limited to the specific examples listed herein. The presentation of these examples is intended to deepen the comprehensive understanding of the disclosure of the present invention.

[0033] Example 1

[0034] A large language model-guided panoramic image description-enhanced visual question-answering system, including a multi-level image information acquisition module, a constraint-enhanced panoramic image description generation module, and a visual question-answering module;

[0035] The multi-level image information acquisition module: generates the global and detailed text representation required for visual input through the visual language model; although VLMs are good at generating image descriptions, they usually produce descriptions that lack visual details and thus cannot faithfully represent the content of the image. The multi-level image information acquisition module focuses on generating more comprehensive and detailed image descriptions through the following two steps.

[0036] The constraint-enhanced panoramic image description generation module uses context constraint examples as references to guide the synthesis of panoramic image descriptions and reduce the hallucination problem when generating descriptions;

[0037] The visual question answering module: uses the powerful reasoning of a large language model to extract accurate answers;

[0038] The multi-level image information acquisition module includes two image information acquisition modules.

[0039] Example 2

[0040] A large language model guided panoramic image description enhanced visual question answering method, comprising the following steps:

[0041] Step 1: transform the image v i Input into the visual language model BLIP2, and use the generated general image description as the global image description, denoted as c g ;

[0042] c g =BLIP2(v i )

[0043] Detailed image description: In order to obtain finer image details, we further use the dense image description model GRiT and OCR to describe the image v i Processing is performed to generate K different detailed descriptions, expressed as

[0044] Step 2: Use context-constrained examples as references to guide the synthesis of panoramic image descriptions and reduce the hallucination problem when generating descriptions;

[0045] Step 3, leverage the powerful reasoning capabilities of the large language model with QA examples as prompts to extract accurate answers.

[0046] Based on the multi-level image information acquisition module, we use LLM to describe the global image c g and detailed image description The information in the image is refined and merged into a more specific and cohesive description. Although the image description obtained from the multi-level image information acquisition module contains useful information, constraints must still be incorporated in the summarization and reasoning process based on the large language model to mitigate the hallucination bias that usually occurs when summarizing and reasoning with LLM.

[0047] Using the context constraint example as a reference, the specific method is: based on the multi-level image information acquisition module, LLM is used to convert the global image description c g and detailed image description Refine and consolidate the information in the document into a more specific and cohesive description;

[0048]

[0049] Among them, L 1 represents a context constraint example, L2 Represents a constraint instruction.

[0050] The context constraint examples are as follows: using standard ICL examples as key constraints to promote the model to generate answers that meet the requirements, ensuring the consistency of its output and being faithful to the image content;

[0051] In previous studies, ICL is considered to play an important role in few-shot learning with large language models (LLMs). In addition, standard ICL examples can serve as key constraints to promote the model to generate answers that meet the requirements, ensuring the consistency of its output and being faithful to the image content. To this end, we carefully selected 10 standard examples written by humans as context constraints; these examples include global image descriptions, detailed image descriptions, and standard image descriptions written by humans; standard examples follow two main principles: ① Cover the visual details required in the prompt; ② Describe the main object, such as the global image description.

[0052] Constrained instructions for image description generation. To enhance the LLM’s ability to remain consistent with image authenticity when generating image descriptions with more details, we formulated prompt instructions to stimulate the model’s ability in generalization and summarization. To reduce the potential risk of LLM generating incorrect content, we further instruct the model to “avoid inferences beyond the image content and use concise wording.” The following is an example of a prompt for panoramic image description synthesis: “Please merge the global description and the detailed description to generate a more specific description. The global description provides a description of the overall scene, while the detailed description supplements the details of the global description. Avoid inferences beyond the image content and use concise wording.”

[0053] After converting images to text, the visual question answering task naturally transforms into a text-based question answering task. We used the following prompt: “Given the provided context, answer the question”. In addition, we believe that answers to similar examples are likely to be similar, which can further motivate LLM to generate more consistent answers. To achieve this goal, we link ICL examples together to construct prompts. Each example contains a context generated by our model, a question, and an answer. Given that LLM sometimes produces irrelevant text when generating answers, we explicitly specify in the prompt that “the answer should be no longer than two words”.

[0054] The QA example is as follows: CLIP is used to calculate the similarity between the test sample and the training sample; CLIP contains a visual encoder and a text encoder, which are used to process images and text respectively;

[0055] The question and panoramic image description are fed into CLIP’s text encoder, the image is fed into CLIP’s visual encoder, and then the dot product between the image and the text is calculated to evaluate the similarity between the two.

[0056]

[0057] Where τ is a learnable temperature parameter, E represents the first M examples; question q i and the panoramic image description c p Connect to form t i .

[0058] The specific experimental results are shown in Table 1.

[0059] Table 1 shows the experimental results of the proposed method on two public visual question answering datasets.

[0060] method OK-VQA VQAv2 LaKo 47.0 68.1 PICa 48.0 56.1 The present invention 54.5 73.2

[0061] As can be seen from Table 1, the proposed method outperforms the two baseline methods on both datasets, fully verifying its effectiveness. It is worth noting that compared with the PICa model that uses global image captions as contextual knowledge, our method significantly improves the average performance of the two datasets by 11.8%. This result shows that the panoramic image description driven by the large language model can provide the model with richer visual information.

[0062] The contents not described in detail in the specification of the present invention belong to the prior art known to the professional and technical personnel in the field. Although the illustrative specific embodiments of the present invention are described above to facilitate the understanding of the present invention by the technical personnel in the field, it should be clear that the present invention is not limited to the scope of the specific embodiments. For the ordinary technical personnel in the field, as long as various changes are within the spirit and scope of the present invention defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A large language model-guided panoramic image description-enhanced visual question answering system, characterized by: It includes a multi-level image information acquisition module, a constraint-enhanced panoramic image description generation module, and a visual question-answering module; The multi-level image information acquisition module generates global and detailed text representations required for visual input through a visual language model; The constraint-enhanced panoramic image description generation module uses context constraint examples as references to guide the synthesis of panoramic image descriptions and reduce the hallucination problem when generating descriptions; The visual question answering module uses the powerful reasoning ability of the large language model to extract accurate answers.

2. The large language model guided panoramic image description enhanced visual question answering system according to claim 1, characterized in that: The multi-level image information acquisition module includes two image information acquisition modules.

3. A large language model-guided panoramic image description enhanced visual question answering method, characterized by: The following steps are involved: Step 1: transform the image v i Input into the visual language model BLIP2, and use the generated general image description as the global image description, denoted as c g ; c g =BLIP2(v i ) Use the dense image description model GRiT and OCR to quantify the image v i Processing is performed to generate K different detailed descriptions, expressed as Step 2: Use context-constrained examples as references to guide the synthesis of panoramic image descriptions and reduce the hallucination problem when generating descriptions; Step 3, leverage the reasoning power of a large language model with QA examples as prompts to extract accurate answers.

4. The method for visual question answering with panoramic image description enhancement guided by a large language model according to claim 3, characterized in that: Using the context constraint example as a reference, the specific example is: based on the multi-level image information acquisition module, LLM is used to convert the global image description c g and detailed image description Refine and consolidate the information in the document into a more specific and cohesive description; Among them, L1 represents the context constraint example and L2 represents the constraint instruction.

5. The large language model guided panoramic image description enhanced visual question answering method according to claim 3, characterized in that: The context constraint example specifically uses standard ICL examples as key constraints to promote the model to generate answers that meet the requirements, ensuring the consistency of its output and being faithful to the image content.

6. The method for visual question answering with panoramic image description enhancement guided by a large language model according to claim 3, characterized in that: The QA example is as follows: CLIP is used to calculate the similarity between the test sample and the training sample; CLIP includes a visual encoder and a text encoder, which are used to process images and text respectively; The question and panoramic image description are fed into CLIP’s text encoder, the image is fed into CLIP’s visual encoder, and then the dot product between the image and the text is calculated to evaluate the similarity between the two. Where τ is a learnable temperature parameter and E represents the first M examples. i and the panoramic image description c p Connect to form t i .

Images