Multi-hop visual question and answer method based on inference path prompt

Abstract

Multi-modal multi-hop questions and answers need to be reasoned across multiple evidence fragments (texts and images). However, most of the existing methods often depend on reasoning shortcuts, for example, answers are positioned by matching questions with single words, which not only reduces the performance of the methods, but also weakens the credibility of the methods. In order to solve the problem, the invention provides a reasoning framework based on prompts, and aims to identify an intermediate unknown hop in the problem to generate a corresponding reasoning path for multi-modal multi-hop questions and answers. Specifically, an intermediate unknown detection and alignment module is introduced and used for recognizing intermediate unknown jumps in questions, aligning the intermediate unknown jumps with objects / texts in images / subtitles, then converting the recognized unknown jumps and corresponding answers into prompt sentences, namely intermediate reasoning paths, and finally connecting the prompt sentences with evidences. And guiding an existing multi-mode question-answer model to predict a final answer. Experiments on a MuMuQA data set show that the method is effective.

Description

Technical Field

[0001] The present invention belongs to the field of multi-hop visual question answering, and more specifically, relates to a multi-hop visual question answering method based on reasoning path prompts. Background Art

[0002] Multi-hop visual question answering is a challenging task that combines computer vision and natural language processing techniques. Its core is to answer image-related questions by integrating multiple pieces of evidence, including documents and images. Different from traditional visual question answering, multi-hop visual question answering requires not only a deeper understanding of the visual scene and context, but also the ability of multi-hop reasoning to arrive at the correct answer.

[0003] Existing multi-hop visual question answering methods have made some progress through end-to-end visual language models. However, while these methods can derive the correct answer, they fail to prove that the model follows the ideal reasoning path to arrive at the answer, such as providing a reasoning chain or explanation for the answer. In addition, existing end-to-end methods often fall into the trap of answering questions using reasoning shortcuts, that is, locating the answer by matching keywords in the question with a single sentence. In multi-hop visual question answering scenarios, multi-hop visual questions usually require the synthesis of multiple evidences, including images, captions, and context. Therefore, answering questions without following the expected reasoning path is inefficient and unreliable.

[0004] Providing reasoning paths when answering complex visual questions is considered an efficient and promising strategy in multi-hop visual question answering. However, annotating the intermediate unknown jumps and generating reasoning paths is expensive and time-consuming. Therefore, in order to help hint-based methods make better decisions and arrive at answers, it is necessary to design a method that can automatically identify the intermediate unknown jumps and generate corresponding reasoning paths. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-hop visual question answering method with reasoning path prompts. The present invention has two main components: 1) Detection and alignment of intermediate unknowns: first, entity phrases are extracted from the question as candidate intermediate unknown jumps; then, objects are extracted from the image for each unknown jump by using object detection technology; finally, the answers corresponding to the intermediate unknown jumps are identified by aligning the detected objects with the captions. 2) Prompt-based multi-hop visual question answering: first, the intermediate unknown jumps are combined with the corresponding answers to generate intermediate reasoning paths; then the questions, reasoning paths, contexts and images are connected into prompts; finally, this information is input into the prompt-based multi-hop visual question answering model to predict the final answer.

[0006] The present invention has the following beneficial effects:

[0007] 1) The present invention introduces an intermediate unknown detection and alignment module to extract intermediate unknown jumps and their corresponding answers. This module improves the detection accuracy of intermediate unknown entities and provides a certain degree of explainability for multimodal multi-hop question answering;

[0008] 2) We propose a scheme to generate high-quality reasoning paths for prompt-based multi-hop visual question answering, which can avoid the possibility of the model taking reasoning shortcuts to a certain extent; BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of a specific implementation method of the multi-hop visual question answering method based on reasoning path prompts of the present invention;

[0010] Figure 2 is a flow chart of detection and alignment of intermediate unknown jumps in the present invention;

[0011] Figure 3 is a flow chart of answer reasoning using a multimodal question-answering model in the present invention; DETAILED DESCRIPTION

[0012] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0013] Example

[0014] Figure 1 1 is a flowchart of a specific implementation of the multi-hop visual question answering method based on reasoning path prompts of the present invention. Figure 1 As shown, the specific steps of the multi-hop visual question answering method based on reasoning path prompts of the present invention include:

[0015] S101: Construct multimodal multi-hop question answering training and testing datasets:

[0016] Collect multimodal and multi-hop data according to actual needs. The data includes questions, images, image descriptions, and contexts.

[0017] S102: Detection and alignment of unknown intermediate jumps:

[0018] Select the entity extractor, object detection model, and image-text matching model according to actual needs. Use the entity extractor to extract entity phrases from the question and image description as candidate intermediate unknown jumps. Then, use the object detection model to extract objects from the image for each unknown jump through object detection technology. Finally, use the image-text matching module to align the detected objects with the image description.

[0019] In this embodiment, the entity extractor adopts the Stanza tool, the target detection model adopts the GroundingDINO model, and the image-text matching model adopts the CLIP model.

[0020] Figure 2 : is a flow chart of the detection and alignment of the intermediate unknown jump in the present invention. Figure 2 As shown, the specific steps of detecting and aligning the intermediate unknown jumps in the present invention include:

[0021] S201: Entity Extraction:

[0022] The Stanza tool is used as an entity extractor to extract entities from the question. The goal is to extract the entities / noun phrases as candidates for the intermediate unknown jumps. The extracted candidates for the intermediate unknown jumps are C h ={h1,h2,...,h m}.

[0023] S202: Target Detection:

[0024] The extracted intermediate unknown jump candidate C h ={h1,h2,...,h m} as the text object for target detection, the GroundingDINO model is used to detect the target in the image. The process can be expressed as follows:

[0025] {(obj j ,logit j )}=GroundingDINO(Img,h i )

[0026] where obj j Indicates that i The corresponding j-th candidate. logit j represents the confidence in the labeling process. We choose i The corresponding candidate object o with the highest confidence i As an intermediate object: Finally, we can get the intermediate object set C of all intermediate jumps o ={o1,o2,...,o m}, each o iBoth correspond to h i .

[0027] S203: Image text matching:

[0028] Similar to the entity extraction step, named entities and entity phrases C are extracted from the image description. e ={e1,e2,...,e m When extracting candidate entities, since images are usually long, a large number of entities irrelevant to the image may be extracted, which will affect the efficiency and accuracy of subsequent entity alignment. To avoid this problem, the present invention uses CLIP to remove irrelevant entities, thereby excluding entities irrelevant to the image. Specifically, the present invention uses image Img as the target image, candidate entity C e As the text to be matched, use CLIP's text encoder and image encoder to respectively e and Img, and obtain their feature representation F C and F Img The process can be expressed by the following formula:

[0029] F C = CLIPTextEncoder(C e )

[0030] F Img = CLIPImageEncoder(Img)

[0031] in n represents the number of candidate entities, and d represents the feature dimension after encoding. Then, we use F C and F Img Calculate each candidate entity e i Cosine similarity of image Img Finally, the top k candidate entities with the highest similarity are selected as entities related to the image, denoted as Then use the intermediate object C o Candidate entities related to Matching is performed to indirectly achieve the alignment from the candidate intermediate to the candidate entity. As text, match the candidate object C o As the target image, the CLIP model is then used to calculate and j The cosine similarity between: Finally, the candidate entity with the highest similarity is selected as i The corresponding intermediate unknown entity

[0032] S103: Reasoning path prompt (Prompt) construction:

[0033] The present invention utilizes the alignment results of the intermediate unknown jumps to construct the reasoning path prompts, and combines the input question (Question), image (Image), image description (Caption), and context (Context) to form all the inputs of the question-answering model.

[0034] S104: Answer reasoning using a multimodal question-answering model:

[0035] The constructed model input and the annotated answers are used to train the multimodal question answering model. Finally, the trained multimodal question answering model is used to perform multimodal multi-hop question answering reasoning.

[0036] Figure 3 FIG. 1 is a flowchart of the method for reasoning about answers using the multimodal question-answering model in the present invention. Figure 3 As shown, the specific steps of using the multimodal question-answering model to perform answer reasoning in the present invention include:

[0037] S301: Feature encoding of input text and image:

[0038] The present invention encodes the constructed text input and image input into feature vectors. Specifically, the present invention uses BERT as a text encoder to encode the input text, and this process can be expressed by the following formula:

[0039] H T =TextEncoder(Input)

[0040] in Then use DETR as the image encoder to encode the image features. This process can be expressed by the following formula:

[0041] H I =ImageEncoder(Img)

[0042] in

[0043] S302: Fusion of text and image features:

[0044] The present invention uses the Gate Fusion mechanism to fuse text and image features. Specifically, the qkv projection matrix in the multi-head attention mechanism is first used to act on H T , H I , H I On top, cross-attention is realized, so as to obtain the interactive feature H f I usion , the process can be expressed using the following formula:

[0045]

[0046] Then use the Gate Fusion mechanism to fuse H T as well as The process can be expressed using the following formula:

[0047]

[0048]

[0049] Where W l and W v represents the parameter matrix that can be learned.

[0050] S303: Use the fused features to perform text reasoning:

[0051] Finally, the fused features are put into the RoBERTa model for text reasoning.

[0052] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention.

[0053] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention. In this experimental verification, the multimodal multi-hop question answering dataset MuMuQA is used, which is composed of English news data such as VOA, including the context of English news, images related to the news, and image descriptions. The dataset contains 21,570 training samples and 1,120 test samples.

[0054] Four comparison methods were set up in this experimental verification, namely the end-to-end methods OSCAR and BLIP2, the problem decomposition-based method MuMuQA, and the thought chain-based method MMCOT. The present invention uses F1-score as an evaluation indicator to evaluate the overlap between the model output text and the annotated answer. The present invention evaluates the effects of each method in the two steps of identifying intermediate unknown jumps and predicting the final answer.

[0055] The present invention is implemented by PyTorch and trained on NVIDIA RTX3090 GPU.

[0056] Table 1 is a statistical table of the effects of the present invention and the comparative method under different tasks in this embodiment.

[0057]

[0058] Table 1

[0059] As shown in Table 1, it can be seen from the results in Table 1 that the present invention achieves the best results in the prediction of the intermediate unknown jumps and the prediction of the final answer of the MuMuQA dataset, thereby verifying the effectiveness of the present invention.

[0060] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A multi-hop visual question answering method based on reasoning path prompts, characterized in that: The following steps are involved: S1: Collect multi-modal multi-hop data according to actual needs. The data includes questions, images, image descriptions, and contexts. S2: Select entity extractors, object detection models, and image-text matching models according to actual needs. Use the entity extractor to extract entity phrases from the question and image description as candidate intermediate unknown jumps. Then, use the object detection model to extract objects from the image for each unknown jump through object detection technology. Finally, use the image-text matching module to align the detected objects with the image description. The specific steps include: S2.1: Use the Stanza tool as an entity extractor to extract entities from the question. The goal is to extract the entities / noun phrases as candidates for the intermediate unknown jumps. The extracted candidates for the intermediate unknown jumps are C h ={h1,h2,...,h m }. S2.2: Extract the candidate C of the intermediate unknown jump h ={h1,h2,...,h m } as the text object for target detection, the GroundingDINO model is used to detect the target in the image. The process can be expressed as follows: {(obj j ,logit j )}=GroundingDINO(Img,h i ) where obj j Indicates that i The corresponding j-th candidate. logit j represents the confidence in the labeling process. We choose i The corresponding candidate object o with the highest confidence i As an intermediate object: Finally, we can get the intermediate object set C of all intermediate jumps o ={o1,o2,…,o m }, each o i Both correspond to h i . S2.3: Similar to the entity extraction step, named entities and entity phrases C are extracted from the image description. e ={e1,e2,...,e m When extracting candidate entities, since images are usually long, a large number of entities irrelevant to the image may be extracted, which will affect the efficiency and accuracy of subsequent entity alignment. To avoid this problem, the present invention uses CLIP to remove irrelevant entities, thereby excluding entities irrelevant to the image. Specifically, the present invention uses image Img as the target image, candidate entity C e As the text to be matched, use CLIP's text encoder and image encoder to respectively e and Img, and obtain their feature representation F C and F Img The process can be expressed by the following formula: F C =CLIPTextEncoder(C e ) F Img =CLIPImageEncoder(Img) in n represents the number of candidate entities, and d represents the feature dimension after encoding. Then, we use F C and F Img Calculate each candidate entity e i Cosine similarity of image Img Finally, the top k candidate entities with the highest similarity are selected as entities related to the image, denoted as Then use the intermediate object C o Candidate entities related to Matching is performed to indirectly achieve the alignment from the candidate intermediate to the candidate entity. As text, match the candidate object C o As the target image, the CLIP model is then used to calculate and j The cosine similarity between: Finally, the candidate entity with the highest similarity is selected as i The corresponding intermediate unknown entity S3: The alignment results of the intermediate unknown jumps are used to construct reasoning path prompts, and the input question (Question), image (Image), image description (Caption), and context (Context) constitute all the inputs of the question-answering model. S4: Use the multimodal question-answering model to reason about the answer. The specific steps are as follows: S4.1: Encode the input text and image. Encode the constructed text input and image input into feature vectors. Specifically, the present invention uses BERT as a text encoder to encode the input text. This process can be expressed by the following formula: H T =TextEncoder(Input) in Then use DETR as the image encoder to encode the image features. This process can be expressed by the following formula: H I =ImageEncoder(Img) in S4.2: Use Gate Fusion mechanism to fuse text and image features. Specifically, first use the qkv projection matrix in the multi-head attention mechanism to act on H T , H I , H I To achieve cross-attention, we can get the interactive features. The process can be expressed using the following formula: Then use the Gate Fusion mechanism to fuse H T as well as The process can be expressed using the following formula: Where W l and W v represents the parameter matrix that can be learned. S4.3: Put the fused features into the RoBERTa model for text reasoning.

Images