A generation system and method for few-shot multi-modal explanation visual question answering
Patent Information
- Application Number
- CN202611508209.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-28
- Publication Date
- 2026-11-03
AI Technical Summary
Existing technologies in visual question answering suffer from problems such as missing visual evidence and a mismatch between question and example logic, which prevents the model from effectively addressing the issue and makes it impossible for existing technologies to provide high-quality multimodal explanations.
We adopt a method for generating visual question answering with few samples and multimodal interpretations. By constructing a local search graph, we perform rigorous visual existence verification and bounding box localization. Combined with an adaptive sample selector, we generate multimodal interpretations that include visual bounding box anchor points.
It effectively solves the problems of missing visual evidence and mismatch between question and example logic, generates high-quality multimodal explanations, and improves the model credibility and answer prediction accuracy of visual question answering.
Smart Images
Figure CN123030727A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, particularly the interdisciplinary field of computer vision and natural language processing, and especially to a generation system and method for visual question answering with few samples and multimodal interpretation. Background Technology
[0002] Visual question answering (VQA) requires models to answer natural language questions based on images. To improve the transparency and reliability of model decisions, the Few-Shot Multimodal Explanation Visual Question Answering (FS-MEVQA) task has been proposed, requiring models to generate multimodal explanations rooted in specific visual regions using a small number of training samples while providing answers. However, existing techniques are mainly based on an "answer-oriented" paradigm, treating explanation generation as a byproduct of predicting the final answer, such as... Figure 2 As shown, this leads to two key flaws: Severe lack of visual evidence: When generating explanations, the system relies heavily on the final answer and often ignores the reference entities related to the answer, resulting in a break in the reasoning chain when answering combinatorial questions.
[0003] Problem-sample mismatch: Assigning the same few samples to all problems ignores the potential differences in logical dependencies between different problems (such as spatial relationships and isolated attributes), causing the model to apply the wrong inference template. Summary of the Invention
[0004] To overcome the shortcomings of the prior art, the present invention provides a generation system and method for visual question answering with few samples and multimodal interpretation, so as to solve the problems of lack of visual evidence and problem-sample logic mismatch in the prior art.
[0005] To achieve the above objectives, the present invention adopts the following technical solution, including: A method for generating visual question answering for few-sample, multimodal interpretation includes the following steps: S1 receives the input image and the corresponding question, and uses the answering agent to predict the initial answer; S2, Based on the question and the initial answer, a local search graph is constructed using a graph agent, and the local search graph extracts visual entity nodes and the edges between the nodes; S3, Based on the local search graph, the detection agent performs visual existence verification and bounding box localization on the input image, removes non-existent nodes and relation edges, and generates a fact update search graph; S4, based on an adaptive sample selector, integrates textual semantic similarity and graph structure similarity to retrieve samples from the sample library that have the same potential logical dependencies as the current problem; S5, based on the question, the initial answer, the fact update search graph, and the retrieved examples, the interpreting agent generates a multimodal interpretation containing visual bounding box anchors.
[0006] Preferably, before constructing the local search graph in step S2, a dynamic routing mechanism is also included: Assess whether the initial answer contains the specific visual evidence required to constitute an explanation; If the initial answer is a confirmatory response that does not contain specific visual evidence, then the question will only be used as text input for graph construction. If the initial answer introduces specific visual evidence, then the question and the initial answer are concatenated as the text input for graph construction.
[0007] Preferably, the specific visual evidence is a visual entity.
[0008] Preferably, in step S3, generating the fact update search graph specifically includes: Visual existence check: For each node in the local search graph, verify whether it physically exists in the input image within the context defined by the relation edges; Visual localization: Extract the spatial bounding box coordinates only for nodes that are verified to exist; Search graph update: For the local search graph, prune the nodes that are missing verifications and their dependent relationship edges, and assign the obtained bounding box coordinates to the nodes that are verified to exist, thus forming the fact update search graph.
[0009] Preferably, in step S4, the scoring mechanism for retrieving samples from the sample library includes: Use a text encoder to calculate the textual semantic similarity between the current problem and the candidate sample problems; Extract the entity set and relation edge set of the current fact update search graph and the candidate sample local search graph, calculate the Jaccard similarity of the entity set and the Jaccard similarity of the relation edge set respectively, and sum the two by weight to obtain the graph structure similarity. The text semantic similarity and the graph structure similarity are weighted and fused to obtain the final retrieval score. The candidate samples are sorted in descending order of retrieval score, and the top k samples with the highest scores are selected.
[0010] Preferably, the response agent, graph agent, detection agent, and explanation agent are all independent pre-trained neural network sub-models.
[0011] This invention provides a generation system for few-shot, multimodal interpretable visual question answering, applicable to the aforementioned method for generating few-shot, multimodal interpretable visual question answering. The system includes: Answer prediction module: Used to receive the input image and the corresponding question, and predict the initial answer; Graph-guided evidence verification module: used to parse the question and the initial answer into a local search graph containing visual entity nodes and edge relationships between nodes, and to perform visual existence verification and bounding box localization on the input image to generate a fact update search graph; Adaptive Sample Retrieval Module: This module retrieves samples from the sample library that have the same underlying logical dependencies as the current problem by evaluating text semantic similarity and local graph structure similarity. Multi-modal interpretation generation module: Used to generate multimodal interpretations based on verified visual facts and retrieved examples.
[0012] The present invention provides a readable storage medium having a computer program stored thereon, which, when executed, implements the aforementioned method for generating visual question answering for few-sample multimodal interpretation.
[0013] The present invention provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for generating visual question answering for few-sample multimodal interpretation.
[0014] This invention provides a computer program product comprising a computer program / instruction that, when executed by a processor, implements the aforementioned method for generating visual question answering for few-sample, multimodal interpretation.
[0015] The advantages of this invention are: (1) This invention proposes an explanation-centered paradigm that avoids visual illusions and omissions of evidence in the generation process through structured graph construction.
[0016] (2) The present invention introduces a graph-guided visual retrieval and bounding box enhancement mechanism to ensure that the interpretation is based on real visual facts.
[0017] (3) The present invention designs an adaptive sample selector that integrates text semantics and graph structure features to dynamically retrieve the best few-sample demonstration context.
[0018] (4) This invention introduces an explanation-centered paradigm. By constructing a training-free collaborative multi-agent framework (CMAF), the problem is transformed into a local search graph for rigorous visual verification. Combined with reference examples aligned with dynamic retrieval logic, a high-quality multimodal explanation is finally generated.
[0019] (5) The system of the present invention includes a graph-guided evidence verification module and an adaptive sample selection module. The graph-guided evidence verification module is used to actively parse the question and the initial answer into a local search graph, and to perform strict visual existence verification and bounding box localization on the input image to ensure that the generated explanation is anchored to real and complete visual facts. The adaptive sample selection module is used to dynamically retrieve reference samples by fusing text semantics and graph structure similarity, and to provide precise guidance for the generation of explanations that have the same potential logical dependencies as the current question, thereby synthesizing highly coherent multimodal explanations.
[0020] (6) The present invention can effectively solve the defects of lack of visual evidence and problem-example logic mismatch in the prior art. The effectiveness of the present invention system has been verified on public datasets. Its performance is significantly better than the existing baseline methods, and it significantly improves the accuracy of answer prediction and the quality of multimodal interpretation generation. It has broad application prospects.
[0021] (7) This invention improves the credibility of visual question answering (VQA) models, making them easier for manual verification. Users not only see the answer but also which region of the image the model points to as evidence. It is applicable to medical image interpretation, industrial defect detection, and autonomous driving perception question answering, where the correctness of the evidence must be verified in high-risk scenarios.
[0022] (8) The multimodal interpretation generation of the present invention helps to quickly locate the error type of the model and accurately locate the fault link. For example, incorrect bounding box positioning belongs to the visual detection problem; correct object recognition but incorrect reasoning logic belongs to the reasoning logic defect; fabricated image without entity belongs to the hallucination problem. Attached Figure Description
[0023] Figure 1 This is a flowchart of the method of the present invention.
[0024] Figure 2 This is a schematic diagram illustrating two shortcomings of existing methods.
[0025] Figure 3 This is a schematic diagram illustrating two innovative aspects of the method of this invention.
[0026] Figure 4 This is a complete block diagram of the method of an embodiment of the present invention. Detailed Implementation
[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0028] Example 1 like Figure 1 As shown, this invention provides a method for generating visual question answering for few-sample, multimodal interpretation, comprising the following steps: S1, Initial Answer Prediction: Receive the input image and the corresponding question, and use the answering agent to predict the initial answer.
[0029] Among them, the response agent (Multimodal Large Language Model MLLM) is combined with the input image. and input issues Predict the initial answer : ; in, This represents the responding agent; To respond to the agent's prompt parameters.
[0030] S2, Local search graph construction: Based on the question and the initial answer, a local search graph is constructed using a graph agent (Large Language Model LLM), which extracts key entity nodes and their combined relationship edges.
[0031] A dynamic routing mechanism is included before constructing the local search graph, introducing indicator functions. Used to evaluate the initial answer Does it contain the specific visual evidence required to constitute a multimodal interpretation? If the initial answer is a confirmatory response that does not contain specific visual evidence, then the question is used only as textual input for graph construction. For example, in binary confirmatory questions (such as "yes / no"), the complete spatial context is already contained within the question. In the middle, at this time Then the initial answer will be hidden. At this point, the text input for the graph agent is .
[0032] If the initial answer introduces a visual entity, then the question and the initial answer are concatenated as the text input for graph construction. For example, if it is an open-ended discovery question (e.g., asking "What is to the left of a man?" and answering "dog"), the answer introduces a new core entity. Then the problem Compared to the initial answer Concatenation, where the text input of the graph agent is... ;in, Indicates splicing.
[0033] This dynamic routing mechanism effectively prevents "reasoning shortcuts" and evidence omissions caused by the model knowing the confirmatory answer in advance.
[0034] In step S2, a local search graph is constructed using graph agents. : Graph Agent Enter text Transform into a structured graph : ; Among them, the node set Includes all extracted visual entities; set of relation edges It includes compositional or spatial dependencies, and the relation edges are represented as directed triples. , , Represented as the i-th and j-th nodes (visual entities). Represents a node , Relationship; This indicates that the input is given to the graph agent. Instructions and requirements.
[0035] S3, Graph-guided evidence verification: Based on the local search graph, a detection agent (Multimodal Large Language Model MLLM) is used to perform rigorous visual existence verification and bounding box localization on the input image, eliminating non-existent nodes and relationships, and generating a fact update search graph.
[0036] In step S3, the step of generating the fact update search graph specifically includes: S31, Visual Existence Check: For each entity node in the local search graph, verify whether it physically exists in the input image within the context defined by the relation edges.
[0037] Local search graph This serves as a "verification checklist." For each node in the image... Detecting the intelligent agent in the input image The set of relation edges Within the defined relational context, perform a binary existence check: .in, Indicates the detection agent, This represents the existence value of node v, indicating the number of existing nodes. Assign 1 to the missing node. Assign 0.
[0038] S32, Visual Positioning: Extract the precise spatial bounding box coordinates only for entities that are verified to exist.
[0039] For those that have passed the existence verification (i.e.) Extract accurate 2D spatial bounding box coordinates for node 1). .in, These are the coordinates of the bottom left and top right corners of the bounding box, respectively.
[0040] S33, Search Graph Update: Strictly prune missing nodes and their dependent edges, and assign the obtained bounding box coordinates to the verified existing nodes to form a fact-updated search graph. .
[0041] For the original local search graph Perform rigorous pruning. Filter out all. The nodes and their attached edges are then extracted. Subsequently, the successfully extracted bounding box coordinates are... Bind to the corresponding node whose existence verification has passed, and generate a fact update search graph. ;in, For the updated set of nodes, This is the updated set of relation edges.
[0042] This mechanism forms the core of the invention's "fact-based explanation generation" principle.
[0043] S4, Adaptive Sample Selection: Based on an adaptive sample selector, it integrates textual semantic similarity and graph structural similarity to retrieve a small number of samples (sample examples) from the sample library that have the same potential logical dependencies as the current problem.
[0044] To overcome the shortcomings of traditional methods where static sample allocation leads to logical mismatches, this invention designs an Adaptive Sample Selector (AES) to dynamically retrieve the top k demonstration samples. To find logically isomorphic reference samples, the model simultaneously calculates semantic similarity and graph structure similarity.
[0045] In S4, the scoring mechanism for retrieving fewer samples includes: S41, use a text encoder to calculate the textual semantic similarity between the current problem and the candidate sample problems.
[0046] Text semantic similarity calculation: using a text encoder (such as the BGE-M3 model, denoted as...) ), to the current problem The problem of samples in the candidate pool Map to dense vectors and calculate cosine similarity. As text semantic similarity .
[0047] S42, extract the entity set and relation edge set of the current fact update search graph and the candidate sample local search graph, calculate the Jaccard similarity of the entity set and the Jaccard similarity of the relation set respectively, and sum the two by weight to obtain the graph structure similarity.
[0048] Graph structure similarity calculation: To capture deep reasoning logic (such as spatial reasoning), further compare the search graphs of the two (current fact update search graph). Local search graph of candidate samples First, the relational predicates and entity synonyms are standardized and aligned; then, the current facts are extracted to update the search graph. Standardized aligned entity set and relation edge set Local search graph of candidate samples Standardized aligned entity set and relation edge set Then calculate and Jaccard similarity and calculation and Jaccard similarity Finally, the weighted similarity of the graph structures is obtained. ,in, , These are the weights for entity similarity and relation edge similarity, respectively.
[0049] S43, the text semantic similarity and the graph structure similarity are weighted and fused to obtain the final retrieval score. The candidate samples are sorted in descending order according to the retrieval score, and the top k samples with the highest scores are selected.
[0050] Comprehensive search scoring: Introducing a balanced hyperparameter Calculate the final retrieval score Based on the scores, sort them in descending order, and extract the top k samples (in a preferred embodiment, k is 8) to form the contextual prompts. .
[0051] S5, Multimodal Explanation Generation: Based on the question, the initial answer, the fact update search graph, and the retrieved few examples, a multimodal explanation containing visual bounding box anchors is generated using an explanation agent (Large Language Model LLM).
[0052] Explain the intelligent agent Receive current question Initial answer Fact update search graph with bounding box coordinates and examples of contextual hints for structural height alignment. .
[0053] Under these multiple conditions, seamless model synthesis can explain the multimodal interpretation of the reasons behind the model's predicted answers. : ; Because the reference examples provide perfectly matching logical guidance, and the visual facts are verified and injected in advance, the generated explanations not only conform to the natural language syntax, but each visual evidence also carries precise bounding box coordinate anchoring, thus solving the defects of missing visual evidence and problem-example mismatch.
[0054] In this invention, the response agent, graph agent, detection agent, and explanation agent are independent pre-trained neural network sub-models. Specifically, the response agent and detection agent use the Qwen3.5-8B model, while the graph agent and explanation agent use the Qwen3-4B model.
[0055] like Figure 3 As shown, this invention provides a method for generating visual question answering for multimodal interpretation with few samples, solving the problems of missing visual evidence and logical mismatch between questions and examples in the prior art.
[0056] like Figure 4 As shown, in this embodiment of the invention, the input image is a picture of a ski resort, and the corresponding input question is "Is the man to the right of the skis wearing a tie?". The initial answer is predicted to be "no" using an answering agent. A graph agent is used to construct a local search graph based on the question and the initial answer, extracting key entity nodes and their combination relationship edges. A detection agent is used to perform visual evidence verification on the image (visual existence verification and bounding box localization), eliminating non-existent nodes and relationships, updating the search graph, and generating a fact-updated search graph. An adaptive sample selector is used to perform adaptive sample selection, fusing text semantic similarity (text semantic similarity between the current question and candidate sample questions) and graph structure similarity (graph structure similarity between the current fact-updated search graph and the candidate sample local search graph) to obtain the final retrieval score, and selecting the top k samples. An explanation agent is used to generate a multimodal explanation containing visual bounding box anchors based on the question, the initial answer, the fact-updated search graph, and the selected top k samples.
[0057] Example 2 This invention also provides a generation system for visual question answering with few samples and multimodal interpretation, comprising: The answer prediction module receives the input image and natural language question, and uses a visual language model to predict the initial answer; The graph-guided evidence verification module is used to parse the question and the initial answer into a local search graph containing visual entity nodes and combination relationship edges, and to perform strict visual existence verification and bounding box localization on the input image to generate a fact update search graph. The adaptive sample selection module is used to perform fusion calculations based on the textual semantic similarity and graph structure similarity between the current problem and candidate samples, and retrieve the fewest samples that are most relevant to the underlying reasoning logic of the current problem; The multimodal explanation generation module is used to update the search graph, retrieved few samples, the question, and the initial answer based on the facts, and generate a multimodal explanation that includes text explanation and corresponding visual positioning boxes.
[0058] Example 3 In this embodiment, examples are used to demonstrate that the proposed method and system for generating visual question answering for few-sample multimodal interpretation is effective.
[0059] Experiments were conducted on the publicly available SME benchmark dataset, and the results are shown in Table 1. Answer prediction performance was measured using answer accuracy (Acc), the quality of the generated multimodal explanations was measured using five common natural language automation evaluation metrics (BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE), and the richness of the visual regions included in the multimodal explanations was measured using detection accuracy and evidence recall. Visual question answering generation was performed using existing methods such as REX, VCIN, GLM-4V, Qwen-VL, Gemini-1.5, GPT-4V, SME, and the method of this invention. Among these methods, the SME method is the state-of-the-art.
[0060] Table 1 shows that the present invention achieves optimal results in both answer accuracy (Acc) and multimodal explanation generation metrics (such as CIDEr and SPICE). In particular, the structured graph-guided visual localization significantly improves localization accuracy and evidence recall metrics, demonstrating that the explanations provided by the present invention contain more complete visual evidence, and that the quality of the explanations generated by the present invention is higher when guided by appropriate examples.
[0061] Table 1. Experimental results of each method on the dataset.
[0062] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for generating visual question answering for few-sample, multimodal interpretation, characterized in that, Includes the following steps: S1 receives the input image and the corresponding question, and uses the answering agent to predict the initial answer; S2, Based on the question and the initial answer, a local search graph is constructed using a graph agent, and the local search graph extracts visual entity nodes and the edges between the nodes; S3, Based on the local search graph, the detection agent performs visual existence verification and bounding box localization on the input image, removes non-existent nodes and relation edges, and generates a fact update search graph; S4, based on an adaptive sample selector, integrates textual semantic similarity and graph structure similarity to retrieve samples from the sample library that have the same potential logical dependencies as the current problem; S5, based on the question, the initial answer, the fact update search graph, and the retrieved examples, the interpreting agent generates a multimodal interpretation containing visual bounding box anchors.
2. The method for generating visual question answering for few-sample, multimodal interpretation according to claim 1, characterized in that, Before constructing the local search graph in step S2, a dynamic routing mechanism is also included: Assess whether the initial answer contains the specific visual evidence required to constitute an explanation; If the initial answer is a confirmatory response that does not contain specific visual evidence, then the question will only be used as text input for graph construction. If the initial answer introduces specific visual evidence, then the question and the initial answer are concatenated as the text input for graph construction.
3. The method for generating visual question answering for few-sample, multimodal interpretation according to claim 2, characterized in that, The specific visual evidence is a visual entity.
4. The method for generating visual question answering for few-sample, multimodal interpretation according to claim 1, characterized in that, In step S3, generating the fact update search graph specifically includes: Visual existence check: For each node in the local search graph, verify whether it physically exists in the input image within the context defined by the relation edges; Visual localization: Extract the spatial bounding box coordinates only for nodes that are verified to exist; Search graph update: For the local search graph, prune the nodes that are missing verifications and their dependent relationship edges, and assign the obtained bounding box coordinates to the nodes that are verified to exist, thus forming the fact update search graph.
5. The method for generating visual question answering for few-sample, multimodal interpretation according to claim 1, characterized in that, In step S4, the scoring mechanism for retrieving samples from the sample library includes: Use a text encoder to calculate the textual semantic similarity between the current problem and the candidate sample problems; Extract the entity set and relation edge set of the current fact update search graph and the candidate sample local search graph, calculate the Jaccard similarity of the entity set and the Jaccard similarity of the relation edge set respectively, and sum the two by weight to obtain the graph structure similarity. The text semantic similarity and the graph structure similarity are weighted and fused to obtain the final retrieval score. The candidate samples are sorted in descending order of retrieval score, and the top k samples with the highest scores are selected.
6. The method for generating visual question answering for few-sample, multimodal interpretation according to claim 1, characterized in that, The response agent, graph agent, detection agent, and explanation agent are all independent pre-trained neural network sub-models.
7. A generative system for visual question answering with few samples and multimodal interpretation, characterized in that, A method for generating visual question answering for few-sample, multimodal interpretation, applicable to any one of claims 1-6, comprising: Answer prediction module: Used to receive the input image and the corresponding question, and predict the initial answer; Graph-guided evidence verification module: used to parse the question and the initial answer into a local search graph containing visual entity nodes and edge relationships between nodes, and to perform visual existence verification and bounding box localization on the input image to generate a fact update search graph; Adaptive Sample Retrieval Module: This module retrieves samples from the sample library that have the same underlying logical dependencies as the current problem by evaluating text semantic similarity and local graph structure similarity. Multi-modal interpretation generation module: Used to generate multimodal interpretations based on verified visual facts and retrieved examples.
8. A readable storage medium, characterized in that, It stores a computer program, which, when executed, implements a method for generating visual question answering for few-sample multimodal interpretation as described in any one of claims 1-6.
9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the generation method for visual question answering based on any one of claims 1-6.
10. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the generation method for visual question answering oriented towards few-sample multimodal interpretation as described in any one of claims 1-6.