Multimodal multi-hop question answering method and apparatus, system, storage medium

Abstract

The application discloses a multi-modal multi-hop question and answer method and device, system and storage medium, which combines a reinforcement learning teacher with an implication tree, distills a student model to improve the generation quality of the implication tree, and introduces multi-path generation to improve the generalization capability; further, a multi-modal expert is used to iteratively reason on sub-questions in the implication tree, and continuously update intermediate answers. By adopting the technical scheme of the application, the multi-modal multi-hop question and answer accuracy and the implication tree generation quality are improved.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a multimodal multi-hop question-answering method, apparatus, system, and storage medium. Background Technology

[0002] Multimodal multi-hop question answering (MQA) is a highly challenging task in the field of artificial intelligence. It requires systems to answer complex questions through multi-step logical reasoning across multiple data sources, including text, images, and tables. Visual QA was first proposed to answer questions based solely on images (i.e., purely visual input). Later, WebQA and MultimodalQA required combining information from free text, semi-structured tables, and images to answer multi-hop reasoning questions.

[0003] In addressing the challenges of MMQA, one approach involves training separate QA models for different modalities and breaking down the question into several sub-questions for step-by-step reasoning. While this approach is simple and straightforward for solving MMQA, it can struggle when answering questions requiring cross-modal reasoning because there is no established interaction between the models used. Furthermore, information loss can occur during step-by-step reasoning, leading to poor question-answering performance. Another approach advocates developing a single multimodal model capable of simultaneously processing inputs from different sources and modalities to produce the final answer to the question. By incorporating strategies such as vision-language pre-training, multimodal models can gain a more nuanced understanding of multimodal context, enhancing their cross-modal reasoning capabilities and contributing to improved question-answering performance. However, this approach requires extensive pre-training and fine-tuning to adapt to different inputs. To address the complexities associated with multimodal reasoning, such methods first transform multimodal knowledge into a unified text representation. By converting different modalities into a common language, coherent and standardized inputs are ensured for subsequent processing. Following the unification process, text-based methods are used to perform retrieval and question-answering tasks.

[0004] MURAG designed a multimodal converter architecture to accept text and image feature inputs and built a dataset of millions of data points for pre-training models. Liu et al. used an image captioning model and a table linearization method to unify multimodal information into text, proposing a new multimodal question-answering paradigm. However, the lack of constraints in this process resulted in significant information redundancy, impacting model performance. SKURG designed an entity fusion method to answer multimodal questions. Hu et al. trained an image description model to generate text descriptions for images and input these descriptions into a large language model, enabling it to understand image content and generate answers. Liu et al. used the multimodal large model LLaVA to generate more accurate image descriptions and then constructed different context learning templates for each modality, allowing GPT-3 to demonstrate its powerful performance in this task. Zhang et al. were the first to incorporate entailment trees into multimodal question answering, using entailment trees to improve the interpretability of reasoning, but the low quality of entailment trees also remained a problem. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a multimodal multi-hop question answering method, apparatus, system, and storage medium, which solves the technical problems existing in the prior art in multimodal multi-hop question answering tasks, such as low quality of implication tree generation, poor logical coherence, insufficient interpretability, and loss of key visual information and semantic deviation caused by converting multimodal information into text.

[0006] To achieve the above objectives, the present invention provides the following solution:

[0007] A multimodal, multi-hop question answering method includes:

[0008] Step S1: Determine whether the input problem needs to be decomposed. If so, decompose it into sub-problems. Based on the decomposed problem, retrieve evidence from the existing text evidence library and image evidence library to obtain evidence related to the problem.

[0009] Step S2: Based on the evidence and questions obtained from the retrieval, the reinforcement learning teacher is introduced into the implication tree generation training, and the teacher model's capabilities are transferred to the lightweight student model through distillation.

[0010] Step S3: Use the student model to generate implication trees through multiple paths, and use the implication tree optimal selection module to evaluate and select the optimal implication tree;

[0011] Step S4: Using the selected implication tree as the reasoning framework, perform cross-modal alignment and semantic aggregation on the intermediate answers through multiple rounds of iteration;

[0012] Step S5: After the multimodal expert iteration is completed, the obtained intermediate node answers are backfilled layer by layer into the corresponding sub-problem nodes in the implication tree, and then input into the large model to answer, thus obtaining the final answer.

[0013] As a preferred option, in step S3, the student model is used to generate an implication tree through three complementary reasoning strategies: divide and conquer, chain thinking of answer plans, and dynamic synthesis of examples.

[0014] As a preferred option, in step S4, the selected implication tree is used as the reasoning framework. Each sub-problem node is traversed, the corresponding modality expert is dynamically selected to handle each sub-problem, and cross-modal alignment and semantic aggregation of the intermediate answer are performed through multiple rounds of iteration.

[0015] The present invention also provides a multimodal multi-hop question-answering device, comprising:

[0016] The first processing module is used to determine whether the input question needs to be decomposed. If so, it decomposes the question into sub-questions. Based on the decomposed questions, it retrieves evidence from the existing text evidence library and image evidence library to obtain evidence related to the question.

[0017] The second processing module is used to introduce reinforcement learning teachers into implication tree generation training based on the retrieved evidence and questions, and transfer the teacher model capabilities to the lightweight student model through distillation.

[0018] The third processing module is used to generate implication trees using the student model through multiple paths, and to evaluate and select the optimal implication tree using the optimal implication tree selection module.

[0019] The fourth processing module is used to perform cross-modal alignment and semantic aggregation of intermediate answers through multiple rounds of iteration, using the selected implication tree as the reasoning framework.

[0020] The fifth processing module is used to backfill the intermediate node answers obtained after the multimodal expert iteration is completed to the corresponding sub-problem nodes in the implication tree, input them into the large model for answering, and obtain the final answer.

[0021] As a preferred option, the third processing module is used to generate an implication tree using three complementary reasoning strategies: divide and conquer, chain-like thinking of answer plans, and dynamic synthesis of examples.

[0022] As a preferred option, the fourth processing module is used to traverse each sub-problem node using the selected implication tree as the reasoning framework, dynamically select the corresponding modality expert to handle each sub-problem, and perform cross-modal alignment and semantic aggregation on the intermediate answer through multiple rounds of iteration.

[0023] The present invention also provides a multimodal multi-hop question answering system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a multimodal multi-hop question answering method when executed by the processor.

[0024] The present invention also provides a storage medium storing a computer program, which executes a multimodal multi-hop question-answering method when running.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] This invention overcomes the limitation of existing models in generating high-quality inference paths by introducing a reinforcement learning-based teacher model guidance mechanism, resulting in an implication tree with a more rigorous logical structure and semantic coherence. It enhances the robustness of complex reasoning through multi-path generation and optimal path selection strategies, abandoning the traditional approach of completely compressing visual information into text. Instead, it preserves fine-grained multimodal information through a multimodal expert iterative response mechanism, ensuring that the question-and-answer process fully utilizes multimodal details to generate more accurate answers. Attached Figure Description

[0027] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the multimodal multi-hop question-answering method according to an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram illustrating the multi-path implication tree generation and multimodal expert response in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Example 1

[0033] like Figure 1 As shown, the present invention provides a multimodal multi-hop question-answering method, including:

[0034] Step S1, Input and Preprocessing: First, determine whether the input problem needs to be decomposed (e.g., involving multi-hop or multimodal reasoning). If so, decompose it into sub-problems. Then, based on the decomposed problem, retrieve evidence from existing text and image evidence databases to obtain evidence relevant to the problem.

[0035] Step S2: Reinforcement Learning for the Teacher and Distillation of the Student Model: The retrieved evidence and questions are combined into evidence pairs as input to construct training data. The teacher model enhances its ability to generate implication tree reasoning processes through reinforcement learning. Then, the high-quality implication tree reasoning processes and evidence pairs generated by the teacher model are used as supervision signals to distill and train the student model for subsequent implication tree generation.

[0036] Step S3: Multi-path generation of implication trees: The student model is used to generate implication trees through multiple paths. Three complementary reasoning strategies are adopted, including divide and conquer, chain-plan thinking, and dynamic example generation. Then, the optimal implication tree is evaluated and selected using the optimal implication tree selection module.

[0037] Step S4, Multimodal Expert Iteration: Using the selected implication tree as the reasoning framework, traverse each sub-problem node, dynamically select the corresponding modal expert (text expert and image expert) to handle each sub-problem, and perform cross-modal alignment and semantic aggregation on the intermediate answer through multiple rounds of iteration;

[0038] Step S5: Update the implication tree and generate the final answer: After the multimodal expert iteration is completed, the obtained intermediate node answers are backfilled layer by layer into the corresponding sub-problem nodes in the implication tree, and then input into the large model to answer, and obtain the final answer.

[0039] As one embodiment of the present invention, in step S1, given the original problem... This invention first uses a binary classification model to determine whether a problem needs to be decomposed. This model is based on BERT and uses labeled data for supervised learning during the training phase. If the problem is supported... If the number of evidence sources is greater than or equal to two (i.e., involving multiple independent facts or modalities), it is marked as "decomposable" (i.e., a multi-hop problem); otherwise, it is marked as "indecomposable" (a single-hop problem), as shown in Formula 1. This design ensures that the decomposition decision is aligned with the task complexity, avoiding the introduction of unnecessary reasoning overhead for simple problems. Then, for sub-problem decomposition, this invention designs prompts to guide LLM in decomposing complex problems. This invention defines a problem decomposition chain, which is a sequence of sub-problems and corresponding entities separated by some special labels. The original problem... The input is fed into a large model, and context-learning prompts are used to guide the large model to generate two or more sub-problems. As shown in Formula 2:

[0040] ,

[0041] ,

[0042] in It's the original question. It's a prompt. Subproblems

[0043] Next, this invention constructs a multimodal evidence library to store relevant text, image, and tabular evidence retrieved from heterogeneous data sources. The construction of this evidence library is guided by the problem decomposition results, with each sub-problem... Each of these serves as a query signal, and targeted searches are performed in the corresponding modal spaces to ensure that the collected evidence is highly aligned with the reasoning path.

[0044] For different modalities, this invention designs differentiated representation and storage mechanisms. Textual evidence: The original paragraph is directly retained, and relevant fragments are recalled through the aforementioned hybrid retrieval strategy. Tabular evidence: Structured table content is converted into fluent and readable sentences using predefined natural language templates, allowing it to seamlessly integrate into the text semantic space and participate in unified retrieval. Image evidence: This invention simultaneously retains the original image and its image title generated by the multimodal large model. The title is used for efficient text-image cross-modal retrieval, while the original image is fully retained in the evidence base for direct use by the subsequent multimodal expert module. This design avoids the loss of detail caused by traditional methods that completely compress visual information into text descriptions, allowing the model to still access rich visual cues (such as spatial relationships, local object attributes, and chart structures) during the response phase.

[0045] To improve search performance, this invention employs a hybrid search strategy, combining sparse and dense search. Sparse search offers high accuracy when processing queries containing explicit keywords or entities, while dense search effectively captures deep semantic relationships and is suitable for expressing diverse or semantically complex sub-questions. By weightedly fusing the results of the two search methods, more comprehensive and accurate evidence is provided for subsequent reasoning stages.

[0046] Furthermore, the input to the tree generation task consists of the premises. (Evidence in the evidence base) and assumptions (Original Problem) Composition. The goal of this invention is to generate an implication tree. By using corpus Selecting some premises as building blocks, and then applying them to the assumptions. Explanation follows. An implication tree is represented as a quadruple. leaf nodes It was retrieved from the corpus (i.e. ), internal tree nodes This is an intermediate conclusion (i.e., the corpus). Content that does not exist in the model, intermediate conclusions are usually generated by a large language model, and It is a series of implied reasoning steps used to explain the hypothesis. In a tree structure, assume Always serving as the root node and the final conclusion, the implication tree is introduced into multimodal, multi-hop question-and-answer systems. By explicitly constructing hierarchical reasoning from original evidence to the final answer, it not only clearly demonstrates the intermediate conclusions and supporting evidence at each step, but also provides a structured and interpretable logical framework for complex cross-modal reasoning.

[0047] Furthermore, this invention optimizes the implicit tree format by explicitly embedding sub-question nodes into the tree structure, resulting in the following structure: This allows it to not only record "how to deduce," but also clearly present "why the derivation is made this way." This improvement enables implication trees to more precisely characterize the hierarchical reasoning process in multi-hop, multimodal question answering, while providing actionable structured supervision signals for subsequent sub-answer generation, error localization, and model iterative optimization.

[0048] In one embodiment of the present invention, in step S2, the reinforcement learning teacher is as follows:

[0049] This invention combines Reinforcement-Learned Teacher (RLT) with implication tree generation, with the core objective of distilling a high-performance, deployable student model. Unlike the traditional "debate" paradigm, the optimization objective of the teacher model is redefined as: given a problem-evidence pair and the corresponding true implication tree, generating a clear, logically coherent, and easily replicable step-by-step reasoning explanation path for the student model. In this process, the real-time reasoning performance of the student model is used as a reward signal for reinforcement learning, thus forming a closed-loop feedback loop.

[0050] To support effective training of reinforcement learning teachers (RLT), this invention constructs a high-quality multimodal entailment tree corpus. First, question-evidence pairs with clear multi-step inference chains are selected from a multi-hop question-answering benchmark dataset. Then, this invention employs an in-context learning (ICL) strategy to guide the generation of intermediate inference steps in a structured format, displaying the complete entailment tree that conforms to tree-like dependencies. To ensure the logical correctness and structural compliance of the generated results, this invention introduces a dual quality control mechanism:

[0051] (1) Automatic rule filtering: It forces all leaf nodes to come from the original evidence set, prohibits circular reasoning or the introduction of external knowledge, and verifies the connectivity and hierarchical rationality of the tree;

[0052] (2) Manual review and sampling: Domain experts conduct random checks on the generated samples to evaluate their semantic consistency, logical completeness and modal alignment.

[0053] Ultimately, this invention constructs a large-scale training corpus containing 5,000 multimodal entailment tree samples, covering various evidence types including text, table paraphrasing, and images. This corpus provides a reliable initial supervision signal for RLT. The training samples are triples. ,in As evidence, For the question, This is a standard and correct implication tree.

[0054] To drive the teacher model to generate implication tree explanations that are easy for students to understand, this invention uses a two-layer reward function to guide the teacher model in generating logically coherent explanations that are easy for the student model to understand. The student model is then fixed. (Qwen2.5-3B-Instruct) acts as the evaluator, providing differentiable evaluation of each step of the teacher's reasoning output. The reward consists of the following two parts:

[0055] (1) Student Comprehension Rewards )

[0056] Measuring the student model's performance in reading the implication tree reasoning process generated by the teacher model. Then, the confidence level of the prediction for the correct conclusion is calculated. Specifically, only forward propagation is performed on the student model, and its prediction of the standard answer is taken. The token log probability measures the probability that the student model predicts the correct answer after seeing the reasoning process generated by the teacher. The formula is as follows:

[0057] ,

[0058] in It is a student model. That is the correct answer. It is the reasoning process that generates the teacher model. That's a problem.

[0059] (2) Explain the nature reward ( This invention prevents teachers from using expressions that are difficult for students to understand. For the same reasoning passage, the invention addresses teacher conditions (known...) With the answer ) and student conditions (only known) The distributions are calculated separately below, and regularization is performed using the mean and the maximum KL divergence:

[0060] (4)

[0061] in It is a teacher model. It is a student model. That is the correct answer. It is the reasoning process that generates the teacher model. This is the problem. Finally, the reward for RLT is obtained by combining these two terms with the weighting coefficient λ:

[0062] (5)

[0063] The reward function is fully differentiable, providing feedback to the teacher model at each step, thereby guiding it to generate an implication tree interpretation that is both logically faithful and easy for students to understand.

[0064] As one embodiment of the present invention, in step S2, the distillation student model is as follows:

[0065] High-quality implication tree inference paths are generated from a large amount of unlabeled or weakly labeled data using a pre-trained teacher model. A distillation dataset is constructed. The lightweight student model is supervised and fine-tuned using a standard language modeling loss function. Through this step, the student model learns the ability to generate structured implication trees.

[0066] Its loss function is:

[0067] (6)

[0068] in It's a problem. For the final conclusion, It is the reasoning process that generates the teacher model.

[0069] As one embodiment of the present invention, in step S3, to avoid the problem of error propagation or decreased generalization ability caused by the student model's over-reliance on a single, potentially suboptimal, or biased inference path during the reinforcement learning-driven distillation process, the present invention proposes a multi-path entailment tree generation mechanism. This mechanism explores potential inference structures in parallel through three complementary inference paths to improve the diversity and robustness of the generated trees. The first is a divide-and-conquer method, the second is a query plan chaining approach, and the third is dynamic example generation. After generating the entailment tree through multiple paths, an optimal entailment tree selector is used to select the optimal entailment tree for subsequent use.

[0070] The divide-and-conquer approach: This method uses sub-problems, generating a corresponding sub-implementation tree for each sub-problem; then, these subtrees are logically merged to construct the complete implication tree. Finally, the tree structure is simplified through post-processing optimization steps (such as deleting redundant intermediate nodes and merging duplicate reasoning paths). This strategy is particularly effective when dealing with nested or multi-level, multi-hop question-and-answer problems, significantly reducing reasoning complexity.

[0071] Chain-like thinking in answering questions: Chain-like thinking in answering questions refers to the steps that need to be followed when answering questions. This invention introduces an explicit reasoning planning mechanism. First, a high-level answer plan is generated, which clarifies the key steps required to answer the question and their dependent order (e.g., "first identify object A, then combine the text to determine its attribute B, and finally deduce conclusion C"). Subsequently, this plan is used as a guiding signal to constrain the generation process of the implication tree, ensuring that each step of reasoning strictly follows the preset logical flow, thereby improving the coherence of the reasoning chain and its task relevance.

[0072] Dynamic Example Generation: Inspired by the success of Few-shot In-Context Learning, this invention designs an online example generation mechanism: Utilizing a Large Language Model (LLM), several high-quality examples are dynamically synthesized based on the current question and evidence, and then injected into the prompt template. These synthesized examples are highly aligned with the target task in terms of format, granularity, and inference depth, effectively guiding the model to follow the correct structured output specifications and significantly improving the accuracy and completeness of the generated implication tree.

[0073] (7)

[0074] in Representative evidence library, This indicates a prompt. Representative issues.

[0075] After completing the multi-path generation, this invention introduces an optimal implication tree selector, which reorders the candidate trees based on multi-dimensional scoring criteria (including logical consistency, evidence coverage, and implication strength of nodes), and selects the implication tree with the highest comprehensive score as the reasoning architecture for solving subsequent sub-problems and generating answers.

[0076] (8)

[0077] Where Q is the original problem. It is a fact database. Representative candidate pool There are K implication trees inside.

[0078] In one embodiment of the present invention, steps S4 and S5 involve answering each sub-question contained in the previously generated implication tree, since the generated implication tree contains sub-questions, and each sub-question corresponds to evidence. The present invention uses a multimodal expert iterative method, categorizing evidence into image question answering experts and text question answering experts based on different modalities. After generating sub-question answers, these answers replace the intermediate conclusions in the original implication tree, ultimately forming an implication tree containing intermediate answers for the final answer.

[0079] Combined image question answering and text question answering: First, the image modality and text modality are encoded using different encoders.

[0080] (9)

[0081] (10)

[0082] Then, the present invention uses a cross-attention strategy to integrate the question with the image and text for subsequent expert selection.

[0083] (11)

[0084] The multimodal expert model of this invention mainly consists of three parts: a gating network and two task-specific expert networks. The task-specific expert networks are used for image question answering and text question answering tasks, respectively. The gating network is responsible for selecting appropriate experts. Specifically, the gating network selects from the two task-specific networks.

[0085] During training, only a subset of experts are activated and updated through backpropagation. This invention sets the gating layer to a top-K selection, as shown in the following formula.

[0086] (12)

[0087] The Top-k algorithm selects the most suitable expert, either an image or a text expert, for subsequent question answering.

[0088] Given all trainable experts and input representations When performing token-selective routing, the output of the MoE model can be expressed as:

[0089] (13)

[0090] in This represents the value corresponding to the Kth gating score. [I()] represents the k-th expert selected.

[0091] Finally, the QADecoder performs cross-attention with the overall MoE, and the cross-entropy loss is used to calculate the loss, as shown in the following formula:

[0092] (14)

[0093] (15)

[0094] Each sub-question generates a sub-answer, which is then mapped to the sub-question and replaces the original intermediate conclusion to form the final implication tree. To ensure the correct generation of the implication tree, this invention introduces a semantically aware evaluator to evaluate it.

[0095] The sub-answers generated by the multimodal experts are then populated back into the corresponding nodes of the implication tree, replacing the original intermediate placeholders. Finally, the refined implication tree containing the complete chain of evidence and the sub-answers is input into the final answer model to generate the answer A to the question.

[0096] The main contributions of this invention are as follows:

[0097] The core contribution lies in constructing a joint optimization framework of "reinforcement learning teacher guidance—multi-path generation—multi-modal expert iteration." This method, by introducing a reinforcement learning teacher and combining it with a dual-reward mechanism, distills the student model, effectively solving the problems of low quality and poor logical coherence in existing entailment tree generation. Simultaneously, by utilizing multi-path generation and optimal tree selection strategies, it overcomes the flaw of error propagation easily generated by a single inference path, significantly enhancing the robustness of complex inference. Furthermore, this invention does not use the traditional modality-to-text conversion method, but instead introduces a multi-modal expert iteration mechanism to directly utilize the original image features, achieving highly efficient inference with high interpretability and accuracy while preserving fine-grained cross-modal information.

[0098] This invention introduces a reinforcement learning-based teacher model. By modeling the rationality, consistency, and correctness of the implication tree interpretation as reward signals, it optimizes the inference path generation process end-to-end, thereby providing the student model with a high-quality, interpretable inference structure. Furthermore, it employs a multi-path generation strategy, constructing multiple implication tree generation paths in parallel and performing path quality evaluation and optimal selection. This effectively alleviates the vulnerability of single-path inference to noise or local optima in complex scenarios, significantly improving the stability and robustness of the implication tree. In addition, traditional methods often pre-convert non-textual modal information, such as visual information, into text descriptions, potentially leading to the loss of crucial details. This invention designs a multimodal expert iterative response mechanism that dynamically calls and fuses original image and text features during inference, ensuring that answer generation fully utilizes fine-grained information from multiple sources, including images and text. This collaborative design enables this invention to significantly outperform existing technologies in three key dimensions: inference structure quality, robustness, and multimodal information utilization.

[0099] Example:

[0100] The objective of this invention is to evaluate the answering capability of the proposed method in multimodal, multi-hop question answering. Therefore, this invention is trained and evaluated on the WebQA and MultimodalQA datasets. WebQA is a multimodal, multi-hop question answering dataset with over 34,000 QA pairs. 44% of the image-based queries and 99% of the text-based queries require two or more knowledge sources. Each question contains confounding factors, which the model must consider and find the correct clues to provide the correct answer. MultimodalQA is a QA dataset containing 29,918 examples, requiring the integration of information across free text, semi-structured tables, and images. Each question also includes visual and textual confounding factors. 35.7% of the questions require cross-modal reasoning.

[0101] WebQA uses the BARTScore to measure the fluency and keyword accuracy of answers, represented as QA-FL and QA-ACC. These two scores are multiplied to obtain the QA score. Clue retrieval can be easily evaluated using the F1 score. MultimodalQA evaluates the model in three different settings: (1) questions requiring a single modality to answer; (2) questions requiring multimodal reasoning; and (3) all questions (ALL). Evaluation metrics require support for a list of answers, so performance is measured by word-level F1 scores and exact match (EM) of predicted answers.

[0102] In the implementation, for WebQA and MultimodalQA, a list of candidate cues is provided, and the model needs to find the most relevant cues to evaluate the accuracy of cue retrieval. A pre-trained language model (such as BERT-base) is used as the encoder to evaluate the accuracy of cue retrieval and as a problem decomposability discriminant model to determine whether the original question needs to be decomposed. A multimodal large language model (such as LLaVA-1.5) is used to handle visually relevant question-answering tasks. To reduce training costs and ensure performance, RLT is initialized based on a large language model with a certain parameter scale. In this embodiment, Qwen2.5-7B-Instruct is selected as the teacher model. The student model is a lightweight model with a small number of parameters. In this embodiment, the Qwen-3B model is used as the student model and is used to calculate the reward feedback for RLT interpretation during reinforcement learning. In the multimodal expert iteration module, image modality processing can adopt an image description model (such as BLIP), and the overall fusion training model can adopt a Transformer architecture model (such as T5). The reinforcement learning (RL) phase trained the main model for approximately 125 steps, with a batch size of 1,024. A constant learning rate strategy was employed, with the learning rate preferably set to [value missing]. In the GRPO (Group Relative Policy Optimization) algorithm, the group size is set to 64. During the distillation phase, inference trajectories generated by the teacher model are collected as the distillation dataset from the RL training process. The student model is fine-tuned using this complete dataset or a randomly sampled subset (e.g., a 1K subset). This embodiment directly uses the model's raw output for fine-tuning without applying additional post-processing to verify the model's generative capabilities. In the multimodal expert training phase, the model structure employs standard self-attention and cross-attention mechanisms. In both single-modal and cross-modal coding, each coding module uses a standard 8 attention heads. The optimizer uses the ADAM algorithm, with a learning rate set to [missing information]. Finally, the results, after expert iterative optimization, are input into a large model (such as Qwen2.5-7B-Instruct) to generate the final natural language answer.

[0103] As shown in Table 1, the implication tree generation accuracy of the present invention is significantly improved compared with other methods. These results demonstrate that the method of the present invention can more effectively improve the quality of implication tree generation and provides a higher-quality structured foundation for subsequent multimodal multi-hop question answering research.

[0104] Table 1

[0105]

[0106] This invention compares the proposed method with state-of-the-art methods, achieving competitive results on both the WebQA and MultimodalQA datasets. The results of this invention on the WebQA dataset are shown in Table 2.

[0107] Table 2

[0108]

[0109] The results of this invention on the MultimodalQA dataset are shown in Table 3. Under both unimodal and multimodal settings, the method of this invention outperforms the state-of-the-art methods in question answering tasks, fully demonstrating the advantages of the method of this invention in reasoning ability.

[0110] The method of this invention performs particularly well on datasets such as WebQA and MultimodalQA, which require precise evaluation of the accuracy of the final answer. This is mainly due to two key designs: first, this invention introduces a reinforcement learning-based teacher model to guide the generation of the implication tree, effectively ensuring the logical correctness of the implication tree generation; second, through a multimodal expert iterative mechanism, fine-grained multimodal information is preserved, ensuring that the question-answering process can fully utilize multimodal detailed information.

[0111] Table 3

[0112]

[0113] In this experiment, to verify the effectiveness of each module, this invention ablated the problem decomposition, reinforcement learning, multi-path implication tree generation, and multimodal expert iteration modules. When absolving the problem decomposition module, the original problem was directly used as input and as a prompt for generating image captions in the large model. When absolving the reinforcement learning and multi-path implication tree generation modules, the retrieved evidence and sub-problems were directly concatenated and input into the large model. When absolving the multimodal expert iteration module, the iterative optimization process was skipped, and the initially generated implication tree was directly fed into the large model for answer generation. The results of the ablation experiment are shown in Table 4. It can be seen that removing any component leads to performance degradation, with the absence of the multimodal expert iteration module causing the most significant performance decline, indicating its crucial role in fusing fine-grained multimodal information. The problem decomposition and reinforcement learning mechanisms also significantly contribute to the final results.

[0114] Table 4

[0115]

[0116] This invention presents case studies in multimodal datasets, such as... Figure 2 As shown, this example presents a complex reasoning sample. The model generates multiple candidate entailment trees through a multi-path strategy, and the final answer is obtained through iterative decomposition of the answers by multimodal experts. Finally, intermediate nodes are updated with the answers to sub-questions. This process demonstrates that the method of this invention can construct structurally sound and semantically accurate entailment trees for complex multimodal reasoning tasks, effectively guiding the reasoning process.

[0117] For simple reasoning tasks, large language models can typically generate structurally correct entailment trees, thus providing accurate answers. However, when faced with complex multi-hop or multimodal reasoning samples, directly relying on large models often leads to incorrect entailment tree structures and illogical reasoning paths. A student model trained through distillation learns high-quality reasoning structures, while a multi-path entailment tree generation strategy is combined. This combined strategy significantly improves the accuracy of entailment tree generation in complex scenarios, effectively enhancing reasoning accuracy.

[0118] Example 2

[0119] The present invention also provides a multimodal multi-hop question-answering device, comprising:

[0120] The first processing module is used to determine whether the input question needs to be decomposed. If so, it decomposes the question into sub-questions. Based on the decomposed questions, it retrieves evidence from the existing text evidence library and image evidence library to obtain evidence related to the question.

[0121] The second processing module is used to introduce reinforcement learning teachers into implication tree generation training based on the retrieved evidence and questions, and transfer the teacher model capabilities to the lightweight student model through distillation.

[0122] The third processing module is used to generate implication trees using the student model through multiple paths, and to evaluate and select the optimal implication tree using the optimal implication tree selection module.

[0123] The fourth processing module is used to perform cross-modal alignment and semantic aggregation of intermediate answers through multiple rounds of iteration, using the selected implication tree as the reasoning framework.

[0124] The fifth processing module is used to backfill the intermediate node answers obtained after the multimodal expert iteration is completed to the corresponding sub-problem nodes in the implication tree, input them into the large model for answering, and obtain the final answer.

[0125] As one embodiment of the present invention, the third processing module is used to generate an implication tree using a student model through three complementary reasoning strategies: divide and conquer, chain thinking of answer plans, and dynamic synthesis of examples.

[0126] As one embodiment of the present invention, the fourth processing module is used to traverse each sub-problem node using the selected implication tree as the reasoning framework, dynamically select the corresponding modality expert to handle each sub-problem, and perform cross-modal alignment and semantic aggregation on the intermediate answer through multiple rounds of iteration.

[0127] Example 3

[0128] The present invention also provides a multimodal multi-hop question answering system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a multimodal multi-hop question answering method when executed by the processor.

[0129] Example 4

[0130] The present invention also provides a storage medium storing a computer program, which executes a multimodal multi-hop question-answering method when running.

[0131] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal, multi-hop question-answering method, characterized in that, include: Step S1: Determine whether the input problem needs to be decomposed. If so, decompose it into subproblems. Based on the decomposed question, evidence is retrieved from the existing text evidence database and image evidence database to obtain evidence related to the question; Step S2: Based on the evidence and questions obtained from the retrieval, the reinforcement learning teacher is introduced into the implication tree generation training, and the teacher model's capabilities are transferred to the lightweight student model through distillation. Step S3: Use the student model to generate implication trees through multiple paths, and use the implication tree optimal selection module to evaluate and select the optimal implication tree; Step S4: Using the selected implication tree as the reasoning framework, perform cross-modal alignment and semantic aggregation on the intermediate answers through multiple rounds of iteration; Step S5: After the multimodal expert iteration is completed, the obtained intermediate node answers are backfilled layer by layer into the corresponding sub-problem nodes in the implication tree, and then input into the large model to answer, thus obtaining the final answer; In step S3, the student model is used to generate an implication tree through three complementary reasoning strategies: divide and conquer, chain thinking of answer plans, and dynamic synthesis of examples. Specifically: The divide-and-conquer approach involves generating a separate sub-implementation tree for each subproblem, then logically merging these subtrees to construct a complete implication tree, and finally simplifying the tree structure through post-processing optimization steps. Chain thinking for answering questions: First, generate a high-level answer plan, clarifying the key steps required to answer the question and their order of dependence; Subsequently, this plan was used as a guiding signal to constrain the generation process of the implication tree and ensure that each step of reasoning strictly follows the preset logical flow; Dynamic example generation: Using a large language model (LLM), several high-quality examples are dynamically synthesized based on the current question and evidence, and then injected into the prompt template.

2. The multimodal multi-hop question-answering method as described in claim 1, characterized in that, In step S4, the selected implication tree is used as the reasoning framework. Each sub-problem node is traversed, the corresponding modality expert is dynamically selected to handle each sub-problem, and cross-modal alignment and semantic aggregation of the intermediate answer are performed through multiple rounds of iteration.

3. A multimodal multi-hop question-answering apparatus for implementing the multimodal multi-hop question-answering method of claim 1, characterized in that, include: The first processing module is used to determine whether the input problem needs to be decomposed, and if so, it decomposes it into sub-problems. Based on the decomposed question, evidence is retrieved from the existing text evidence database and image evidence database to obtain evidence related to the question; The second processing module is used to introduce reinforcement learning teachers into implication tree generation training based on the retrieved evidence and questions, and transfer the teacher model capabilities to the lightweight student model through distillation. The third processing module is used to generate implication trees using the student model through multiple paths, and to evaluate and select the optimal implication tree using the optimal implication tree selection module. The fourth processing module is used to perform cross-modal alignment and semantic aggregation of intermediate answers through multiple rounds of iteration, using the selected implication tree as the reasoning framework. The fifth processing module is used to backfill the intermediate node answers obtained after the multimodal expert iteration is completed to the corresponding sub-problem nodes in the implication tree, input them into the large model for answering, and obtain the final answer.

4. The multimodal multi-hop question-answering device as described in claim 3, characterized in that, The fourth processing module is used to traverse each sub-problem node using the selected implication tree as the reasoning framework, dynamically select the corresponding modality expert to handle each sub-problem, and perform cross-modal alignment and semantic aggregation on the intermediate answers through multiple rounds of iteration.

5. A multimodal, multi-hop question-answering system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program performing the multimodal multi-hop question-answering method as described in any one of claims 1-2 when executed by the processor.

6. A storage medium, characterized in that, The storage medium stores a computer program that, when executed, performs the multimodal multi-hop question-answering method as described in any one of claims 1-2.

Images