A multimodal retrieval and question answering method based on structure awareness and faithful evidence guidance.

Abstract

This invention provides a multimodal retrieval and question answering method based on structure awareness and faithful evidence-guided generation. The method includes: a coarse-grained multimodal hybrid retrieval step: receiving a user query containing a query image and a text question; calculating a multimodal relevance score between the user query and documents in an external knowledge base based on multimodal features; and recalling a set of candidate documents that are visually and semantically related to the query image based on the multimodal relevance score; and a structure-aware fine-grained graph retrieval step: performing semantic parsing on the candidate document set, constructing a heterogeneous knowledge graph reflecting their inherent relationships, performing personalized information dissemination on the heterogeneous knowledge graph, and accurately extracting a set of fine-grained paragraphs as evidence from the candidate document set. This invention forces a large language model to anchor its response to the retrieved context through strict alignment constraints, significantly improving the evidence localization capability under multimodal features and ensuring absolute fidelity in the generation process.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, specifically a multimodal retrieval and question answering method based on structure awareness and faithful evidence-guided generation. Background Technology

[0002] Knowledge-based visual question answering aims to address complex queries that require reasoning across visual content and incorporating external knowledge sources. Traditional visual question answering typically relies solely on the pixel content of images. With the development of multimodal large language models, the dominant solution in this field has shifted to a retrieval-enhanced generative paradigm. In this paradigm, the system first retrieves relevant evidence fragments from an external knowledge base and then inputs them into the multimodal large language model to generate the final answer. By building the generation process on retrieved domain-specific information, this paradigm effectively extends the knowledge boundaries of the large language model beyond its internal parameters, significantly enhancing the model's ability to handle knowledge-intensive multimodal problems.

[0003] Despite the progress made, existing retrieval augmentation frameworks still face several fundamental challenges. First, there is the "structural blindness" in the retrieval process: existing methods typically treat knowledge items as isolated fragments, relying primarily on shallow similarity matching. This flattened retrieval paradigm ignores the implicit semantic connections within complex contexts, making it difficult for models to capture key evidence from relevant nodes and filter out noise. Second, there is the problem of "generational disjointness and prior bias": existing models lack explicit alignment mechanisms, failing to force their reasoning process to be strictly anchored to the retrieved context. Strong pre-trained prior knowledge often overshadows external evidence, leading to illusions, and the generated answers are often not logically derived from the evidence they cite, resulting in unfaithful reasoning trajectories. Furthermore, existing optimization methods (such as traditional reinforcement learning) often only focus on the accuracy of the final answer, completely ignoring the quality of intermediate reasoning and the verification of the fidelity of evidence. Summary of the Invention

[0004] This invention addresses the problems of "search structure blindness" and "disconnection and prior bias in the generation process" that exist in existing retrieval-enhanced generation paradigms when dealing with massive amounts of multimodal knowledge.

[0005] To address the aforementioned issues, this invention provides a multimodal retrieval question-answering method, system, electronic device, and storage medium based on structure awareness and faithful evidence-guided generation.

[0006] In a first aspect, the present invention provides a multimodal retrieval question answering method based on structure awareness and faithful evidence-guided generation, comprising the following steps: Coarse-grained multimodal hybrid retrieval steps: Receive user queries containing query images and text questions, calculate the multimodal relevance score between the user query and documents in an external knowledge base based on multimodal features, and recall a set of candidate documents that are visually and semantically related to the query image based on the multimodal relevance score; The structure-aware fine-grained graph retrieval steps are as follows: Semantic parsing is performed on the candidate document set to construct a heterogeneous knowledge graph reflecting their inherent relationships. Combined with the multimodal relevance scores, personalized information dissemination is performed on the heterogeneous knowledge graph to precisely extract a set of fine-grained paragraphs as evidence from the candidate document set; and... The evidence-guided reinforcement learning generation steps are as follows: the fine-grained paragraph set is used as the evidence context input to the multimodal large language model, and a composite reward mechanism including evidence fidelity assessment is used to train the multimodal large language model for reinforcement learning, so as to discipline the model to strictly follow and refer to the evidence context when generating answers.

[0007] Optionally, the coarse-grained multimodal hybrid retrieval step further includes: Generate a text summary that preserves the core semantics for each document in the external knowledge base; Using a visual encoder and a text encoder, the query image, the text summary of the document, and the image attached to the document are mapped to a shared feature space, respectively. Calculate the semantic similarity between the query image and the document summary, and the visual similarity between the query image and the document image; Based on the weighted fusion score of semantic similarity and visual similarity, the documents in the external knowledge base are sorted, and the top-ranked documents are recalled as the candidate document set.

[0008] Optionally, the weighted fusion score is calculated using the following formula: ; in, Hyperparameters that balance the contributions of visual and semantic similarity For visual encoders, To query images, For text encoders, For document Abstract, For document The fusion score, This represents the similarity calculation function. This refers to an image that is attached to the document.

[0009] Optionally, in the structure-aware fine-grained graph retrieval step, constructing a heterogeneous knowledge graph further includes: Open information is extracted from each paragraph in the candidate document set to extract triple information containing entities and relationships between entities; Using the paragraphs and entities as nodes, and the inclusion relationships between paragraphs and entities and the logical relationships between entities as edges, the heterogeneous knowledge graph is constructed.

[0010] Optionally, the personalized information dissemination based on multimodal relevance scores on the heterogeneous knowledge graph includes: Calculate the text similarity between the text question and each paragraph to obtain the first weight; The document-level multimodal relevance scores obtained in the coarse-grained multimodal hybrid retrieval step are normalized to obtain a calibration score; The calibration score is used to modulate the first weight to obtain the initial weight of the paragraph node, thereby injecting visual prior information into the graph network; Based on the initial weights of the paragraph nodes, a personalized graph propagation algorithm is run on the heterogeneous knowledge graph to spread the relevance along the edges of the graph. After the graph network converges, the top-ranked segment nodes are extracted as the fine-grained segment set based on the steady-state distribution scores of the nodes.

[0011] Optionally, the initial weight of the paragraph node is calculated using the following formula: ; in, Indicates calibration score, Indicates the initial weights. This indicates the problem of calculating text and each paragraph. Dense retrieval score.

[0012] Optionally, the composite reward mechanism includes: Formatted rewards are used to verify whether the model output contains structured labels for the specified reasoning process, sources of evidence, and final answer. Paragraph source reward, used to evaluate whether the evidence paragraphs referenced by the model in the evidence source label are consistent with preset real evidence paragraphs; and Accuracy rewards are used to evaluate whether the final answer output by the model is correct.

[0013] Optionally, the assignment of the accuracy reward is subject to a gating mechanism: an accuracy reward is given to the correct answer only if the paragraph source reward is a positive reward; if the paragraph source reward is a negative reward, the accuracy reward is zero even if the answer is correct.

[0014] Optionally, the evidence-guided reinforcement learning generation step further includes: For the same query, the multimodal large language model can generate multiple different answer trajectories. The total reward for each answer trajectory is calculated based on the aforementioned composite reward mechanism; Based on the group relative strategy optimization algorithm, the relative advantage of each answer trajectory within the group is calculated, and the model weights are updated using this relative advantage to train the model to generate answers that strictly follow the evidence context.

[0015] Secondly, embodiments of the present invention provide a multimodal retrieval and question-answering system based on structure awareness and faithful evidence-guided generation, including: Coarse-grained multimodal hybrid retrieval module: used to receive user queries containing query images and text questions, calculate the multimodal relevance score between the user query and documents in an external knowledge base based on multimodal features, and recall a set of candidate documents that are visually and semantically related to the query image based on the multimodal relevance score; The structure-aware fine-grained graph retrieval module is used to perform semantic parsing on the candidate document set, construct a heterogeneous knowledge graph reflecting its inherent relationships, and, in conjunction with the multimodal relevance scores, perform personalized information dissemination on the heterogeneous knowledge graph, accurately extracting a set of fine-grained paragraphs as evidence from the candidate document set; and The evidence-guided reinforcement learning generation module is used to input the fine-grained paragraph set as evidence context into the multimodal large language model, and to train the multimodal large language model using a composite reward mechanism that includes evidence fidelity assessment, so as to discipline the model to strictly follow and refer to the evidence context when generating answers.

[0016] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call logical instructions in the memory to execute the steps of the method provided in the first aspect.

[0017] Fourthly, embodiments of the present invention provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal retrieval question-answering method based on structure awareness and faithful evidence-guided generation as described in the first aspect.

[0018] The beneficial effects of the multimodal retrieval question answering method based on structure awareness and faithful evidence-guided generation in this invention are as follows: This invention first utilizes a coarse-grained hybrid search that integrates visual and textual modalities to recall candidate documents, effectively overcoming the limitations of single-modal retrieval and narrowing the search space. Second, it implements structure-aware, fine-grained graph retrieval, capturing structural dependencies by constructing a heterogeneous knowledge graph, thereby effectively filtering noise and accurately locating specific supporting evidence. Finally, it introduces an evidence-guided reward-based reinforcement learning strategy, which strictly requires that a reward be given only when the model selects the correct evidence and arrives at the correct answer. This strict alignment constraint forces the large language model to anchor its responses to the retrieved context, significantly improving the evidence localization capability under multimodal features and ensuring absolute fidelity in the generation process. Attached Figure Description

[0019] Figure 1 This is a flowchart of a multimodal retrieval question answering method based on structure awareness and faithful evidence guidance in an embodiment of the present invention; Figure 2 This is a schematic diagram of the multimodal retrieval question answering method based on structure awareness and faithful evidence-guided generation in an embodiment of the present invention; Figure 3 This is a structural block diagram of the multimodal retrieval and question answering system based on structure awareness and faithful evidence in an embodiment of the present invention. Figure 4 This is a structural block diagram of the electronic device in an embodiment of the present invention. Detailed Implementation

[0020] To better understand the purpose, technical solution, and advantages of this application, the application is described and explained below in conjunction with the accompanying drawings and embodiments.

[0021] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0022] First, a brief description of the solution of the present invention: This invention aims to address the problems of "search structure blindness" and "disconnection and prior bias in the generation process" in existing retrieval-enhanced generation paradigms when dealing with massive amounts of multimodal knowledge, thereby achieving accurate fine-grained evidence localization and credible, faithful multimodal reasoning generation.

[0023] This invention provides a multimodal question answering method based on structure awareness and faithful evidence-guided generation. The overall input of the system is a user query containing images and text questions, as well as a massive external multimodal knowledge base; the overall output of the system is an accurate answer derived through rigorous reasoning.

[0024] In order to accurately locate evidence from a massive knowledge base and guide model generation, the core execution process of this invention is divided into three progressive stages: first, coarse-grained screening; then, fine-grained structural purification; and finally, constrained reinforcement learning generation.

[0025] The method framework of this invention mainly consists of three stages, and the specific processing flow is as follows: In the first step (coarse-grained multimodal hybrid retrieval), the model aims to reduce the search space of the vast external knowledge base and filter out initial noise. This invention first generates concise text summaries for complex documents in the external knowledge base to preserve core semantics and eliminate redundancy. Subsequently, this invention utilizes visual and text encoders to map the query image, candidate document summaries, and associated candidate document images into a shared feature space. By comprehensively calculating the semantic similarity between the query image and candidate summaries, as well as the visual similarity between the query image and document images, this invention performs a weighted fusion scoring, thereby efficiently retrieving the set of candidate documents most relevant to the query.

[0026] In the second step (structure-aware fine-grained graph retrieval), the model needs to capture the hidden fine-grained semantic relationships in candidate documents to extract precise textual evidence fragments. This invention first utilizes open information extraction techniques to extract entities and relationships between entities in each paragraph of the candidate documents, constructing a schema-free heterogeneous knowledge graph. Then, this invention injects the multimodal relevance scores obtained in the first stage as prior relevance weights into the paragraph nodes of the graph structure, implementing a multimodal-guided personalized graph propagation strategy. Through this topology propagation method, this invention can aggregate complementary evidence across isolated document paragraphs, ultimately outputting the highest-ranked set of fine-grained paragraphs as the context for subsequently generated precise evidence.

[0027] In the third step (evidence-guided reinforcement learning generation), to force the large language model to strictly adhere to the selected evidence when generating answers, this invention introduces a unique evidence-guided reinforcement learning (RL) optimization strategy. This invention designs a composite reward mechanism, including format reward, paragraph reward, and accuracy reward. The accuracy reward employs a strict alignment constraint gating mechanism: the system only rewards the result when the model explicitly cites the correct supporting evidence paragraphs during reasoning and arrives at the correct answer. Combined with a group-relative policy optimization algorithm, this invention forces the large language model to strictly anchor its reasoning trajectory to the retrieved context, effectively suppressing the illusion caused by the model relying on unverified prior knowledge, and ensuring the fidelity and reliability of the generated results.

[0028] like Figure 1 and Figure 2 As shown in the figure, a multimodal retrieval question answering method based on structure awareness and faithful evidence-guided generation provided in this embodiment of the invention includes the following steps: S1. Coarse-grained multimodal hybrid retrieval steps: Receive user queries containing query images and text questions, calculate the multimodal relevance scores between the user query and documents in the external knowledge base based on multimodal features, and recall a set of candidate documents that are visually and semantically related to the query image based on the multimodal relevance scores. The coarse-grained multimodal hybrid retrieval step further includes: Generate a text summary that preserves the core semantics for each document in an external knowledge base; Using a visual encoder and a text encoder, the query image, the text summary of the document, and the image attached to the document are mapped to a shared feature space, respectively. Calculate the semantic similarity between the query image and the document summary, and the visual similarity between the query image and the document image; Based on a weighted fusion score of semantic and visual similarity, documents in an external knowledge base are ranked, and the top-ranked documents are recalled as a candidate document set.

[0029] Specifically, since external knowledge bases typically contain millions of documents, performing fine-grained retrieval directly would incur enormous computational overhead and easily introduce a large amount of irrelevant interference. Therefore, this step aims to utilize multimodal features to perform preliminary and efficient filtering on the massive knowledge base, narrowing the search space to a small batch of candidate documents most relevant to the query image.

[0030] 1. Offline document summary generation Complete knowledge base documents are typically very long and contain a large amount of trivial details. To build an efficient retrieval index, this invention pre-processes the documents to reduce their dimensionality. For each document in the external document collection... (Contains multiple paragraphs of text) The system will concatenate all paragraphs of the document into a single file. And input it into the large language model In the middle, combined with specific instruction prompts P, the model extracts key information such as the entity's core history and features, generating a concise summary. These summaries will replace the lengthy original documents as the benchmark for subsequent text matching.

[0031] 2. Joint feature extraction and hybrid scoring When a user query (containing a query image) is received When dealing with text-based problems, this invention utilizes both visual and textual modal information for matching. Specifically, it uses a visual encoder. and text encoder The "query image," "candidate document summary," and "images attached to candidate documents" are uniformly mapped to a shared feature space. The scoring mechanism of this invention is not a single-dimensional comparison, but adopts a weighted fusion strategy: on the one hand, it calculates the semantic similarity between the query image and the document summary (to solve the problem of visual similarity but different connotations), and on the other hand, it calculates the visual similarity between the query image and the document image (to solve the visual alignment problem when the text description is ambiguous).

[0032] In this embodiment, the weighted fusion score is calculated using the following formula: ; in, Hyperparameters that balance the contributions of visual and semantic similarity For visual encoders, To query images, For text encoders, For document Abstract, For document The fusion score, This represents the similarity calculation function. This refers to an image that is attached to the document.

[0033] 3. Candidate document recall Based on the above fusion score The system performs a rapid descending sort of the entire knowledge base, only extracting the top-ranked entries. One document. This batch of recalls. It not only matches the query object visually, but is also highly relevant in core semantics, and will serve as the corpus pool for the next stage of fine-tuning.

[0034] S2, Structure-Aware Fine-Grained Graph Retrieval Steps: Semantic parsing of the candidate document set, construction of a heterogeneous knowledge graph reflecting its inherent relationships, and personalized information dissemination on the heterogeneous knowledge graph by combining multimodal relevance scores, and precise extraction of a set of fine-grained paragraphs as evidence from the candidate document set; In this embodiment, the construction of the heterogeneous knowledge graph in the structure-aware fine-grained graph retrieval step further includes: Open information is extracted from each paragraph in the candidate document set to extract triples containing entity and relationship information between entities; A heterogeneous knowledge graph is constructed using paragraphs and entities as nodes, and the inclusion relationship between paragraphs and entities and the logical relationship between entities as edges.

[0035] Personalized information dissemination on heterogeneous knowledge graphs by combining multimodal relevance scores includes: Calculate the text similarity between the text question and each paragraph to obtain the first weight; The document-level multimodal relevance scores obtained in the coarse-grained multimodal hybrid retrieval step are normalized to obtain calibration scores; The initial weights of the paragraph nodes are obtained by modulating the first weights using calibration scores, thereby injecting visual prior information into the graph network. Based on the initial weights of paragraph nodes, a personalized graph propagation algorithm is run on a heterogeneous knowledge graph to spread relevance along the edges of the graph. Once the graph network converges, the top-ranked segment nodes are extracted as a fine-grained segment set based on the steady-state distribution scores of the nodes.

[0036] Specifically, in the candidate document set obtained in the previous step In this context, while documents may be generally relevant, the key evidence needed to answer specific questions may be hidden only in a few particular paragraphs. Traditional flat text retrieval ignores the logical connections between contexts. This stage aims to capture the implicit relationships between paragraphs by constructing a knowledge graph, thereby achieving precise paragraph-level evidence extraction.

[0037] 1. Construction of Heterogeneous Knowledge Graphs To break down barriers between paragraphs, the system utilizes open information extraction technology. Each paragraph Deep analysis is performed to extract triples with logical relationships (such as "entity A - contains - entity B"). Based on this information, the system constructs a schema-free heterogeneous knowledge graph for the current query. In this graph, nodes are divided into two categories: specific "paragraph nodes" and extracted "entity nodes." Edges are also divided into two categories: "inclusion relationship edges" between paragraphs and their containing entities, and "logical relationship edges" between entities. Through this topological structure, related paragraphs that were originally scattered in different documents are tightly connected together through shared entities.

[0038] 2. Multimodal sensing prior initialization Before disseminating information on the constructed graph, it is necessary to determine the "starting point" (i.e., initial weight) of the query. Conventional methods only calculate the similarity between the question text and the paragraph text, which loses the valuable visual information obtained in the first stage. The core innovation of this invention lies in injecting global multimodal scores as prior information into the graph nodes. First, the dense retrieval scores between the text question and each paragraph are calculated. Secondly, the document-level multimodal scores from the first stage are normalized in the candidate set to obtain calibration scores. Finally, the multimodal score is used to modulate text similarity to obtain the initial weights of paragraph nodes. The initial weights of paragraph nodes are calculated using the following formula: ; in, Indicates calibration score, Indicates the initial weights. This indicates the problem of calculating text and each paragraph. Dense retrieval score.

[0039] This operation ensures that document paragraphs that are visually more relevant to the query receive higher initial activation values ​​in the graph network. Simultaneously, it activates key entity nodes in the graph based on the semantic similarity between the question and the triples.

[0040] 3. Personalized image dissemination and precise evidence selection After the weight initialization is complete, the system in the graph The system runs a personalized PageRank algorithm. Relevance is "randomly walked" and diffused along the network edges between entities and paragraphs. This means that if a paragraph itself does not contain the direct keywords of the query, but it has a strong entity association with high-weight paragraphs that do contain those keywords, its importance score will increase. After the graph network converges, the system directly extracts the top-ranked paragraphs from the steady-state distribution. The paragraph nodes represent the precise set of textual evidence after filtering out a large amount of noise. .

[0041] S3. Evidence-guided reinforcement learning generation steps: The fine-grained paragraph set is used as the evidence context input to the multimodal large language model, and a composite reward mechanism including evidence fidelity assessment is used to train the multimodal large language model for reinforcement learning, so as to discipline the model to strictly follow and cite the evidence context when generating answers.

[0042] In this embodiment, the composite reward mechanism includes: Formatted rewards are used to verify whether the model output contains structured labels for the specified reasoning process, sources of evidence, and final answer. Paragraph source reward, used to evaluate whether the evidence paragraphs cited by the model in the evidence source label are consistent with the preset real evidence paragraphs; and Accuracy rewards are used to evaluate whether the final answer output by the model is correct.

[0043] Furthermore, the assignment of accuracy rewards is subject to a gating mechanism: an accuracy reward is given to the correct answer only if the paragraph source reward is a positive reward; if the paragraph source reward is a negative reward, the accuracy reward is zero even if the answer is correct.

[0044] The evidence-guided reinforcement learning generation steps further include: For the same query, the multimodal large language model can generate multiple different response trajectories. The total reward for each answer trajectory is calculated based on a composite reward mechanism; Based on the group relative strategy optimization algorithm, the relative advantage of each answer trajectory within the group is calculated, and the model weights are updated using this relative advantage to train the model to generate answers that are strictly based on the evidence context.

[0045] Specifically, even with the precise evidence provided in the second stage, multimodal large language models, due to their pre-training mechanisms, often tend to ignore input evidence and make subjective guesses (i.e., illusions), or their answers lack logical connection to the cited evidence. This stage aims to discipline the model through specific reinforcement learning mechanisms, ensuring that its statements are "based on evidence."

[0046] 1. Standardized model output structure When inputting queries and selected evidence into the large language model, the system forces the model to output according to a specific thought chain format via instructions. (Complete output) Must contain, in sequence: enclosed in <think>The step-by-step reasoning process within the label, surrounded by <evidence>The specific citation within the tag (must specify which document and paragraph it is from), and the text enclosed in... <answer>The final, concise answer within the tags. This structured output forms the basis for subsequent automated machine evaluation and reward.

[0047] 2. Calculation of rewards guided by composite evidence Unlike traditional reinforcement learning that only considers the correctness of the final answer, this invention designs a rigorous coupled reward mechanism where positive feedback is only obtained when both the process and the result are correct. Specific computational dimensions include: Formatting Bonus: First, check if the output completely contains the three tags mentioned above. If it conforms to the specifications, a positive score is awarded; otherwise, the generation is considered a failure, and subsequent scoring is interrupted.

[0048] Paragraph source reward: The system automatically parses the model in <evidence>Document title entered in the tag and the document title containing the true facts required for the question. Perform a comparison. If the model successfully selects the correct source from the provided context, then... Otherwise, it is 0. This is intended to encourage the model to actively rely on external input rather than its own illusions.

[0049] Accuracy-gated reward: This is crucial for ensuring that the large model generates results faithful to the retrieved content. The system compares the model's answer with the standard answer. However, this invention sets a gating constraint: even if the answer happens to be correct, if the paragraph source reward in the previous step is 0 (i.e., no correct evidence is cited), the accuracy reward will still be forcibly reset to zero. The formula is expressed as: ; Finally, the system linearly sums up the above rewards to obtain the total evaluation feedback generated this time. .

[0050] 3. Group-based strategy optimization algorithm After obtaining the total reward, the system enters the model weight update phase. To improve training efficiency, this invention employs the Group Relative Policy Optimization (GRPO) algorithm. Specifically, for each query, the system instructs the large language model to generate G different response trajectories. The total reward for each of these G responses is calculated, and the relative advantage within the group (i.e., how good or bad a particular response is compared to the group's average) is determined. The model updates its own weights through gradient backpropagation of these relative advantages. Through continuous reinforcement learning, the model eventually learns how to locate key sentences from complex data and rigorously derives reliable, illusion-free final answers based on these sentences.

[0051] According to the above scheme, the present invention can fully construct a multimodal question-answering model with good performance.

[0052] In summary, compared to existing retrieval-enhanced visual question answering methods, this invention dynamically combines structure-aware multimodal graph retrieval with evidence-guided reinforcement learning strategies, offering significant advantages and positive effects. First, in the retrieval phase, this invention effectively overcomes the limitations and visual ambiguities of traditional single-modal retrieval by combining coarse-grained multimodal hybrid retrieval with fine-grained structure-aware graph retrieval. Utilizing a personalized graph propagation mechanism guided by multimodal prior scores, it can aggregate complementary evidence across isolated paragraphs without training expensive reorderers, accurately locating fine-grained knowledge and significantly filtering noise in complex contexts. Second, in the generation phase, this invention innovatively introduces an evidence-guided reinforcement learning strategy, designing a composite reward mechanism that includes format specification, paragraph source verification, and final accuracy. This strict alignment constraint forces the multimodal large language model to derive answers based on correctly cited evidence, fundamentally alleviating the "illusion" and reasoning disconnect problems caused by over-reliance on pre-trained prior knowledge, ensuring the absolute fidelity and logical consistency of the generated results. Finally, by employing a training-free graph structure propagation mechanism and an efficient group relative policy optimization algorithm that does not require value function critics, this invention significantly reduces model training and computational resource consumption while constructing a knowledge-based multimodal visual question answering framework that combines high efficiency, high accuracy, and high reliability.

[0053] This invention also provides a multimodal retrieval question-answering system based on structure awareness and faithful evidence-guided generation. This system is used to implement the above-described method embodiments, and details already described will not be repeated. The terms "module," "unit," and "subunit" used below refer to combinations of software and / or hardware that perform a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.

[0054] like Figure 3 As shown, Figure 3 This is a structural block diagram of the multimodal retrieval and question-answering system based on structure awareness and faithful evidence guidance in this invention. The system includes: Coarse-grained multimodal hybrid retrieval module 101: is used to receive user queries containing query images and text questions, calculate the multimodal relevance score between the user query and documents in an external knowledge base based on multimodal features, and recall a set of candidate documents that are visually and semantically related to the query image based on the multimodal relevance score. The structure-aware fine-grained graph retrieval module 102 is used to perform semantic parsing on the candidate document set, construct a heterogeneous knowledge graph reflecting its inherent relationships, and, in conjunction with the multimodal relevance scores, perform personalized information dissemination on the heterogeneous knowledge graph, accurately extracting a set of fine-grained paragraphs as evidence from the candidate document set; and The evidence-guided reinforcement learning generation module 103 is used to input the fine-grained paragraph set as evidence context into the multimodal large language model, and to train the multimodal large language model through reinforcement learning using a composite reward mechanism that includes evidence fidelity assessment, so as to discipline the model to strictly follow and refer to the evidence context when generating answers.

[0055] This system is used to implement the above method embodiments, so it will not be described in detail here.

[0056] like Figure 4 As shown in the figure, an electronic device provided by an embodiment of the present invention includes: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute the following method: Coarse-grained multimodal hybrid retrieval steps: Receive user queries containing query images and text questions, calculate the multimodal relevance score between the user query and documents in an external knowledge base based on multimodal features, and recall a set of candidate documents that are visually and semantically related to the query image based on the multimodal relevance score; The structure-aware fine-grained graph retrieval steps are as follows: Semantic parsing is performed on the candidate document set to construct a heterogeneous knowledge graph reflecting their inherent relationships. Combined with the multimodal relevance scores, personalized information dissemination is performed on the heterogeneous knowledge graph to precisely extract a set of fine-grained paragraphs as evidence from the candidate document set; and... The evidence-guided reinforcement learning generation steps are as follows: the fine-grained paragraph set is used as the evidence context input to the multimodal large language model, and a composite reward mechanism including evidence fidelity assessment is used to train the multimodal large language model for reinforcement learning, so as to discipline the model to strictly follow and refer to the evidence context when generating answers.

[0057] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0058] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the methods provided in the above embodiments.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / evidence> < / answer> < / evidence> < / think>

Claims

1. A multimodal retrieval and question-answering method based on structure awareness and faithful evidence-guided generation, characterized in that, Includes the following steps: Coarse-grained multimodal hybrid retrieval steps: Receive user queries containing query images and text questions, calculate the multimodal relevance score between the user query and documents in an external knowledge base based on multimodal features, and recall a set of candidate documents that are visually and semantically related to the query image based on the multimodal relevance score; The structure-aware fine-grained graph retrieval steps are as follows: semantic parsing is performed on the candidate document set to construct a heterogeneous knowledge graph that reflects its inherent relationships. Combined with the multimodal relevance score, personalized information is disseminated on the heterogeneous knowledge graph to accurately extract a set of fine-grained paragraphs as evidence from the candidate document set. as well as The evidence-guided reinforcement learning generation steps are as follows: the fine-grained paragraph set is used as the evidence context input to the multimodal large language model, and a composite reward mechanism including evidence fidelity assessment is used to train the multimodal large language model for reinforcement learning, so as to discipline the model to strictly follow and refer to the evidence context when generating answers.

2. The method according to claim 1, characterized in that, The coarse-grained multimodal hybrid retrieval step further includes: Generate a text summary that preserves the core semantics for each document in the external knowledge base; Using a visual encoder and a text encoder, the query image, the text summary of the document, and the image attached to the document are mapped to a shared feature space, respectively. Calculate the semantic similarity between the query image and the document summary, and the visual similarity between the query image and the document image; Based on the weighted fusion score of semantic similarity and visual similarity, the documents in the external knowledge base are sorted, and the top-ranked documents are recalled as the candidate document set.

3. The method according to claim 2, characterized in that, The weighted fusion score is calculated using the following formula: ； in, Hyperparameters that balance the contributions of visual and semantic similarity For visual encoders, To query images, For text encoders, For document Abstract, For document The fusion score, This represents the similarity calculation function. This refers to an image that is attached to the document.

4. The method according to claim 1, characterized in that, In the structure-aware fine-grained graph retrieval step, constructing a heterogeneous knowledge graph further includes: Open information is extracted from each paragraph in the candidate document set to extract triple information containing entities and relationships between entities; Using the paragraphs and entities as nodes, and the inclusion relationships between paragraphs and entities and the logical relationships between entities as edges, the heterogeneous knowledge graph is constructed.

5. The method according to claim 4, characterized in that, The personalized information dissemination on the heterogeneous knowledge graph by combining multimodal relevance scores includes: Calculate the text similarity between the text question and each paragraph to obtain the first weight; The document-level multimodal relevance scores obtained in the coarse-grained multimodal hybrid retrieval step are normalized to obtain a calibration score; The calibration score is used to modulate the first weight to obtain the initial weight of the paragraph node, thereby injecting visual prior information into the graph network; Based on the initial weights of the paragraph nodes, a personalized graph propagation algorithm is run on the heterogeneous knowledge graph to spread the relevance along the edges of the graph. After the graph network converges, the top-ranked segment nodes are extracted as the fine-grained segment set based on the steady-state distribution scores of the nodes.

6. The method according to claim 5, characterized in that, The initial weight of the paragraph node is calculated using the following formula: ； in, Indicates calibration score, Indicates the initial weights. This indicates the problem of calculating text and each paragraph. Dense retrieval score.

7. The method according to claim 1, characterized in that, The composite reward mechanism includes: Formatted rewards are used to verify whether the model output contains structured labels for the specified reasoning process, sources of evidence, and final answer. Paragraph source reward, used to evaluate whether the evidence paragraphs referenced by the model in the evidence source label are consistent with preset real evidence paragraphs; and Accuracy rewards are used to evaluate whether the final answer output by the model is correct.

8. The method according to claim 7, characterized in that, The assignment of the accuracy reward is subject to a gating mechanism: an accuracy reward is given to the correct answer only if the paragraph source reward is a positive reward; if the paragraph source reward is a negative reward, the accuracy reward is zero even if the answer is correct.

9. The method according to claim 7 or 8, characterized in that, The evidence-guided reinforcement learning generation step further includes: For the same query, the multimodal large language model can generate multiple different answer trajectories. The total reward for each answer trajectory is calculated based on the aforementioned composite reward mechanism; Based on the group relative strategy optimization algorithm, the relative advantage of each answer trajectory within the group is calculated, and the model weights are updated using this relative advantage to train the model to generate answers that strictly follow the evidence context.

10. A non-transitory computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the multimodal retrieval and question-answering method based on structure awareness and faithful evidence-guided generation as described in any one of claims 1 to 9.

Images