A method for obtaining answers to questions based on a dynamic inference graph for a multi-modal large model

Abstract

The application relates to the technical field of document question answering, in particular to a question answering acquisition method for a multi-modal large model based on a dynamic reasoning graph, which comprises the following steps: receiving a target document set provided by a target user end and a corresponding target question sentence; inputting the target question sentence into a large language model to acquire a target knowledge graph, so as to generate a target question answering answer text corresponding to the target question sentence; and sending the target question answering answer text and basic analysis data corresponding to the target question answering answer text to the target user end. It can be known that the method acquires a knowledge graph, the nodes in the graph represent atomic facts, and the edges represent typed logical relationships, so that the entire reasoning process is changed from "descriptive text" to "formal structure", so that the machine can automatically check the reasoning path, check the logical consistency and perform integrity auditing, and the problem that the reasoning process is a "black box" and cannot be deeply examined is fundamentally solved.

Description

Technical Field

[0001] This invention relates to the field of document question answering technology, and in particular to a method for obtaining question answers for multimodal large models based on dynamic reasoning graphs. Background Technology

[0002] In the technological evolution aimed at improving the interpretability (XAI) and decision traceability of large language models, three main representative technical approaches have emerged, each dedicated to enhancing transparency at different levels. The first approach focuses on visualizing the reasoning process, with thought chain hints, thought tree frameworks, and self-consistency methods at its core. These methods guide the model to generate human-like, progressively advancing natural language reasoning steps through specific hint mechanisms, thereby making the implicit thought process partially explicit. However, its output remains at the level of free text, essentially a "narration" of reasoning rather than a structured, formalized, machine-parseable representation, thus failing to support automated verification and in-depth auditing of logical validity.

[0003] The second category focuses on tracing and verifying the conclusion, with retrieval-enhanced generative architectures as a typical example. This method couples the retrieval system with the generative model, attaching retrieved reference document fragments as evidence when generating the answer, aiming to strengthen the factual basis of the answer and reduce illusions. However, its tracing ability is relatively coarse, usually pointing to the entire document or a long block of text, making it difficult to accurately locate the subtle units of evidence supporting a specific argument, such as a cell in a table, a specific data point in a chart, or a specific area in an image, and even more so, it cannot clearly depict the combination and deductive relationships between multiple pieces of evidence.

[0004] The third category aims to handle complex multi-source information fusion, primarily employing multi-hop reasoning techniques. This method mimics the step-by-step analytical approach of humans, constructing a linear reasoning chain by linking multiple pieces of evidence to solve complex problems. However, this sequentially dependent chain structure itself has inherent fragility; any retrieval error or comprehension deviation in any intermediate link can be propagated and amplified, ultimately leading to a break in the reasoning path or a deviation from the conclusion.

[0005] In summary, while existing technologies have made beneficial explorations in many aspects, none have systematically constructed a complete chain of evidence that combines deep structuring, precise fine-grained tracing, non-linear robust logical expression capabilities, and supports end-to-end machine review and interactive verification. This fundamental deficiency limits the reliable application of existing methods in scenarios with extremely high requirements for accuracy, transparency, and auditability, such as financial risk control, judicial evidence collection, and medical auxiliary diagnosis. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a question-answering method for multimodal large models based on dynamic reasoning graphs. By introducing dynamic reasoning graphs as a core structured representation, it achieves a fundamental improvement over the deficiencies described in the background technology.

[0007] According to a first aspect of the present invention, a question-answering method for multimodal large models based on dynamic reasoning graphs is provided, the method comprising the following steps: S100, receive the target document set and the target question statement provided by the target user terminal; S200, the target question statement is input into the large language model to obtain the target intent label corresponding to the target question statement; S300, Obtain the target knowledge graph based on the target document set; S400, Traverse the target knowledge graph and generate the target question-and-answer text corresponding to the target question statement; S500, the target question-and-answer text and the corresponding basic analysis data are sent to the target user terminal.

[0008] The present invention has at least the following beneficial effects: This invention transforms the unstructured natural language reasoning chains (such as thought chains) in traditional methods into dynamic reasoning graphs that machines can directly parse and verify. Nodes in the graph represent atomic facts, and edges represent typified logical relationships, transforming the entire reasoning process from "descriptive text" into a "formal structure." This transformation enables machines to automatically verify reasoning paths, check logical consistency, and audit integrity, fundamentally solving the problem of the "black box" nature and inability to deeply examine the reasoning process in the background technology.

[0009] By forcibly binding each atomic fact node to a precise location in the source document (such as cell coordinates or text line positions), this invention achieves "cell-level" or "data point-level" tracing from the answer assertion to the original evidence. Simultaneously, the typified edges in the graph clearly reveal how evidence is combined into the final conclusion through logical operations such as comparison, inference, and calculation. This completely overcomes the shortcomings of background technologies (such as RAG) in terms of coarse granularity of citations and ambiguous logical relationships between evidence, providing a complete, transparent, and verifiable chain of evidence.

[0010] The dynamic reasoning graph employed in this invention is a non-linear, networked structure that allows conclusions to be drawn from multiple evidence nodes via different paths. This structure avoids the fragile linear chains upon which multi-hop reasoning in prior art relies. Even if a sub-path in the graph contains uncertainty or error, the system can still arrive at robust conclusions through other parallel evidence paths, thereby significantly improving the fault tolerance and overall robustness of reasoning for complex problems.

[0011] By designing an "evidence-constrained answer generator," the language generation process is strictly limited to the "translation" of the constructed reasoning graph. Furthermore, techniques such as probability masks are employed to physically prevent the model from introducing factual information outside the graph. This mechanism significantly suppresses the generation of "factual illusions" at the source, ensuring a high degree of fidelity between the output text and the source evidence—a reliability guarantee not found in traditional generation methods (including CoT and RAG).

[0012] The interactive explanation interface provided by this invention not only presents the answer but also deeply integrates it with the underlying reasoning graph and the original document. Users can intuitively and conveniently trace and verify the process step-by-step from the final conclusion to the reasoning subgraph, then to the atomic facts, and finally to the location of the original document. This closed-loop experience upgrades interpretability from "passive presentation" to "active exploration," greatly enhancing users' trust in and sense of control over the AI ​​output in critical decision-making scenarios.

[0013] In summary, this invention, through its technical system of "structured graph construction," "fine-grained evidence binding," "non-linear logic organization," "constraint content generation," and "interactive verification closed loop," systematically addresses the core pain points of current large models in complex document question answering, namely low transparency, difficulty in traceability, and poor reliability. This provides a solid technical foundation for trustworthy AI applications in high-value, high-risk fields such as financial analysis, legal judgment, and medical diagnosis. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a question-answering method for multimodal large models based on dynamic reasoning graphs, provided as an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1 like Figure 1 As shown, this embodiment provides a question-answering method for multimodal large models based on dynamic reasoning graphs. The method includes the following steps: S100, receives the target document set and the target question statement provided by the target user.

[0018] Specifically, in step S100, a query request from the target user is received through a standardized interface. The query request from the target user includes the target document set and the target question statement.

[0019] Furthermore, the target document set includes one or more multimodal electronic documents to be analyzed, the formats of which include, but are not limited to, PDF documents, scanned image files, presentations, or documents containing structured tables, and the document content can cover multiple modalities such as text, tables, and charts.

[0020] Furthermore, the target question statement is natural language text input by the target user, used to query specific information contained in the target document set or to request logical analysis based on the target document set.

[0021] Preferably, after receiving the target document set in step S100, a multimodal document preprocessing process is automatically started: optical character recognition is performed on the scanned documents or images contained therein to extract text content, and a pre-trained document layout analysis model is used to perform unified structured parsing on all documents to identify their logical area divisions. At the same time, a precise source pointer containing document identifiers, page numbers and page coordinate information is generated for each locatable text fragment, table cell or image area.

[0022] S200, the target question statement is input into the large language model to obtain the target intent tag corresponding to the target question statement.

[0023] Specifically, in step S200, the target question statement is input into a fine-tuned natural language understanding model. This model, built upon a large language model, is used for deep semantic parsing of user queries. The model outputs a structured target intent label, which not only includes question type classification (e.g., comparison, calculation, causal inference, fact query), but also further parses the constraints such as entities, attributes, and timeframes relevant to the question, providing clear semantic guidance for subsequent knowledge graph construction and reasoning.

[0024] S300, Obtain the target knowledge graph based on the target document set.

[0025] Specifically, step S300 includes the following sub-steps: S301: Based on the target intent tag, extract all relevant atomic facts from the preprocessed document content.

[0026] Furthermore, each atomic fact is the smallest indivisible unit of information with independent semantics, and is bound to its precise source pointer in the source document.

[0027] Preferably, the preprocessed document is the document obtained by automatically starting the multimodal document preprocessing process after receiving the target document set in step S100.

[0028] S302: Using all extracted atomic facts as initial nodes, and employing a logical reasoning model, based on the reasoning direction indicated by the target intent label, automatically identify and establish logical relationships between nodes to form typified directed edges. The relationship types include, but are not limited to, comparison, inference, calculation, induction, and support.

[0029] S303: During the construction process, the logical reasoning model can perform iterative reasoning based on existing nodes and edges to generate new derived fact nodes and add them to the graph until the graph can fully support the solution to the target problem.

[0030] S304 ultimately generates a structured, dynamic reasoning graph containing atomic fact nodes, derived fact nodes, and typified edges, which serves as the target knowledge graph.

[0031] S400, traverse the target knowledge graph and generate the target question and answer text corresponding to the target question statement.

[0032] Specifically, step S400 also includes the following steps: S401, when the target knowledge graph is input into an answer generator under evidence constraints, the answer generator performs topological sorting on the target knowledge graph to determine the logical dependency order between nodes; S402, following the reasoning path from the original evidence node to the final conclusion node, transforms the content of each node and the logical relationships between them into a coherent natural language description.

[0033] A further understanding is that the answer generator contains evidence constraints that constrain the content of the target knowledge graph. Through a probability masking mechanism, it prohibits the introduction of factual information outside the target knowledge graph, ensuring that every assertion in the generated target question-and-answer answer text originates from the graph and can naturally embed the corresponding evidence source identifier in the text.

[0034] S500, the target question-and-answer answer text and the basic analysis data corresponding to the target question-and-answer answer text are sent to the target user terminal; further understood as: the interactive explanation interface deployed on the target user terminal will receive and present the basic analysis data corresponding to the target question-and-answer answer text: and simultaneously display the target question-and-answer answer text and the visual reasoning graph generated based on the target knowledge graph in a side-by-side or hierarchical view. Specifically, in step S500, the basic analysis data corresponding to the target question-and-answer answer text is the complete target knowledge graph structured data corresponding to the target question-and-answer answer text.

[0035] Furthermore, the S500 step also includes an interactive verification step with an interactive, interpretable interface: S501: Based on the user's click operation on any statement (or key information point) in the target question and answer answer text, the system obtains one or more conclusion nodes corresponding to the statement in the underlying target knowledge graph.

[0036] S502, based on the conclusion node, the system automatically obtains all its upstream dependent nodes and connecting edges in the target knowledge graph, immediately constructs and generates a complete reasoning subgraph supporting the conclusion, and highlights the subgraph in the graph visualization area of ​​the interactive interface.

[0037] S503, based on the user's further click selection operation on any atomic fact node in the highlighted inference subgraph, the system obtains the precise source pointer information bound to that node.

[0038] S504, based on the source pointer (including document identifier, page number and coordinate information), locate the original source of the atomic fact in the corresponding target document view (such as a specific text line, table cell or image area) and highlight it visually.

[0039] The above implementation achieves the following results: This invention transforms the unstructured natural language reasoning chains (such as thought chains) in traditional methods into dynamic reasoning graphs that machines can directly parse and verify. Nodes in the graph represent atomic facts, and edges represent typified logical relationships, transforming the entire reasoning process from "descriptive text" into a "formal structure." This transformation enables machines to automatically verify reasoning paths, check logical consistency, and audit integrity, fundamentally solving the problem of the "black box" nature and inability to deeply examine the reasoning process in the background technology.

[0040] By forcibly binding each atomic fact node to a precise location in the source document (such as cell coordinates or text line positions), this invention achieves "cell-level" or "data point-level" tracing from the answer assertion to the original evidence. Simultaneously, the typified edges in the graph clearly reveal how evidence is combined into the final conclusion through logical operations such as comparison, inference, and calculation. This completely overcomes the shortcomings of background technologies (such as RAG) in terms of coarse granularity of citations and ambiguous logical relationships between evidence, providing a complete, transparent, and verifiable chain of evidence.

[0041] The dynamic reasoning graph employed in this invention is a non-linear, networked structure that allows conclusions to be drawn from multiple evidence nodes via different paths. This structure avoids the fragile linear chains upon which multi-hop reasoning in prior art relies. Even if a sub-path in the graph contains uncertainty or error, the system can still arrive at robust conclusions through other parallel evidence paths, thereby significantly improving the fault tolerance and overall robustness of reasoning for complex problems.

[0042] By designing an "evidence-constrained answer generator," the language generation process is strictly limited to the "translation" of the constructed reasoning graph. Furthermore, techniques such as probability masks are employed to physically prevent the model from introducing factual information outside the graph. This mechanism significantly suppresses the generation of "factual illusions" at the source, ensuring a high degree of fidelity between the output text and the source evidence—a reliability guarantee not found in traditional generation methods (including CoT and RAG).

[0043] The interactive explanation interface provided by this invention not only presents the answer but also deeply integrates it with the underlying reasoning graph and the original document. Users can intuitively and conveniently trace and verify the process step-by-step from the final conclusion to the reasoning subgraph, then to the atomic facts, and finally to the location of the original document. This closed-loop experience upgrades interpretability from "passive presentation" to "active exploration," greatly enhancing users' trust in and sense of control over the AI ​​output in critical decision-making scenarios.

[0044] In summary, this invention, through its technical system of "structured graph construction," "fine-grained evidence binding," "non-linear logic organization," "constraint content generation," and "interactive verification closed loop," systematically addresses the core pain points of current large models in complex document question answering, namely low transparency, difficulty in traceability, and poor reliability. This provides a solid technical foundation for trustworthy AI applications in high-value, high-risk fields such as financial analysis, legal judgment, and medical diagnosis.

[0045] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. It should also be understood that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A question-answering method for multimodal large models based on dynamic reasoning graphs, characterized in that, The method includes the following steps: S100, receive the target document set and the target question statement provided by the target user terminal; S200, the target question statement is input into the large language model to obtain the target intent label corresponding to the target question statement; S300, Obtain the target knowledge graph based on the target document set; S400, Traverse the target knowledge graph and generate the target question-and-answer text corresponding to the target question statement; S500, the target question-and-answer text and the corresponding basic analysis data are sent to the target user terminal.

2. The method according to claim 1, characterized in that, The preprocessing of the target document set required in step S100 includes the following steps: S101, Perform optical character recognition on the scanned documents or images in the target document set to extract text content; S102, using a document layout analysis model to perform structured parsing of the extracted text content and identify logical region divisions, so as to divide different text fragments, table cells or image regions according to the logical regions. S103 processes each locatable text segment, table cell, or image region to generate a precise source pointer containing document identifiers, page numbers, and coordinate information.

3. The method according to claim 1, characterized in that, According to the method of claim 1, step S300 includes specific implementation steps for constructing a target knowledge graph: S301, based on the target intent tag, extract all relevant atomic facts from the structured document data, and bind each atomic fact to its precise source pointer and use it as the initial node of the graph; S302, using a logical reasoning model, identify and establish typed logical relationship edges between the initial nodes based on the target intent label; S303, based on existing nodes and edges, perform iterative reasoning to generate new derived fact nodes and add them to the graph until the graph can support the solution to the target problem, forming the dynamic reasoning graph as the target knowledge graph.

4. The method according to claim 1, characterized in that, The specific steps in step S400 of generating the target question-and-answer text include: S401, Perform topological sorting on the target knowledge graph to determine the logical dependency order between nodes; S402, following the reasoning path from the original evidence node to the final conclusion node, the content of the node and the logical relationship between the nodes are transformed into a coherent natural language description; wherein, the generation process is constrained by a probability masking mechanism to prohibit the introduction of factual information outside the target knowledge graph.

5. The method according to claim 1, characterized in that, The interactive verification in step S500 specifically includes: S501, in response to the user's click operation on the assertion in the target question and answer answer text, obtain and highlight the complete reasoning subgraph supporting the assertion; S502, in response to the user's click operation on the atomic fact node in the reasoning subgraph, locate and highlight the original evidence position in the source document view according to the source pointer bound to the node.

6. The method according to claim 3, characterized in that, The iterative reasoning in step S303 is a dynamic and continuous graph expansion process. Specifically, the logical reasoning model continuously uses newly generated derived fact nodes as inputs for subsequent reasoning, identifies new logical relationships between them and other nodes, and creates new edges and nodes until one or more final conclusion nodes that can directly answer the target question statement are formed in the dynamic reasoning graph, and no new related reasoning paths can be expanded.

7. The method according to claim 3, characterized in that, The atomic facts are the smallest, indivisible units of information extracted from multimodal documents, possessing independent semantics. Their specific types include: Textual facts are sentences or phrases extracted from a document paragraph that express a complete factual statement; Tabular facts, that is, the values ​​or content in table cells, need to be defined in conjunction with their row and column headings; Chart-based facts are key data points, trend descriptions, or chart conclusions extracted from charts.

8. The method according to claim 4, characterized in that, The probability masking mechanism is implemented in the following way: at each time step when the answer generator decodes and generates the target question-and-answer answer text, the probability distribution of the words predicted by the model is restricted to a preset subset of words related to the content of the graph nodes to be translated in the current step and the edge type.

Images