A multi-modal multi-agent based document understanding and question answering method and system

Abstract

The application discloses a kind of based on multi-modal multi-agent's document understanding and question and answer method and system, the method is first to the document and generates text paragraph and page image for pre-processing, then adopt retrieval enhancement generation framework from text and page image two channels retrieval and query relevant context, obtain text candidate set and image candidate set;Then utilize five kinds of functional agents to carry out preliminary fusion and form candidate conclusion, form cross-modal key information limit subsequent analysis range, carry out fine-grained review and analysis in text and image modality respectively, output with page number, paragraph, area and other location indication intra-modal conclusion, finally to multi-source result carries out consistency analysis and weighted fusion generation final answer.The application can simultaneously integrate text and visual clues, improve complex multi-modal reasoning ability, improve accuracy in document question and answer and the like task.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering technology, and in particular to a document understanding and question answering method and system based on multimodal and multi-agent systems. Background Technology

[0002] Document comprehension and question answering (DocQA) aims to retrieve relevant information from a given document and generate a verifiable answer to a natural language question. Real-world documents are typically lengthy, spanning multiple pages, and have complex layouts, containing both textual information and visual elements (such as tables, graphs, page layouts, and annotations). Due to the high information density and its dispersion across different pages and modalities, the system must not only efficiently extract context from a large corpus but also establish consistent semantic and positional alignment between textual and visual evidence, and on this basis, perform reliable inference. Single-modal processing flows face limitations in this scenario: on the one hand, pure text-based processes struggle to fully utilize charts and layout cues; on the other hand, relying solely on image-based processes easily overlooks limiting conditions, definitions, and contextual dependencies within the text.

[0003] To alleviate the aforementioned challenges, existing technologies are mainly developing along two lines: one is text-based Retrieval-Augmented Generation (RAG), which indexes text fragments obtained through OCR or structural analysis and retrieves several fragments for use by the generation model during queries; the other is multimodal or image-based methods, including Large Vision-Language Models (LVLMs) and image RAGs, which directly encode page images and perform retrieval and inference within the visual domain. See details... Figure 1 The former (text-based approach) exhibits good scalability for pure text problems, but it lacks sufficient utilization of visual evidence such as tables, graphics, and layouts. Single-text links struggle to acquire and align with visual evidence in a timely manner, leading to missing evidence or misunderstandings, and it is susceptible to OCR and structural parsing errors. The latter (visual evidence) can access visual information, but in scenarios with long documents and multiple search results (e.g., Top-k, k≥2), common simple splicing or weak fusion strategies struggle to achieve selective integration and fine-grained focusing, making cross-modal consistency difficult to guarantee. Furthermore, when search results expand from Top-1 to Top-k, noise and redundancy increase significantly. Existing methods lack the ability to selectively integrate and focus across multiple pages and sources, and performance does not steadily improve with increasing context. In practical applications, when problems require simultaneous reliance on both textual and visual evidence for evidence collection and comparison, existing solutions are prone to information omissions or unresolved contradictions. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a document understanding and question answering method and system based on multimodal and multi-agent systems, which can work collaboratively at both the retrieval and reasoning levels and maintain traceability both within and outside the modality. This solves the technical problems of insufficient cross-modal collaboration, long document processing capabilities, and weak information integration capabilities in the prior art, and improves the selective integration capability of cross-modal evidence and the robustness of long document processing.

[0005] Technical solution: The document understanding and question answering method based on multimodal multi-agent technology described in this invention includes the following steps:

[0006] Input the document containing text and images to be understood, along with the question expressed in natural language;

[0007] Extract text from each page of the document to generate text paragraphs; generate page images from each page of the document;

[0008] A text search engine is used to search within text paragraphs based on the question, resulting in a text candidate set; an image search engine is used to search within page images based on the question, resulting in an image candidate set.

[0009] A preliminary answer is generated using a general-purpose intelligent agent based on the question, the text candidate set, and the image candidate set.

[0010] The key intelligent agent generates textual key information and textual descriptions of visual key information based on the question, text candidate set, image candidate set and preliminary answer;

[0011] A text-based intelligent agent is used to generate a semantically understood answer based on the question, a set of text candidates, and key text information; an image-based intelligent agent is used to generate a visually understood answer based on the question, a set of image candidates, and a text description of key visual information.

[0012] A comprehensive intelligent agent is used to generate a final answer based on the question, preliminary answer, semantically understood answer, and visually understood answer.

[0013] Among them, the text agent is a large text model, while the general agent, key agent, image agent, and comprehensive agent are all large visual models.

[0014] Furthermore, the text extraction and generation of text paragraphs for each page of the document includes:

[0015] For each page of the document, if the page is an image or contains scanned content, the OCR method is used to recognize the text; if the page only contains parsable digital text, the text and layout structure are extracted using PDF parsing methods.

[0016] If the document on this page contains both images or scanned content and parsable digital text, then the result with higher confidence is selected as the text segment from both the OCR recognition result and the PDF parsing result; or, the PDF parsing result is used as the main component, and the OCR recognition result is used to complete missing areas and correct low-confidence segments.

[0017] Furthermore, a text search engine is used to search within text paragraphs based on the question, resulting in a text candidate set; an image search engine is used to search within page images based on the question, resulting in an image candidate set including:

[0018] A vectorized index is built for the text paragraphs using a text retrieval tool. The similarity between the text paragraphs and the question is calculated. Several text segments are selected from all text paragraphs in descending order of similarity to obtain a text candidate set.

[0019] The visual context representation of each page image is extracted using an image retrieval tool. The similarity between the page image and the question is calculated. Several text images are selected from all page images in descending order of similarity to obtain an image candidate set.

[0020] Furthermore, the key agent identifies and confirms the key information required to answer the question, serving as orientation clues for the text agent and the image agent; the text key information is used to limit the reasoning scope of the text agent, and the text description of the visual key information is used to indicate the pages and areas of interest to the image agent.

[0021] Furthermore, the text agent focuses on the reasoning scope defined by the key information in the text, comprehensively analyzes the logical connection, causal inference and semantic consistency between paragraphs, and generates a structured semantic understanding answer;

[0022] Guided by the textual description of the key visual information, the image agent analyzes the layout structure, chart elements, annotations, and numerical readings to complete focused reasoning and generate a visual understanding of the case.

[0023] Furthermore, the generation of the final answer using a comprehensive intelligent agent based on the question, preliminary answer, semantically understood answer, and visually understood answer includes:

[0024] If the semantic understanding answer and the visual understanding answer reach the same conclusion, then the consistent conclusion shall prevail, and the final answer shall be fully expressed by combining the contextual information in the preliminary answer.

[0025] If the semantic understanding answer and the visual understanding answer are inconsistent, a final answer is generated based on the question, including: if the question asks for semantic content, the semantic understanding answer is used in combination with the contextual information in the preliminary answer to form a complete expression of the final answer; if the question asks for numerical values, readings, or graph relationships, the visual understanding answer is used in combination with the contextual information in the preliminary answer to form a complete expression of the final answer.

[0026] Furthermore, the text retrieval agent is ColBERTv2, the image retrieval agent is ColPali, the text agent uses Llama-3.1-8B-Instruct as the basic model, and the general agent, key agent, image agent and comprehensive agent all use Qwen2-VL-7B-Instruct as the basic model.

[0027] The document understanding and question answering system based on multimodal and multi-agent systems described in this invention includes:

[0028] The preprocessing module is used to extract text from each page of the document, generating text paragraphs, and generating page images from each page, based on the input document containing text and images to be understood and the questions expressed in natural language.

[0029] The multimodal contextual retrieval module is used to use a text retrieval device to search within text paragraphs based on the question, obtaining a text candidate set; and to use an image retrieval device to search within page images based on the question, obtaining an image candidate set.

[0030] A multimodal, multi-agent answer generation module is used to generate a preliminary answer using a general agent based on the question, a text candidate set, and an image candidate set;

[0031] The key intelligent agent generates textual key information and textual descriptions of visual key information based on the question, text candidate set, image candidate set and preliminary answer;

[0032] A text-based intelligent agent is used to generate a semantically understood answer based on the question, a set of text candidates, and key text information; an image-based intelligent agent is used to generate a visually understood answer based on the question, a set of image candidates, and a text description of key visual information.

[0033] A comprehensive intelligent agent is used to generate a final answer based on the question, preliminary answer, semantically understood answer, and visually understood answer.

[0034] Among them, the text agent is a large text model, while the general agent, key agent, image agent, and comprehensive agent are all large visual models.

[0035] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the document understanding and question answering method based on multimodal and multi-agent systems.

[0036] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the document understanding and question answering method based on multimodal multi-agent systems.

[0037] Beneficial effects: Compared with the prior art, the advantages of this invention are: by constructing a multimodal fusion architecture, introducing a multi-agent collaborative mechanism, and combining it with a dynamic workflow for long documents, this invention achieves deep understanding and efficient processing of complex text and image mixed documents. Specifically:

[0038] (1) Significantly improved cross-modal understanding ability: Through the processing flow of "general, key, specialized and comprehensive", the integrated selection and alignment of text and visual evidence is realized, reducing noise interference caused by long documents and multi-page retrieval;

[0039] (2) Optimization of long document processing efficiency: Hybrid retrieval provides targeted context, and key agents can deeply explore and output traceable evidence within this modality, reducing the intervention of irrelevant content and improving overall efficiency, while ensuring accuracy and reducing unnecessary computational overhead;

[0040] (3) Wide range of application scenarios: The text and visual retrieval tools and the basic model are interchangeable, adaptable to different deployments and hardware conditions, and maintain the universality and stable gains of the framework. Attached Figure Description

[0041] Figure 1 This is a system architecture diagram of existing single-modal context retrieval and LVLM technologies.

[0042] Figure 2 This is a schematic diagram of the multimodal multi-agent document understanding and question answering structure according to an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the multimodal multi-agent document understanding and question answering process according to an embodiment of the present invention. Detailed Implementation

[0044] Example 1

[0045] In retrieval enhancement generation methods, a searchable representation of the document is typically constructed first, and then the retrieved context is provided to the generative model to generate an answer. This type of method includes two common branches: one is text RAG, which indexes text fragments obtained through OCR parsing and retrieves relevant content from the fragment set; the other is image RAG, which visually encodes page images and retrieves Top-k relevant pages before generating the answer. Although RAG can narrow down the context in long document scenarios, it still has the following limitations:

[0046] 1) Long document load and relevance sparsity: The amount of information is huge, and directly processing the entire document is limited by computing resources, and it is difficult to reliably identify the fine-grained evidence most relevant to the problem;

[0047] 2) Insufficient use of visual evidence: When the answer relies on visual elements such as tables, graphics or page layout, it is difficult for a single text link to obtain and align with the visual side in a timely manner, which can easily lead to missing evidence or misunderstanding.

[0048] 3) Lack of cross-modal synthesis: Most RAG schemes only retrieve text or only images, lacking the ability to simultaneously retrieve and align the two modalities and perform evidence-level synthesis;

[0049] In end-to-end approaches based on large visual language models (LVLMs), a common practice is to directly use page images as input for understanding and generation. This approach explicitly utilizes visual evidence, but it still faces the following challenges:

[0050] 1) Cross-modal consistency is difficult to guarantee: There is a lack of mechanisms to guide and verify the key information on both sides of the text and the image, which can easily lead to inconsistent conclusions;

[0051] 2) Insufficient fine-grained text and complex interactions: LVLM has advantages in processing visual content, but performs poorly in scenarios that require fine text details or complex interactions between text and visual elements.

[0052] 3) Insufficient robustness: When the search results are expanded from Top-1 to Top-k, noise and redundancy increase significantly. Existing methods are not capable of selective integration and focusing among multiple pages and multiple sources of evidence, and their performance does not improve steadily with the increase of context.

[0053] To address the above issues, this invention proposes a document understanding and question answering method based on multimodal and multi-agent approaches. Targeting real-world documents containing both text and visual elements, this method constructs a multimodal fusion architecture, introduces a multi-agent collaborative mechanism, and combines it with a dynamic workflow for long documents to achieve deep understanding and efficient processing of complex documents with mixed text and graphics.

[0054] The following is combined Figure 2The method described in this invention is explained in the following three parts.

[0055] I. Multimodal Fusion Architecture.

[0056] a) Dual-channel searchable representation and alignment: Perform OCR and / or PDF parsing on the input document to obtain two types of representations: a set of text fragments and page-level images; establish searchable indexes for each text fragment and establish an explicit mapping relationship (such as page number, segment, block identifier or region indicator) between each text fragment and its page for subsequent cross-modal positioning and alignment.

[0057] b) Paragraph-level retrieval and candidate formation in text channels: In text channels, the query problem is based on the text index. Calculate relevance and select the top-k segments from the full text fragments to form a text candidate set. At the same time, the correspondence between each candidate and its source page and segment identifier is retained as the targeted text context for subsequent analysis;

[0058] c) Page or region retrieval and structure parsing in the visual channel: In the visual channel, the problem is solved based on the page image index. Calculate the correlation and select the top-k pages to form an image candidate set. Perform layout or element parsing on candidate pages (such as title areas, chart areas, table areas, labels and axis labels, etc.) to generate page or area indicators that can be used for focusing, serving as a directional visual context for subsequent visual analysis, and maintaining consistent page-paragraph-region alignment with the text channel.

[0059] II. Multi-agent cooperative system

[0060] a) General intelligent agent: Receives text and visual candidates, performs preliminary fusion of cross-modal contexts, forms a problem-oriented semantic framework and candidate conclusions, and produces prompts for further refinement;

[0061] b) Key agents: Based on the output of the general agent, identify cross-modal key information used to answer the question, including terms, values, tables, graphical elements and their page positions;

[0062] c) Text-based intelligent agent: Within the text domain, it understands and reasons about the key information mentioned above, extracts the key text points directly related to the question, and provides a textual sub-answer.

[0063] d) Image intelligent agent: Within the visual domain, it focuses on the corresponding page or area based on key information, analyzes visual cues such as layout structure, legends and axis labels, table cells and numerical readings, and generates image sub-answers;

[0064] e) Integrated intelligent agent: integrates the outputs of general, text and image intelligent agents to identify common, different and complementary information; resolves and unifies inconsistencies to form the final answer.

[0065] III. Dynamic Workflow.

[0066] a) Preprocessing stage: Perform OCR and / or PDF parsing on the input document to obtain two inputs: a set of text fragments and a page-level image representation, while retaining positional information such as pages, paragraphs, and regions;

[0067] b) Retrieval Phase: A hybrid retrieval strategy is employed to calculate text relevance and visual similarity separately, and the Top-k candidates from the two candidate pools are selected as the directional context (text candidates). Image Candidates );

[0068] c) Analysis phase: The general agent performs preliminary multimodal integration on the rearranged candidates; the key agent extracts cross-modal key information from it. This limits the scope of subsequent analysis. Then, the text agent and the image agent each conduct fine-grained analysis around key information within their respective modalities, outputting the answer (…). );

[0069] d) Output Stage: Based on factors such as consistency, coverage, source diversity, and format matching, the intelligent agent performs weighted fusion and conflict resolution on multi-source conclusions to generate the final output. .

[0070] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0071] like Figure 3 As shown, the document understanding and question answering method based on multimodal multi-agent systems uses parallel text and image retrieval as its foundation to provide directional context for the question. On this basis, a general agent performs preliminary multimodal processing and provides candidate answers, while a key agent extracts key information from the retrieved text and pages. Subsequently, text and image agents perform fine-grained analysis around this key information, and finally, a comprehensive agent fuses the results of each agent and resolves conflicts to output the final answer. The overall process includes: ① Document preprocessing; ② Multimodal context retrieval; ③ Preliminary analysis and key information extraction; ④ Fine-grained analysis by specialized agents; ⑤ Answer synthesis and final output.

[0072] The task definition of the method described in this invention is: given a question q expressed in natural language and its corresponding document set. The goal is to generate an answer 'a' that accurately and comprehensively answers the question, and the answer must be based on the document. The information provided in the document.

[0073] The following describes each step of the method described in this invention.

[0074] Step 1, document preprocessing.

[0075] To enable the simultaneous use of textual and visual information in subsequent processes, this invention first transforms the original document set into a unified input suitable for both textual and visual analysis. The documents are then standardized page by page, establishing a page-level order and basic metadata to ensure that subsequent processes can perform independent and consistent analysis on each page.

[0076] (1.1) Text extraction.

[0077] Let the document set be denoted as Each of them For each page of a document. The system employs a combination of Optical Character Recognition (OCR) and PDF parsing to extract text. This combined approach provides robust text extraction across various document formats and structures. When the page is an image or contains scanned content, OCR technology is used to recognize the text. When the page contains parsable digital text, the text and layout structure are directly extracted via PDF parsing. When both results coexist, the one with higher confidence is prioritized, with the PDF parsing result as the primary source and the OCR result used for filling in missing areas and proofreading low-confidence segments. The extracted text is represented by paragraphs or text fragments as follows:

[0078]

[0079] in Page The number of text fragments included.

[0080] (1.2) Image saving

[0081] Each page is also preserved as a page image to maintain the original layout and visual elements (such as tables, charts, headings, footnotes, etc.), thus providing input for subsequent visual analysis.

[0082] Through the above processing, the document set simultaneously forms two parallel representations: one is a text representation composed of text fragments from each page, and the other is a visual representation composed of original page images from each page; both serve as the basic input for subsequent multimodal retrieval and agent collaborative reasoning, enabling a comprehensive understanding of the document content.

[0083] Step 2, Multimodal Contextual Retrieval.

[0084] This stage aims to consider both text and image modalities simultaneously, retrieving the most relevant context to question q from the document set obtained in the preprocessing stage, and producing a text candidate set. With image candidate set As a reference context for subsequent agent analysis, the overall process is shown in Table 1. The system first establishes vector libraries for the text fragments and page images of each page in the document set, given a problem. Then, the similarity between q and text and image is calculated separately, and the top-ranked candidates from the candidate sets of each modality are selected. One result.

[0085] (2.1) Text retrieval

[0086] Use a text search engine to search each page Extracted text fragment sequence Build a vectorized index. Given a user question. For each text fragment Calculation and Text similarity score This forms a mapping table of "text fragment → score". Sort by score in descending order. The top-k text segments are selected from the list and denoted as:

[0087]

[0088] The k most similar text fragments retrieved As the text context for subsequent Agents; if there are tied scores, the tie can be broken according to preset stable rules such as segment length or page order.

[0089] (2.2) Image retrieval

[0090] Similar to text retrieval, an image retrieval tool extracts visual contextual representations from the images on each page. ,in For the number of visual tokens on the page, For embedding dimensions. Based on the question. Similarity score calculated with page image embedding This yields a mapping table for "page → score". Select the top-k page images from highest to lowest score, denoted as:

[0091]

[0092] Similarly, the k most similar page images retrieved As a visual context for subsequent agents, it provides layout, charts and other visual cues; if there are ties in scores, the same preset stable rules are used to make the ranking decision.

[0093] Table 1 Multimodal Context Retrieval Algorithm

[0094]

[0095] Step 3: Preliminary analysis and key information extraction.

[0096] The goal of this stage is: to address the problem. An initial interpretation is performed, and key information is located within the retrieved context to provide clues for subsequent specialized agents. This stage is completed sequentially by two agents.

[0097] (3.1) General intelligent agent

[0098] System settings for general intelligent agents As a preliminary multimodal integration unit, it also receives the text context obtained from the second stage retrieval. With visual context The agent combines information from both modalities to form a holistic understanding of the context, enabling it to generate a preliminary answer. Their relationship can be expressed as:

[0099]

[0100] This candidate answer provides a key starting point for the next stage of more granular analysis.

[0101] (3.2) Key Intelligent Agents

[0102] Key intelligent agents Refine the search results. The input includes questions. Text context Visual context And the preliminary answers generated by the general intelligent agent. Its main function is to identify and confirm the key information needed to answer questions, and to use it as directional cues for the next specialized agent. Accordingly, the key agent outputs a set of key information from the text side. (such as clips), and textual descriptions of key visual information. (Such as the emphasis on a page or region) It originates from the emphasis on Extraction of key visual elements covered:

[0103]

[0104]

[0105] In summary, the output of this stage With preliminary answer .in, Used to limit the scope of reasoning on the text side. This is then used to indicate pages and areas requiring special attention on the image side. Subsequently, in step 4, the text agent and the image agent will perform fine-grained analysis within their respective modalities based on these clues.

[0106] Step 4: Fine-grained analysis of the dedicated intelligent agent.

[0107] This stage delves deeper into the understanding of both text and visual modalities, focusing on utilizing the key information extracted by the key agents in the previous stage to perform more detailed reasoning and analysis on the retrieved context. By introducing specialized text and image agents, the system can perform targeted focusing processing within their respective modalities, thereby obtaining more accurate intermediate results.

[0108] (4.1) Text-based intelligent agents

[0109] Text agent A T The input is the question. , retrieved text fragments With key intelligent agents Key text information output This agent operates only within the text domain, parsing input based on a Large Language Model (LLM) and possessing fine-grained semantic analysis and reasoning capabilities. It will focus primarily on... The indicated text fragment, for We re-understand and filter the semantic context to extract the evidence information most relevant to the problem.

[0110] In this process, the text agent not only judges literal matching relationships but also comprehensively analyzes the logical connections, causal inferences, and semantic consistency between paragraphs, thereby integrating and summarizing textual information at a deeper level. Through this focused reading and reasoning, the system can capture subtle but crucial semantic clues in a large number of documents, generating structured textual answers, i.e., semantically understood answers.

[0111]

[0112] The result This represents the accurate response generated based on semantic understanding in the text modality, which constitutes one of the important inputs for subsequent multimodal synthesis.

[0113] (4.2) Image-based intelligent agents

[0114] Image agent A I The input includes questions Image retrieval result set and key visual information This intelligent agent is based on a Visual Language Model (VLM) and focuses on the analysis and interpretation of visual content. Specifically, it presents the problem... With the retrieved page collection The information is fed into the VLM (Visual Modeling Library) along with key visual information. Under the guidance of the model, The indicated page or area receives higher attention, with priority given to analyzing its layout structure, chart elements, labels, and numerical readings. Based on this, focused reasoning is performed to generate a visually comprehension-based answer, i.e., a visually comprehension answer.

[0115]

[0116] In summary, text and image agents perform deep analysis within their respective modalities: the former ensures semantic and logical consistency, while the latter ensures the visual verifiability derived from the page's visual representation. The output at this stage includes the text answer. With image answer They will include preliminary answers. The results are then input together into the next stage for answer synthesis, in order to achieve joint reasoning and result fusion across modal information.

[0117] Step 5: Answer synthesis and final output.

[0118] This stage integrates the multi-source results generated in the previous stages, combining preliminary multimodal understanding with the analysis of two types of specialized agents to generate a complete and accurate final answer. Specifically, it receives answers generated by a general agent, a text agent, and an image agent, respectively. Based on this, complete the comprehensive reasoning and output the final answer:

[0119]

[0120] To avoid simple splicing, integrated intelligent agents When summarizing, the approach should prioritize "comparison first, then breakdown, and finally integration." Firstly, regarding... The key conclusions and supporting evidence were compared one by one to identify commonalities, differences, and complementary information:

[0121] 1) When the textual and visual conclusions are consistent on the core points, the consistent conclusion shall prevail, and shall be combined with... Use the provided context to fully express your answer, ensuring that it covers the key points of the question and is presented naturally and fluently.

[0122] 2) If there is a disagreement between the two sides' conclusions, then address the relevant issues. A trade-off must be made between direct relevance and information completeness. When When asking for semantic content such as definitions and conditions, text-based queries should be used preferentially. Combination Provide a complete description within the given context; when When numerical values, readings, or graphical relationships are required, the visual side should be used first. Combination Provide a complete description within the given context.

[0123] In summary, the system generates a single answer. This answer has both... Overall grasp, Text details and The key visual points for the problem Provide a complete and accurate response, which serves as the final output of the multimodal multi-agent process of this invention.

[0124] The multi-agent collaboration process can be found in Table 2.

[0125] Table 2 Multi-agent cooperative algorithm

[0126]

[0127] The method described in this invention will be verified through specific experiments below.

[0128] 1. Experimental setup

[0129] (1) Experimental details

[0130] MDocAgent comprises five agents: a general agent, a key agent, a text agent, an image agent, and a comprehensive agent. Specifically, the text and image retrieval agents use ColBERTv2 and ColPali, respectively. The text agent uses Llama-3.1-8B-Instruct as its base model, while the other four agents use Qwen2-VL-7B-Instruct. In the RAG settings, the top-1 or top-4 segments are retrieved for each sample as the input context.

[0131] (2) Dataset

[0132] Five benchmark datasets—MMLongBench, LongDocURL, PaperTab, PaperText, and FetaTab—were used for evaluation. These datasets cover various scenarios, including open and closed domains, text and visual data, and long and short documents, to ensure the fairness and completeness of the evaluation.

[0133] (3) Evaluation indicators

[0134] Following existing work, GPT-4o was used as the evaluation model for each benchmark. It was used to make a binary classification judgment (correct or incorrect) on the consistency between the model output and the reference answer, and the average accuracy was reported.

[0135] (4) Intelligent agent Prompt design

[0136] (4.1) General intelligent agent

[0137] You are a high-level agent capable of analyzing both text and images simultaneously. Your task is to accurately answer the user's questions using the provided textual and visual information.

[0138] Extract text from both ends: If the image contains text, extract it, taking into account both the text in the image and the provided text content.

[0139] Analyzing visual and textual information: Combining details in images (such as objects, scenes, or patterns) with text to form a comprehensive understanding of the content.

[0140] Answer: Use relevant details from the image and text to give a clear, accurate, and context-aware response.

[0141] Answer criteria:

[0142] If images and text contain similar or overlapping information, cross-check them and use them together to ensure consistency.

[0143] If the image contains information that is not in the text but is relevant to the question, please include it in your answer.

[0144] If the text and images provide conflicting details, please explain the differences and cite the most reliable source.

[0145] (4.2) Key Intelligent Agents

[0146] Based on all the given information, please provide the key information corresponding to the text and image, and return it as a Python dictionary. Only valid dictionary entries are allowed; no extra text should be included. The format is as follows:

[0147] {"text": "critical information for text", "image": "critical information for image"}

[0148] (4.3) Text-based intelligent agents

[0149] You are a text analysis agent. Your job is to extract key information from text and use it to accurately answer users' questions.

[0150] Task:

[0151] Extract key details and focus on the most important facts, data, or opinions relevant to the issue.

[0152] Understand the context and pay attention to the meaning and details.

[0153] Use the extracted information to provide concise and relevant answers, offering clear responses.

[0154] (4.4) Image-based intelligent agents

[0155] You are a high-level image processing agent, specializing in analyzing and extracting information from images. Images may include document screenshots, illustrations, or photographs.

[0156] Task:

[0157] Use OCR to extract text information from images.

[0158] Analyze visual content to identify relevant details (such as objects, patterns, and scenes).

[0159] By combining textual and visual information, it provides accurate and context-aware answers to users' questions.

[0160] (4.5) Integrated Intelligent Agent

[0161] Your task is to aggregate and evaluate the answers provided by multiple agents. You have access to the following information:

[0162] Answer:

[0163] The individual answers from each of the intelligent agents.

[0164] Task:

[0165] Analysis: Evaluate the quality, consistency, and relevance of each answer, and identify commonalities, differences, or gaps in reasoning.

[0166] In conclusion, the most accurate and reliable information can be synthesized based on this evidence.

[0167] Final Answer: Provide a final, well-reasoned answer. Your conclusion should reflect consensus (if any), or choose the most credible and well-supported answer.

[0168] 2. Explanation of Experimental Results

[0169] This document systematically compares the described multimodal multi-agent framework with state-of-the-art RAG-based methods and their LVLM methods on multiple benchmarks. Detailed results are shown in Table 3. Overall, MDocAgent outperforms all compared methods on all benchmarks.

[0170] In the Top-1 retrieval setting, relying on only one text fragment and one page of images, MDocAgent demonstrates a significant performance improvement. It achieves a score of 0.389 on the PaperText benchmark, a 13.7% improvement over the second-best method, M3DocRAG; and a score of 0.590 on the FetaTab benchmark, an 18.7% improvement over the second-best method. Compared to the best LVLM (Qwen2.5-VL-7B) and the best text RAG baseline (ColBERTv2+Llama-3.1-8B), MDocAgent achieves average improvements of 48.1% and 20.7% across all benchmarks, respectively. Furthermore, compared to the state-of-the-art image RAG method, M3DocRAG, MDocAgent achieves an average improvement of 9.4%. These results indicate that incorporating visual information and employing a collaborative multi-agent structure helps to more effectively alleviate information overload, focus on fine-grained key points, and improve cross-modal understanding.

[0171] Under the Top-4 retrieval setting, with each sample using 4 text fragments and 4 pages of images, MDocAgent's advantages are further demonstrated: consistently achieving the highest scores across all benchmarks. On average, it shows a 70.3% improvement compared to Qwen2.5-VL-7B; and average improvements of 9.0% and 5.0% compared to M3DocRAG and ColBERTv2+Llama-3.1-8B, respectively. Notably, M3DocRAG's performance in the Top-4 setting is slightly lower than ColBERTv2+Llama-3.1-8B, suggesting limitations in its selective integration capabilities in multi-evidence scenarios. The above demonstrates that this method can effectively utilize the additional contextual information provided by Top-4 retrieval, achieving greater benefits as the number of search results increases.

[0172] Table 3 Performance comparison of MDocAgent with LVLMs and RAG-based methods

[0173]

[0174] 3. Quantitative analysis.

[0175] To evaluate the effectiveness and contribution of each component of the proposed multimodal multi-agent framework, we conducted a quantitative analysis in three aspects: First, we performed ablation experiments to examine the impact of removing a single agent or a combination of agents on the overall performance; second, we performed fine-grained analysis on MMLongBench and LongDocURL according to the information source type to pinpoint the sources of performance improvement; and third, we evaluated the replaceability of the image-side RAG backend to verify the stability and generalization ability of the framework under different implementations.

[0176] (1) Ablation research

[0177] To evaluate the role of each agent within the framework, the complete method (MDocAgent) and several variants were compared, and the results are shown in Table 4. First, the text-removing agent ( ) or image intelligence ( Both approaches lead to significant performance drops, indicating that introducing specialized agents for text and visual modalities respectively is crucial for overall performance; this difference is most pronounced on benchmarks such as LongDocURL and PaperText. The former is richer in visual cues, while the latter is richer in textual cues, indirectly confirming the value and necessity of specialized processing. These results demonstrate that setting up specialized agents for each modality and enabling them to cooperate is a key factor in achieving performance improvements.

[0178] Simultaneously, with the removal of general-purpose intelligent agents and key intelligent agents (MDocAgent) s In contrast, the complete approach maintains a consistent advantage across all datasets, highlighting the importance of these two types of agents: the general agent performs initial integration of text and image modalities early in the process, providing a holistic understanding of the context; without this integration step, the ability of subsequent agents to focus on key information and respond effectively is significantly weakened. Building upon this, the key agent is used to identify and extract crucial information necessary for problem-solving; removing the key agent limits the framework's ability to effectively locate and utilize key information, thus weakening the orientation and efficiency of subsequent specialized agents. In summary, the ablation experiments clearly demonstrate that the general agent and the key agent provide the necessary integration and focusing mechanisms for specialized agents, and the synergy of all three can produce a significant gain effect.

[0179] Table 4 Performance comparison of different MDocAgent variants (G is the general agent, C is the key agent)

[0180]

[0181] (2) Fine-grained detail analysis

[0182] To further understand the performance improvements brought about by the proposed method, an in-depth analysis of its performance across different information source modalities is conducted. Specifically, Table 5 breaks down the scores of MMLongBench. The results show that MDocAgent outperforms all LVLM baseline methods across all data modalities. For comparisons using RAG methods with Top-1 retrieval, although M3DocRAG has a slight advantage in the Figure category, MDocAgent performs stronger in the Chart, Table, and Text categories, indicating that the framework is more advantageous when handling both textual and visual elements simultaneously. Increasing the retrieval depth to Top-4 further improves MDocAgent's performance across all categories, especially in the Figure category, demonstrating that this method can more effectively utilize and maintain a stable advantage when facing larger-scale and more diverse retrieval contexts.

[0183] Table 5 Performance comparison of different information sources on MMLongBench

[0184]

[0185] Meanwhile, the fine-grained performance of LongDocURL was evaluated, and the results are shown in Table 6. Similar to the conclusions of MMLongBench, MDocAgent outperforms various LVLM baseline methods on this benchmark. In Top-1, MDocAgent performs better in categories such as Layout, Text, and Table, achieving the highest average accuracy of 0.516. When using the Top-4 retrieval strategy, MDocAgent further improves its performance in all categories, averaging 0.577, demonstrating the comprehensive advantages of this method in understanding cross-page layouts, determining long text, and reading table elements.

[0186] Table 6 Performance Comparison of Different Information Sources on LongDocURL

[0187]

[0188] (3) Compatibility analysis

[0189] To evaluate the compatibility of the proposed method with different RAG backends, we integrated two page-level image RAGs, ColPali and ColQwen2-v1.0, into the framework. The results are shown in Table 7. Both achieved comparable performance overall, with an average score of 0.407 across all benchmarks. ColQwen2-v1.0 performed slightly better on MMLongBench, LongDocURL, and FetaTab, while ColPali performed slightly better on PaperTab and PaperText.

[0190] The above results demonstrate that the choice of image-side RAG model has a relatively small impact on overall effectiveness, highlighting the robustness of the proposed multi-agent architecture. Furthermore, the consistent performance across different RAG models further illustrates that the core advantage of the method stems from the multi-agent architecture itself, rather than relying on a specific retrieval model, thus reinforcing the compatibility conclusions of this method.

[0191] Table 7 Performance comparison of image-based RAG models using ColPali and ColQwen2-v1.0 as MDocAgent

[0192]

[0193] In summary, this invention targets real-world documents containing both text and visual elements, employing a Retrieval Enhanced Generation (RAG) framework to retrieve and query relevant context from both text and page image channels. Based on this, five functional agents are established: a general agent, a key agent, a text agent, an image agent, and a comprehensive agent. The general agent performs preliminary fusion of cross-modal context and forms candidate conclusions; the key agent identifies crucial cross-modal information essential for solving the problem, limiting the scope of subsequent analysis; the text agent and the image agent perform fine-grained review and parsing within their respective modalities, outputting in-modal conclusions with location indicators such as page numbers, paragraphs, and regions; and the comprehensive agent performs consistency analysis and weighted fusion of multi-source results to generate the final document understanding output. This collaborative mechanism can simultaneously integrate textual and visual cues, enhancing complex multimodal reasoning capabilities and improving accuracy in tasks such as document question answering. Experimental results show that, on multiple publicly available benchmarks, including MMLongBench and LongDocURL, the proposed solution achieves significant improvements over existing methods, with an average improvement of approximately 12.1%. This invention is applicable to scenarios such as retrieval, extraction, location, and question answering of long documents with mixed text and graphics, such as technical manuals and financial reports.

[0194] Example 2

[0195] The document understanding and question answering system based on multimodal and multi-agent systems described in this invention includes:

[0196] The preprocessing module is used to extract text from each page of the document, generating text paragraphs, and generating page images from each page of the document, based on the input document containing text and images to be understood and the questions expressed in natural language.

[0197] The multimodal contextual retrieval module is used to use a text retrieval device to search within text paragraphs based on the question, obtaining a set of text candidates; and to use an image retrieval device to search within page images based on the question, obtaining a set of image candidates.

[0198] A multimodal, multi-agent answer generation module is used to generate a preliminary answer using a general agent based on the question, a text candidate set, and an image candidate set;

[0199] The key intelligent agent generates textual key information and textual descriptions of visual key information based on the question, text candidate set, image candidate set and preliminary answer;

[0200] A text-based intelligent agent is used to generate a semantically understood answer based on the question, a set of text candidates, and key text information; an image-based intelligent agent is used to generate a visually understood answer based on the question, a set of image candidates, and a text description of key visual information.

[0201] A comprehensive intelligent agent is used to generate a final answer based on the question, preliminary answer, semantically understood answer, and visually understood answer.

[0202] Among them, the text agent is a large text model, while the general agent, key agent, image agent, and comprehensive agent are all large visual models.

[0203] Example 3

[0204] The computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements the document understanding and question answering method based on multimodal and multi-agent systems.

[0205] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory or any other medium that can be used to store program code in the form of instructions or data structures and is accessible by a computer.

[0206] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

[0207] Example 4

[0208] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the document understanding and question answering method based on multimodal multi-agent systems.

Claims

1. A document understanding and question answering method based on multimodal multi-agent systems, characterized in that, Includes the following steps: Input the document containing text and images to be understood, along with the question expressed in natural language; Extract text from each page of the document to generate text paragraphs; generate page images from each page of the document; A text search engine is used to search within text paragraphs based on the question, resulting in a text candidate set; an image search engine is used to search within page images based on the question, resulting in an image candidate set. A preliminary answer is generated using a general-purpose intelligent agent based on the question, the text candidate set, and the image candidate set. The key agent generates textual key information and textual descriptions of visual key information based on the question, text candidate set, image candidate set, and preliminary answer. The key agent identifies and confirms the key information required to answer the question, serving as orientation clues for the text agent and image agent. The textual key information is used to limit the reasoning scope of the text agent, and the textual descriptions of the visual key information are used to indicate the pages and areas of interest for the image agent. A text-based intelligent agent generates a semantically understood answer based on the question, a set of candidate texts, and key textual information. An image-based intelligent agent generates a visually understood answer based on the question, a set of candidate images, and a textual description of key visual information. The text-based intelligent agent focuses on the reasoning scope defined by the key textual information, comprehensively analyzes the logical connections, causal inferences, and semantic consistency between paragraphs, and generates a structured semantically understood answer. Guided by the textual description of the key visual information, the image-based intelligent agent pays more attention to the pages and areas indicated by the textual description of the key visual information, analyzes the layout structure, chart elements, annotations, and numerical readings, completes focused reasoning, and generates a visually understood answer. A comprehensive intelligent agent is used to generate a final answer based on the question, preliminary answer, semantically understood answer, and visually understood answer. If the semantic understanding answer and the visual understanding answer reach the same conclusion, then the consistent conclusion shall prevail, and the final answer shall be fully expressed by combining the contextual information in the preliminary answer. If the semantic understanding answer and the visual understanding answer are inconsistent, a final answer is generated based on the question, including: if the question asks for semantic content, the semantic understanding answer is used in combination with the contextual information in the preliminary answer to form a complete expression of the final answer; if the question asks for numerical values, readings, or graph relationships, the visual understanding answer is used in combination with the contextual information in the preliminary answer to form a complete expression of the final answer. Among them, the text agent is a large text model, while the general agent, key agent, image agent, and comprehensive agent are all large visual models.

2. The document understanding and question answering method based on multimodal multi-agent as described in claim 1, characterized in that, Extracting text from each page of the document to generate text paragraphs includes: For each page of the document, if the page is an image or contains scanned content, the OCR method is used to recognize the text; if the page only contains parsable digital text, the text and layout structure are extracted using PDF parsing methods. If the document on this page contains both images or scanned content and parsable digital text, then the result with higher confidence is selected as the text segment from both the OCR recognition result and the PDF parsing result; or, the PDF parsing result is used as the main component, and the OCR recognition result is used to complete missing areas and correct low-confidence segments.

3. The document understanding and question answering method based on multimodal multi-agent as described in claim 1, characterized in that, A text search engine is used to search through text paragraphs based on the question, resulting in a candidate text set. Using an image retrieval tool, a search is performed on the page images based on the question, resulting in a candidate image set including: A vectorized index is built for the text paragraphs using a text retrieval tool. The similarity between the text paragraphs and the question is calculated. Several text segments are selected from all text paragraphs in descending order of similarity to obtain a text candidate set. The visual context representation of each page image is extracted using an image retrieval tool. The similarity between the page image and the question is calculated. Several text images are selected from all page images in descending order of similarity to obtain an image candidate set.

4. The document understanding and question answering method based on multimodal multi-agent as described in claim 1, characterized in that, The text retrieval agent is ColBERTv2, the image retrieval agent is ColPali, the text agent uses Llama-3.1-8B-Instruct as the basic model, and the general agent, key agent, image agent and comprehensive agent all use Qwen2-VL-7B-Instruct as the basic model.

5. A document understanding and question answering system based on multimodal and multi-agent systems, characterized in that, include: The preprocessing module is used to extract text from each page of the document, generating text paragraphs, and generating page images from each page of the document, based on the input document containing text and images to be understood and the questions expressed in natural language. The multimodal contextual retrieval module is used to perform retrieval in text paragraphs based on the question using a text retrieval tool to obtain a set of text candidates; An image retrieval tool is used to search the page images according to the question to obtain a candidate image set; A multimodal, multi-agent answer generation module is used to generate a preliminary answer using a general agent based on the question, a text candidate set, and an image candidate set; The key agent generates textual key information and textual descriptions of visual key information based on the question, text candidate set, image candidate set, and preliminary answer. The key agent identifies and confirms the key information required to answer the question, serving as orientation clues for the text agent and image agent. The textual key information is used to limit the reasoning scope of the text agent, and the textual descriptions of the visual key information are used to indicate the pages and areas of interest for the image agent. A text-based intelligent agent generates a semantically understood answer based on the question, a set of candidate texts, and key textual information. An image-based intelligent agent generates a visually understood answer based on the question, a set of candidate images, and a textual description of key visual information. The text-based intelligent agent focuses on the reasoning scope defined by the key textual information, comprehensively analyzes the logical connections, causal inferences, and semantic consistency between paragraphs, and generates a structured semantically understood answer. Guided by the textual description of the key visual information, the image-based intelligent agent pays more attention to the pages and areas indicated by the textual description of the key visual information, analyzes the layout structure, chart elements, annotations, and numerical readings, completes focused reasoning, and generates a visually understood answer. A comprehensive intelligent agent is used to generate a final answer based on the question, preliminary answer, semantically understood answer, and visually understood answer. If the semantic understanding answer and the visual understanding answer reach the same conclusion, then the consistent conclusion shall prevail, and the final answer shall be fully expressed by combining the contextual information in the preliminary answer. If the semantic understanding answer and the visual understanding answer are inconsistent, a final answer is generated based on the question, including: if the question asks for semantic content, the semantic understanding answer is used in combination with the contextual information in the preliminary answer to form a complete expression of the final answer; if the question asks for numerical values, readings, or graph relationships, the visual understanding answer is used in combination with the contextual information in the preliminary answer to form a complete expression of the final answer. Among them, the text agent is a large text model, while the general agent, key agent, image agent, and comprehensive agent are all large visual models.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the document understanding and question answering method based on multimodal multi-agent as described in any one of claims 1-4.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the document understanding and question answering method based on multimodal multi-agent as described in any one of claims 1-4.

Images