Document question-answering system and method based on structured semantic understanding

Abstract

The invention provides a document question-answering system and method based on structured semantic understanding. The method comprises the steps that a multi-modal document and a question related to the document are given; constructing structured document mapping through the structured semantic understanding agent; and the inverse reasoning agent searches document contents related to the questions based on the structured document mapping to generate answers. According to the method, structured document mapping is introduced, the document can sense the overall structure of the document, and the context can be introduced more accurately, so that the position of the required reference content can be better positioned in a cross-page and cross-chapter long document question and answer task.

Description

Technical Field

[0001] This invention belongs to the field of large language model applications, specifically a document question answering system and method based on structured semantic understanding. Background Technology

[0002] With the rapid development of computer and information technology, efficient knowledge extraction from massive amounts of electronic documents has become a critical problem that urgently needs to be solved. While generative artificial intelligence technology based on Large Language Models (LLMs) has demonstrated powerful capabilities in the field of document question answering, its inherent knowledge limitations and the "illusion" problem—that is, generating content that does not conform to reality—severely restrict its application in specialized and privatized knowledge scenarios. Retrieval-enhanced generation (RAG) technology, by integrating external knowledge base retrieval with text generation, effectively alleviates these shortcomings, significantly improving the accuracy and reliability of question answering, and is widely used in fields such as intelligent customer service and enterprise knowledge bases.

[0003] To optimize RAG retrieval accuracy, patent application CN202410300934.0 discloses a "RAG knowledge question answering method based on fused vector and keyword retrieval." The core of this scheme lies in generating multiple semantically similar queries through LLM to expand retrieval coverage, then performing vector search and keyword search in parallel. High-dimensional vector similarity calculation and the traditional BM25 algorithm are used to recall relevant document fragments, respectively. Finally, inverse ranking fusion (RRF) technology is employed to synthesize the dual retrieval results to generate the answer. This method attempts to overcome the limitations of a single retrieval mode through technological complementarity, improving information recall in general scenarios.

[0004] However, the aforementioned technologies still have fundamental shortcomings, especially when dealing with complex and information-intensive documents. First, existing RAG systems generally employ a "flattened" slicing strategy, dividing documents into independent text blocks according to fixed lengths or simple rules, crudely destroying the document's inherent deep semantic structure. The hierarchical and causal logical relationships carried by elements such as headings, lists, and tables are severed, resulting in fragmented key information. When a question requires a comprehensive understanding of content across paragraphs or tables, fragmented retrieval struggles to recall the complete context, affecting the accuracy of the answer. Second, the retrieval results exhibit a "flattened" characteristic; the returned list of text blocks lacks original structural information, forcing LLM to expend additional computational resources to reconstruct logical relationships, easily leading to information misunderstanding or loss. For example, data in financial reports needs to be understood in conjunction with table headings and row / column labels, while isolated fragments containing only numbers are highly likely to cause model misjudgment. Furthermore, the lack of cross-document information integration capabilities prevents the system from effectively establishing semantic connections between multiple documents, making it difficult to meet the needs of complex reasoning. These problems stem from the neglect of document structure information, which limits the performance of RAG technology in advanced document question answering tasks, and urgently require innovative solutions centered on document structure reconstruction to break through these limitations. Summary of the Invention

[0005] This invention proposes a document question-answering system based on structured semantic understanding.

[0006] The technical solution for achieving the objective of this invention is: a document question-answering system based on structured semantic understanding, comprising:

[0007] Given a module, a structured semantic understanding agent, and a reflective reasoning agent,

[0008] The given module is used to provide a multimodal document and a document-related question;

[0009] The structured semantic understanding agent is used to construct structured document mappings;

[0010] The reflective reasoning agent is used to find document content related to the question based on structured document mapping and generate answers.

[0011] This invention also proposes a document question answering method based on structured semantic understanding, including:

[0012] Given a multimodal document and a question related to the document;

[0013] Construct structured document mappings by understanding intelligent agents through structured semantics;

[0014] The reflective reasoning agent uses structured document mapping to find document content related to the question and generate the answer.

[0015] Compared with the prior art, the significant advantages of this invention are:

[0016] 1. Preservation of structural information: By using the hierarchical representation of structured document mapping, the hierarchical relationships and semantic dependencies of the document are fully preserved, solving the problem of structural information loss in existing technologies.

[0017] 2. Improved Search Accuracy: The three-path search mechanism integrates structured, textual, and visual information, significantly improving the relevance and accuracy of searches. In particular, it provides additional processing for key elements with high information density, effectively increasing the purposefulness of searches and better adapting to actual question-and-answer scenarios.

[0018] 3. Enhanced long document processing capabilities: By introducing structured document mapping, documents can have an awareness of their overall structure and can more accurately incorporate context. This effectively increases the ability of this invention to better locate the required reference content in long document question-and-answer tasks that span multiple pages and chapters.

[0019] 4. Enhanced Reasoning Ability: Reflective reasoning mechanisms can dynamically evaluate and supplement information, demonstrating higher accuracy in complex reasoning tasks. Furthermore, the structured document representation and the use of a large language model during the retrieval phase leverage the logical reasoning capabilities of the large language model, further enhancing the logical coherence of the referenced content.

[0020] 5. Universality and Extensibility: This invention uses plain text as the final representation of the structured document mapping. This allows the generated document representation to be generated and understood using any model, and the previously generated structured document mapping can be reused after changing the model. In contrast, ordinary dense indexes are bound to feature extraction models, requiring the document index to be regenerated after model changes. Furthermore, this framework is designed to be universal, allowing for the replacement of various other models to adapt to different tasks.

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the network structure of the present invention. Detailed Implementation

[0023] To address the problems of document structure information loss, lack of multimodal information correlation, and insufficient complex reasoning ability in existing technologies, this invention provides a multimodal document understanding method and system based on structured document mapping, aiming to achieve the following technical objectives:

[0024] First, it addresses the issue of document structure information loss caused by the flattening process of existing RAG systems by constructing a hierarchical document structure representation that is similar to human thinking, thus fully preserving the hierarchical relationships and semantic dependencies of documents.

[0025] Second, it overcomes the defect of lost text-image correlation in multimodal documents by explicitly modeling the hierarchy and relational dependencies between document elements, thereby achieving a unified representation of multimodal information such as text, tables, and charts.

[0026] Third, improve the system's performance in complex reasoning scenarios by using structured semantic understanding and reflective reasoning to achieve information integration and reasoning across paragraphs and tables.

[0027] Fourth, it provides a general document structure representation framework that supports structured analysis and question answering for different types of documents, thereby improving the system's versatility and scalability.

[0028] A document question answering system based on structured semantic understanding mainly consists of a given module and two agents. The task of the given module is to answer questions given a multimodal document. And a document-related question The system of this invention can provide answers to the questions. .

[0029] The two agents are structured semantic understanding agents, denoted as... And reflective reasoning agents, denoted as The general process for conducting a question-and-answer session is as follows: Given... and The system first constructs a structured document mapping by understanding the intelligent agent through structured semantics. Then, the reflective reasoning agent is based on Find relevant documents related to the question, and then generate an answer. Formal representation:

[0030] , .

[0031] The structured document mapping (SVM) encodes the hierarchy and cross-modal relationships between document elements, ensuring that the reasoning process remains consistent with the document's internal structure. The architecture of these modules is described in detail below.

[0032] Structured semantic understanding agents:

[0033] Most raw multimodal documents lack predefined data models or fixed formats; their content is organized naturally and freely, and their logical structures are largely identified by humans rather than machines—in other words, they are unstructured documents. The task of a structured semantic understanding agent is to hierarchically parse the raw multimodal documents, derive a logical structure, and fill this structure with multimodal content and element associations, providing structured knowledge for downstream reasoning and question answering.

[0034] The structured semantic understanding agent mainly consists of three modules: raw data parsing module, document understanding module, and data persistence module.

[0035] The raw data parsing module processes the raw documents, unifying the inconsistent formats of the raw data into a format that facilitates subsequent processing.

[0036] The document understanding module parses and understands the document structure after processing, and constructs a structured representation of the document.

[0037] The data persistence module is used to store the structured representation of the document and the original document content after it has been formatted in a unified way, and to create an index for downstream tasks to retrieve and use.

[0038] In a further embodiment, the structured document representation is the core of the invention. It is generated by a structured semantic understanding agent and subsequently provides a semantic and logical index for the retrieval work of a reflective reasoning agent.

[0039] Structured document mapping uses a hierarchical structure to summarize the logical structure of the document level by level:

[0040] Chapter level: Summarize the document from a semantic and logical perspective, dividing the document into a tree-like chapter structure.

[0041] Page layer: As the physical unit of document organization, the page plays a crucial role in human understanding and logical retrieval. Each page corresponds to a chapter in the chapter layer.

[0042] Element layer: For each page, extract non-plain text modal information such as charts, tables, and images, and attach them to the bottom of each page.

[0043] In a further embodiment, the reflective reasoning agent is mainly divided into two parts: a retrieval unit and a generator.

[0044] The retrieval unit is responsible for retrieving relevant reference content from the index built by the structured document understanding agent. It also retrieves the original document content from the index and provides it to the generator. The retrieval process consists of three paths: structured semantic retrieval, visually dense retrieval, and text-dense retrieval. Structured semantic retrieval primarily relies on the logical information of the structured text mapping for understanding and retrieval.

[0045] Generator: Based on the provided references and the question, it generates an answer to the problem. It also incorporates a reflection mechanism: after generation, it verifies the quality of the generated answer; if it fails to meet the standards, it regenerates; if it meets the standards, it outputs the answer.

[0046] A document question answering method based on structured semantic understanding includes:

[0047] Data processing: The input file is parsed, and each page is saved in its original format as both plain text and plain image, while element extraction is performed. For documents whose text content can be directly parsed, plain text indicates that extraction can be done directly; otherwise, OCR technology is used for recognition.

[0048] Document understanding and the establishment of structured document mapping: Document understanding is carried out in the form of page-by-page incremental understanding, which mainly consists of two parts: the generation of document structure, and the filling and indexing of structure content.

[0049] When generating the document structure, a large language model is used to summarize the document page by page, as shown in the following formula.

[0050]

[0051]

[0052] The initial document summary is empty. For each page, the document summary agent first uses a large language model to try to find existing chapter titles in the document as a reference for chapter division. If none are found, it creates its own summary. Simultaneously, when summarizing each page, it needs to input not only the content of each page but also the summaries of the previous page and the currently formed chapters, allowing the document summary agent to generate a suitable document summary. Furthermore, the output of each page includes three parts: the updated document chapter division based on the newly input page, a summary of the page content, and a description of the key elements appearing on each page.

[0053] Once all pages have been processed, we obtain a final structured document mapping of a document through some simple format conversions and index associations, which we save for later use.

[0054] Dense Index Building: In addition to using structured document mapping, this invention also introduces dense retrieval methods, which are commonly used in current document question-answering systems, as a supplement. Dense retrieval refers to representing the textual or visual features of a document using high-dimensional vectors and building an index for retrieval. This invention uses both visual and textual indexes, describing the text and visual features of the document page using vectors, and using these vectors to build the index.

[0055] Three-way search:

[0056] Once the structured document mapping is established, it's necessary to address user questions, given a query. The reflective reasoning agent retrieves relevant document elements from the structured document map using three complementary retrieval paths:

[0057] Structured semantic retrieval: Utilizing hierarchical chapter, page, and element summaries from a structured document mapping, it locates semantically relevant elements. The intelligent agent navigates the summary tree, interprets the textual descriptions of chapters and pages, identifies elements whose content aligns with the query intent, and obtains the search results. ,Right now:

[0058]

[0059] Structured semantic retrieval directly uses a large language model to receive structured document mappings as input, allowing it to perform searches at three granularities: chapter, page, and in-page elements, based on the structured summary. This greatly enhances the retrieval capabilities of document question answering systems.

[0060] Text feature retrieval and visual feature retrieval: The text feature retrieval engine will... The code is encoded as a text representation, and its similarity to the text embeddings of elements in the structured document mapping is calculated. The top-k most similar page content is then retrieved to obtain the search results. The visual feature retrieval system projects the query onto the visual feature space, retrieves visually relevant pages, and obtains the search results. .Right now:

[0061]

[0062] Final search results It is a result that aggregates three paths:

[0063] Reflective answer generation: Given aggregated search results The reflective generation module generates the final answer through iterative reasoning and structured reflection. Each retrieved element (including pages and key elements) is processed to extract its multimodal features: Then a generative model is used. Integrating textual and visual features to generate candidate answers: Generative models need to consider the emphasis of the problem content and choose whether to focus on textual or visual references.

[0064] To ensure semantic integrity and structural consistency, this invention also incorporates an evaluator based on a large language model to reflectively evaluate the generated answers. If the answer is incomplete, the agent traverses the structured document mapping hierarchy to retrieve additional relevant elements. The features are then fed back to the generator for iterative optimization. And so, we get the final answer.

[0065] Example

[0066] To enable those skilled in the art to implement this invention, a detailed implementation of a multimodal document understanding system based on structured document mapping is described below. This embodiment uses an automated analysis scenario of enterprise financial reports as an example to fully illustrate the entire process from system deployment to question-answer generation. Performance evaluation results of this invention on publicly available datasets are also provided, demonstrating the effectiveness of the invention.

[0067] Implementation Environment Configuration

[0068] The system described in this embodiment is deployed and runs in the following hardware and software environment:

[0069] Hardware environment: Two server nodes equipped with NVIDIA RTX 3090 GPUs (24GB VRAM) are used, configured with Intel Xeon Gold 6330 processors (28 cores, 2.0GHz base frequency), 256GB DDR4 memory, and a high-speed NVMe solid-state storage array for persistent storage of the document library and vector database. A single node can process four document analysis tasks in parallel, and the dual-node cluster supports eight concurrent requests.

[0070] Software environment: Operating system is Ubuntu 22.04 LTS, Python version 3.10.12. Core dependencies include: PyTorch 2.1.0, transformers 4.35.0, ColBERTv2 retrieval engine, ColPali visual retrieval engine, as well as the PDF parsing library pymupdf 1.23.5 and the chart extraction tool pdffigure2. The large language model uses GPT-4o (API version 2024-05-13) as the agent inference engine for both structured semantic understanding and reflective reasoning agents. The vector database uses FAISS 1.7.4 for efficient similarity retrieval.

[0071] Embodiments of the Document Question-and-Answer System of this Invention

[0072] Structured semantic understanding agents:

[0073] Step 1: Multimodal document preprocessing

[0074] 1. Upload the PDF financial report to be analyzed (e.g., "XX Company 2024 Annual Financial Report", 156 pages in total) to the system input terminal. The structured semantic understanding agent first decomposes the document into a set of pages using the pymupdf library, rendering each page as a PNG image at a resolution of 300 DPI, while simultaneously extracting the text stream embedded in the PDF.

[0075] 2. Perform element detection on each page: Use the pdffigure2 tool to identify and extract visual elements such as figures, data tables, and trend charts, and generate independent image files.

[0076] Step 2: Element-level multimodal feature encoding

[0077] 1. Text feature generation: For each text element (including paragraph text, table title, and chart annotation), a 128-dimensional dense vector representation is generated using the text encoder of the ColBERTv2 model.

[0078] 2. Visual Feature Generation: For each visual element (chart, table image), a 128-dimensional visual vector is generated using the visual encoder of the ColPali model. The entire page of image representation generates bimodal features of text and vision.

[0079] 3. Metadata annotation: Attach metadata such as page number and chapter affiliation to each element and store it in the node attributes of the structured document mapping.

[0080] Step 3: Hierarchical Structure Construction and Data Population

[0081] 1. Chapter-level Generation: Using GPT-4o as the structure parser, the system automatically identifies the heading levels (e.g., "1. Operating Results" is the main chapter, and "1.1 Revenue Analysis" is a sub-chapter) by inputting the text content and visual layout information of each page. The system incrementally constructs the chapter structure according to the following formula:

[0082] The function is implemented through a prompt project (example prompt: "Based on the title changes between the current page and the previous page, determine the chapter boundaries and output the chapter index and title"), ultimately forming a tree-like chapter hierarchy with the root node being "XX Company 2024 Annual Financial Report".

[0083] 2. Page-level aggregation: Mount all elements under each page to the corresponding page node. Page node attributes include page number, chapter reference, and element list.

[0084] 3. Element-level representation: The leaf nodes store the text embedding, visual embedding, and pointer to the original content (pointing to the image file or CSV data path) of each element, thus completing the construction of the structured document mapping.

[0085] Reflective reasoning agent:

[0086] Step 1: Three-path retrieval execution. Taking the user query "What are the main reasons for the decline in gross profit margin in East China in the fourth quarter of 2024 compared to the third quarter?" as an example:

[0087] 1. Structured Semantic Retrieval: The reflective reasoning agent traverses the chapter summary tree of the structured document mapping, locates the chapter "3. Regional Operation Analysis" → "3.2 East China", extracts all pages (pages 82-89) under this chapter and table elements containing the keyword "gross profit margin", and obtains the retrieval results (text field on page 82, table 8 on page 85, and trend chart on page 87, respectively).

[0088] 2. Text Feature Retrieval: The query is encoded as a vector, and the cosine similarity is calculated with all text element vectors in the structured document mapping. The Top-4 strategy is used to recall the most similar elements. The result is recorded as (including the CSV data vector in Table 8).

[0089] 3. Visual Feature Retrieval: Project the query onto the visual space and retrieve trend charts that are visually related to "decline in gross profit margin", resulting in the bar chart on page 87.

[0090] 4. Result aggregation: The union of the results retrieved a total of 5 results, including 2 texts, 1 table and its CSV data, and 1 trend chart.

[0091] Step Two: Reflective Generation Mechanism

[0092] 1. Initial Answer Generation: Input the retrieved element features into the generator (i.e., GPT-4o). An example prompt is: "Answer the question based on the following structured information [list of elements]. Note the connection between the table data and the trend chart." The generated candidate answer is: "The decline in gross profit margin in East China in Q4 2024 was mainly due to a 3.2% increase in raw material prices and a decrease in capacity utilization to 78%."

[0093] 2. Assessment and Feedback: Use the Large Language Model evaluator to determine the completeness of the answer. An example evaluator prompt is: "Check if the answer explains the comparison 'compared to Q3' and if it cites specific data." The assessment result is 'No' because the Q3 data comparison is missing.

[0094] 3. Iterative retrieval: Based on the evaluation feedback, the reflective reasoning agent retrieves relevant elements for "East China Q3 gross profit margin" in the structured document mapping, actively locates the Q3 data column in Table 8 (page 85), and supplements the elements.

[0095] 4. Secondary generation: Input the new element and the original element into the generator to generate the final answer: "The gross profit margin in East China in Q4 2024 was 22.5%, a decrease of 3.6 percentage points from 26.1% in Q3. The main reasons include: (1) the cost of raw materials increased by 3.2% month-on-month, resulting in an increase in unit cost; (2) the capacity utilization rate in Q4 dropped to 78%, lower than 85% in Q3, resulting in an increase in fixed cost amortization."

[0096] 5. Termination Judgment: The evaluator confirms that the answer contains comparative data, causal analysis, and multimodal evidence citations, outputs 'Yes', and terminates the iteration.

[0097] Experimental verification:

[0098] Experimental results on multiple benchmark datasets such as MMLongBench and LongDocURL show that the method of this invention is significantly better than the traditional RAG method in terms of document structure preservation, retrieval accuracy, and inference consistency, thus verifying the effectiveness and advancement of this invention.

[0099] Table 1 shows the comparison results between our method and various baseline methods. As the table shows, our method achieves state-of-the-art performance on all benchmark datasets, demonstrating its powerful capabilities in multimodal document question answering tasks. Specifically, the configuration using Top-4 retrieval significantly outperforms the strongest baseline method (MDocAgent), improving average accuracy by 12.4%. The most significant improvement is reflected in the number of MMLongBench benchmarks.

[0100] The dataset is characterized by a large number of complex questions, such as those requiring cross-page reasoning and document element understanding. Thanks to its chapter-level hierarchical structure and structured retrieval through structured document mapping, the method of this invention excels at integrating and effectively organizing information from multiple pages, thus overcoming the key limitations of traditional RAG methods in handling cross-page document question answering. Furthermore, the fine-grained analysis and localization of key elements in this invention improves performance for complex reasoning problems involving graphs. This capability is further demonstrated by the significant improvements achieved on PaperTab and FetaTab, datasets containing large amounts of tabular and graphical data, whose question design is closely related to multimodal understanding.

[0101]

[0102] In contrast, LongDocURL and PaperText consist primarily of text content with relatively few charts, which makes it difficult for the method of this invention to fully leverage its multimodal capabilities. Nevertheless, the method of this invention still achieves competitive improvements on these datasets, demonstrating the advantages of chapter-level hierarchies even in primarily text-based scenarios.

[0103] Regarding the number of candidate contexts, Top-4 retrieval consistently delivers a greater performance improvement than Top-1 retrieval, indicating that the amount of supporting evidence plays a crucial role in multimodal document question answering performance. This effect is particularly pronounced in datasets requiring multi-page inference, such as MMLongBench. However, even with Top-1 retrieval, the method of this invention occasionally surpasses the MDocAgent (Top-4) baseline, highlighting the effectiveness of the structured retrieval approach—it more precisely defines the context scope, enabling the method to achieve superior results with fewer context fragments.

[0104] Overall, these findings validate the effectiveness of the present invention in enhancing RAG for multimodal document understanding, particularly in scenarios involving high reasoning complexity, multi-page dependencies, and multimodal contexts.

Claims

1. A document question-answering system based on structured semantic understanding, characterized in that, include: Given a module, a structured semantic understanding agent, and a reflective reasoning agent, The given module is used to provide a multimodal document and a document-related question; The structured semantic understanding agent is used to construct structured document mappings; The reflective reasoning agent is used to find document content related to the question based on structured document mapping and generate answers.

2. The document question-answering system based on structured semantic understanding according to claim 1, characterized in that, The structured semantic understanding agent includes a raw data parsing module, a document understanding module, and a data persistence module. The raw data parsing module is used to process the raw documents and unify the original data with inconsistent formats. The document understanding module is used to parse and understand the document structure after the format has been standardized, and to construct a structured representation of the document. The data persistence module is used to store the structured representation of documents and the formatted document content, and to create an index.

3. The document question-answering system based on structured semantic understanding according to claim 1, characterized in that, Structured document mapping uses a hierarchical structure, summarizing the logical structure of the document level by level, including: Chapter level: Summarize the document from a semantic and logical perspective, dividing the document into a tree-like chapter structure; Page layer: A page serves as the physical unit of document organization, with each page corresponding to a chapter in the chapter layer; Element layer: For each page, extract the information of non-plain text modalities and attach it to the bottom of each page.

4. The document question-answering system based on structured semantic understanding according to claim 1, characterized in that, The reflective reasoning agent includes a retrieval unit and a generator. The retrieval unit is used to retrieve reference content related to the question from the index established by the structured document understanding agent, and at the same time, it obtains the original document content through the index and provides it to the generator. The generator is used to provide answers to questions based on reference content and the question itself. It also incorporates a reflection mechanism to verify the quality of the generated answer after generation. If the quality is not up to standard, the generator is regenerated; if it is up to standard, the answer is output.

5. A document question-answering method based on structured semantic understanding, characterized in that, include: Given a multimodal document and a question related to the document; Construct structured document mappings by understanding intelligent agents through structured semantics; The reflective reasoning agent uses structured document mapping to find document content related to the question and generate the answer.

6. The document question answering method based on structured semantic understanding according to claim 5, characterized in that, The specific process of constructing a structured document mapping through structured semantic understanding of an intelligent agent is as follows: Data processing: For the input file, parse it, and for each page, save it in its original format as plain text and plain image, while extracting elements. Document understanding and the establishment of structured document mapping: Document understanding is carried out in the form of page-by-page incremental understanding, including: document structure generation, and filling and indexing of structural content. After all pages have been processed, a final structured document mapping of a document is obtained through some format conversion and index association work. The textual or visual features of a document are represented using high-dimensional vectors, and an index is created for retrieval.

7. The document question answering method based on structured semantic understanding according to claim 6, characterized in that, During data processing, the initial document summary is empty. For each page, the document summary agent first uses a large language model to try to find the existing chapter titles in the document as a reference for chapter division. If none are found, the agent will summarize them manually. At the same time, when summarizing each page, the content of each page, the previous page, and the currently formed chapter summary are passed in so that the document summary agent can generate a suitable document summary. In addition, the output of each page also includes three parts: the updated document chapter division obtained based on the newly passed page, the page content summary for each page, and a description of the key elements appearing on each page.

8. The document question answering method based on structured semantic understanding according to claim 5, characterized in that, The reflective reasoning agent retrieves relevant document elements from the structured document map using three complementary retrieval paths: Structured semantic retrieval: Utilizing hierarchical chapters, pages, and element summaries in a structured document mapping, semantically relevant elements are located; the agent navigates the summary tree, interprets the textual descriptions of chapters and pages, identifies elements whose content aligns with the query intent, and obtains the retrieval results; Text feature retrieval: The text feature retrieval tool will... The code is encoded as a text representation, and similarity is calculated between it and the text embedding of elements in the structured document mapping. The top-k most similar page content is retrieved to obtain the search results. Visual feature retrieval: The visual feature retrieval system projects the query onto the visual feature space, retrieves visually relevant pages, and obtains the search results; The search results from three complementary search paths are aggregated to obtain aggregated search results, and the final answer is generated through iterative reasoning and structured reflection.

9. The document question answering method based on structured semantic understanding according to claim 5, characterized in that, The specific method for generating the final answer through iterative reasoning and structured reflection is as follows: Multimodal features of each retrieved element are extracted separately, and a generative model is used to integrate textual and visual features to generate candidate answers; The evaluator based on the large language model reflects on and evaluates the generated answer. If the answer is incomplete, the agent traverses the structured document mapping hierarchy to retrieve additional relevant elements and feeds their features back to the generator for iterative optimization to obtain the final answer.

Images