Document question and answer method and device, storage medium and program product

Abstract

The invention discloses a document question and answer method and device, a storage medium and a program product, and the method comprises the steps: carrying out the content retrieval of a first document database based on an inquiry statement, so as to determine a corresponding intermediate retrieval result; performing content retrieval on a second document database based on the inquiry statement and the intermediate retrieval result to determine a retrieval matching result, the first document database and the second document database being databases constructed based on at least one document; detecting whether the retrieval matching result meets a reply condition of the inquiry statement or not; and when the retrieval matching result meets a reply condition, constructing a reply answer according to the retrieval matching result. Therefore, the question answering system can adapt to questions with different semantic complexities and logic levels through a document retrieval mechanism mixed with the neural symbols, and high-quality answers can still be efficiently produced when complex questions are processed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a document question-answering method, device, storage medium, and program product. Background Technology

[0002] Document question answering technology, as an important direction in the field of Natural Language Processing (NLP), aims to extract answers or generate semantically coherent responses from structured or unstructured documents based on user-inputted questions. With the rapid development of deep learning technology, document question answering technology has been widely applied in various fields such as legal analysis, medical document parsing, educational tutoring, and enterprise knowledge management.

[0003] Traditional document question answering systems mostly use information retrieval and matching technologies. By calculating the similarity between keywords in the document and user questions, they directly return text fragments as answers. However, they are limited to shallow semantic matching, lack a deep understanding of the semantics of questions, and are unable to cope with semantic diversity and complex reasoning needs.

[0004] Currently, the industry has not proposed a better solution to the above problems. Summary of the Invention

[0005] This application provides a document question-answering method, device, storage medium, and program product to at least solve the problem of low accuracy in answering complex questions caused by traditional keyword matching.

[0006] In a first aspect, embodiments of this application provide a document question-answering method, comprising: performing content retrieval on a first document database based on a query statement to determine corresponding intermediate retrieval results; performing content retrieval on a second document database based on the query statement and the intermediate retrieval results to determine retrieval matching results, wherein the first document database and the second document database are databases constructed based on at least one document; wherein the first document database is a vector database and the second document database is a relational database, or the first document database is a relational database and the second document database is a vector database; detecting whether the retrieval matching results satisfy the response conditions of the query statement; and constructing a response answer based on the retrieval matching results when the retrieval matching results satisfy the response conditions.

[0007] Secondly, embodiments of this application provide an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the document question-and-answer method of any embodiment of this application.

[0008] Thirdly, embodiments of this application provide a storage medium storing a computer program thereon, characterized in that, when the program is executed by a processor, it implements the steps of the document question-and-answer method of any embodiment of this application.

[0009] Fourthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the document question-and-answer method of any embodiment of this application.

[0010] The beneficial effects of the embodiments of this application are as follows: A joint retrieval mechanism using a first and second document database is employed. In the vector database, vectorized representation supports efficient semantic matching, capturing the deep semantic relationships between queries and documents. In the relational database, structured information enhances retrieval accuracy, ensuring rapid location of target information for questions with clear logical structures or attribute constraints. Thus, this hybrid neural symbolic document retrieval mechanism enables the question-answering system to adapt to questions of varying semantic complexity and logical levels, allowing it to efficiently produce high-quality answers even when handling complex problems. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 The flowcharts of two retrieval strategies in current related technologies are shown; Figure 2 A flowchart illustrating an example of a document question-and-answer method according to an embodiment of this application is shown; Figure 3 A flowchart illustrating another example of a document question-and-answer method according to an embodiment of this application is shown; Figure 4 A flowchart illustrating an example of constructing or updating a relational database according to an embodiment of this application is shown. Figure 5A flowchart illustrating an example of constructing or updating a vector database according to an embodiment of this application is shown; Figure 6 This document illustrates an example of cross-library association storage of non-text elements according to an embodiment of this application; Figure 7 A flowchart illustrating an example of a document question-and-answer method according to an embodiment of this application is shown; Figure 8 A schematic diagram illustrating the effects of an example of iterative vector retrieval and iterative symbol retrieval is shown. Figure 9 A schematic diagram illustrating the effect of an example of a hybrid neural symbol retrieval enhancement system provided according to an embodiment of this application is shown. Figure 10 This illustration shows an example of how a coded unit value in a database is converted into a data entry in a vector storage according to an embodiment of this application. Figure 11 A simulation illustration shows an example of the consistency between human assessment and LLM-based assessment. Figure 12 A simulation diagram illustrating an example of the frequency distribution of the problem length is shown. Figure 13 A schematic diagram of an example database architecture for storing parsed elements of each PDF file is shown. Figure 14 This diagram illustrates the effect of an example of all coded columns in a database. Figure 15 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] It should be noted that with the exponential growth in the number of academic papers, question answering (QA) systems based on Large Language Models (LLMs) have shown great potential in helping researchers extract key details from emerging research. However, single PDF documents often exceed the prompt length limit, and user queries may involve multiple documents. To address these challenges, retrieval augmented generation (RAG) has demonstrated high effectiveness in knowledge-intensive question answering.

[0015] Figure 1 The diagram illustrates two retrieval strategies in the current related technologies, specifically showing the paradigms of neural retrieval and symbolic retrieval strategies, and also reveals the motivation behind the system framework of the hybrid neural symbolic retrieval (NeuSym-RAG) provided in the embodiments of this application. Figure 1 The upper part of the diagram illustrates two paradigms of retrieval strategies, while the lower part shows that PDF documents can be segmented based on different granularities and contain many auxiliary text elements.

[0016] Despite the widespread application of RAG, classic neural retrieval often fails when handling precise queries involving mathematical operations, comparisons, or aggregations. For example, in Figure 1 In the upper left section, because the tables are distributed in different locations within the document, the total number of tables cannot be determined solely by retrieving fragments. On the other hand, symbolic retrieval (such as TAGs) relies on semantic parsing techniques (e.g., text-to-SQL conversion) to directly extract target information from structured databases. However, such precise queries often fail in semantically fuzzy matching or morphological transformations (such as graph-based RAGs and GraphRAGs). Previous literature has largely conducted research on these two paradigms in isolation.

[0017] Furthermore, the most widely adopted document segmentation methods are based on fixed-length continuous tokens and may consider sentence boundaries or finer granularity. However, for studying semi-structured PDF documents such as academic papers, this common practice ignores the inherent structure of chapters and the salient features of tables and figures (such as... Figure 1 (As shown at the bottom). The unique layout of PDF files provides a more structured perspective, which helps in the segmentation and organization of content.

[0018] Therefore, current related technologies discuss neural retrieval and symbolic retrieval separately, neglecting their complementary advantages. Furthermore, traditional single-view segmentation methods fail to fully utilize the rich structural information in PDFs, such as chapters, tables, and figures.

[0019] More specifically, with the continuous development of document question answering technology, the current related technologies mainly include vector-based retrieval enhancement generation schemes, knowledge graph-based retrieval enhancement generation schemes (GraphRAG), structured knowledge-based retrieval enhancement generation schemes (StructRAG), and hybrid vector and knowledge graph-based retrieval enhancement generation schemes (HybridRAG).

[0020] In the vector-based retrieval enhancement generation scheme, the document is pre-segmented into segments according to the number of consecutive tokens, then encoded using a text encoding model, and finally the most similar segment is extracted as context based on the query vector encoded by the user's question to assist in question answer generation.

[0021] In the knowledge graph-based retrieval enhancement generation scheme, a large model is used to generate a general description for each community and the entity relationships under each community in the graph. After receiving a user's question, relevant fragments are extracted within each community to generate local responses. Finally, all answers are summarized to obtain the final result.

[0022] In the structured knowledge-based retrieval enhancement generation scheme, the large model first uses the relevant content extracted from the current question and vector similarity to determine the most suitable organizational structure, such as graph, table, algorithm, directory, etc. Then, the core context is restructured or transformed. Next, the answer to the original question is generated on the structured corpus. In the hybrid approach, vector retrieval and knowledge graph-based retrieval generation are fused together and used as context to be passed to a large model to predict the answer.

[0023] It should be noted that the above description of the relevant technologies is intended to make it easier for the public to understand the spirit and objectives of the present invention, and is not intended to describe the prior art. That is, the relevant technologies are not prior art, but may be undisclosed or in the experimental stage.

[0024] Furthermore, based on the aforementioned relevant technologies, it has at least the following drawbacks: a) It only used neural vector-based retrieval and did not utilize precise querying based on symbolic systems, such as the structured query language SQL. Even with the knowledge graph and structured retrieval schemes, the first still relies entirely on vector retrieval, only using knowledge graphs to restructure the text organization. The second only uses adaptive structural transformation on the extracted core context, rather than applying symbolic language-based queries to the global corpus.

[0025] b) In document segmentation, most still follow segmentation schemes based on continuous terms, failing to consider multiple perspectives. This is especially true for semi-structured PDF documents, which may contain many non-text elements such as figures, tables, and formulas, making the document segmentation methods too simplistic. Note that the two schemes based on knowledge graphs and structured retrieval differ in two ways: one involves detecting entities and relationships during the preprocessing stage to avoid segmentation (or it only considers the second document segmentation scheme), and the other involves structural reorganization of the segmented document fragments. Strictly speaking, neither of these falls within the scope of segmentation perspectives.

[0026] Therefore, the above techniques all rely excessively on vector encoding of text and traditional document segmentation methods, without taking into account the layout and structural characteristics of semi-structured PDF documents.

[0027] In view of this, Figure 2 A flowchart illustrating an example of a document question-and-answer method according to an embodiment of this application is shown.

[0028] like Figure 2 As shown, in step S210, content retrieval is performed on the first document database based on the query statement to determine the corresponding intermediate retrieval results.

[0029] In step S220, content retrieval is performed on the second document database based on the query statement and intermediate retrieval results to determine the retrieval matching results. The first document database and the second document database are databases constructed based on at least one document.

[0030] Here, the first document database is a vector database and the second document database is a relational database, or the first document database is a relational database and the second document database is a vector database.

[0031] In one example of this application's embodiments, if the first document database is a vector database, the query statement can be vectorized, and its similarity can be calculated with the content vectors in the vector database to retrieve the most similar document blocks as intermediate search results. In this case, if the second document database is a relational database, more targeted SQL query conditions can be constructed based on the key fields extracted from the intermediate search results, further narrowing the query scope. For example, the intermediate search results can be used as context, and their content can be jointly encoded with the query statement to generate new query vectors or conditions.

[0032] In another example of this application's embodiments, if the first document database is a relational database, then keyword matching or SQL queries are used to retrieve structured fields in the database, selecting records related to the query statement as intermediate retrieval results. In this case, the second document database is a vector database; the semantic vectors in the intermediate retrieval results are further optimized, and semantic similarity is recalculated by combining the query statement vector to retrieve the target document block.

[0033] In step S230, it is detected whether the search matching results meet the response conditions of the query statement.

[0034] It should be understood that the definition of response criteria can be diverse, such as confidence thresholds for answers, logical consistency, or semantic coverage. For example, natural language understanding techniques (such as text similarity calculation and logical rule validation) can be used to evaluate the relevance between search results and the query. Furthermore, if numerical information (such as data in a table) is involved, the matching results are validated to ensure they meet numerical ranges or constraints.

[0035] In some implementations, external tools can be introduced to evaluate the search matching results, such as introducing a large language model to evaluate whether the search matching results meet the response conditions of the query statement.

[0036] In step S240, when the search matching results meet the response conditions, a response answer is constructed based on the search matching results.

[0037] It should be understood that responses are constructed in a variety of ways based on the search matching results, such as semantic slot filling or generating semantically coherent answers based on context using language models, to ensure that the generated responses conform to natural language expression habits and guarantee user experience.

[0038] Figure 3 A flowchart illustrating another example of a document question-and-answer method according to an embodiment of this application is shown.

[0039] like Figure 3 As shown, in step S310, content retrieval is performed on the first document database based on the query statement to determine the corresponding intermediate retrieval results.

[0040] In step S320, content retrieval is performed on the second document database based on the query statement and intermediate retrieval results to determine the retrieval matching results.

[0041] In step S330, it is detected whether the search matching results meet the response conditions of the query statement.

[0042] In step S341, when the search matching result meets the response conditions, a response answer is constructed based on the search matching result.

[0043] In step S343, when the search matching result does not meet the response conditions, the search strategy action corresponding to the search matching result is determined to iteratively search the first document database or the second document database for content until the search matching result re-determined by the iterative search meets the response conditions.

[0044] In some implementations, when the initial matching results are insufficient, the semantic scope of the query is expanded, for example, by using synonym replacement or contextual supplementation. Switching between a first and second document database, utilizing an alternative retrieval method (such as switching from vector retrieval to relational retrieval) to compensate for the shortcomings of existing methods. Alternatively, the retrieval granularity can be changed through retrieval strategy actions, such as switching from paragraph retrieval to chapter or page retrieval. Thus, a self-optimization mechanism is provided to address the problem of insufficient initial retrieval, resolving complex and fuzzy issues by dynamically adjusting the retrieval strategy, thereby improving the adaptability of the document question-answering system.

[0045] In some examples of embodiments of this application, the application subject of the method of this application embodiment is a large language model, and multiple prompt templates are integrated into the question-answering system to realize the triggering and invocation of different types of operations by the large language model.

[0046] Specifically, based on the first search prompt template, a content retrieval for the first document database is triggered. For example, the template content could be "Based on the following query, retrieve the most relevant document content in the vector database: Query: [User Question]", explicitly instructing the LLM to call the first document database (such as a vector database) to perform a preliminary search. Based on the second search prompt template, a content retrieval for the second document database is triggered. For example, the template content could be "Based on the following query and intermediate search results, query the most relevant information in the relational database: Query: [User Question], Intermediate Search Results: [Summary of Content Returned from Preliminary Search]", this template, combining the intermediate search results and the original query, guides the LLM to perform a precise query for structured data (such as a relational database). Here, by calling the first and second search prompt templates, content retrieval for different document databases is triggered respectively.

[0047] On the other hand, based on the inspection prompt template, the detection response conditions are triggered. For example, the template content could be "Based on the following search results, determine whether the response conditions of the user's query statement are met: Query statement: [User question], Search matching results: [Intermediate or final search results], Condition rules: [Preset logical conditions or semantic requirements]". By clearly listing the criteria for meeting the conditions (such as semantic coverage and logical consistency), it is convenient for LLM to perform condition detection. Based on the action prompt template, the action to determine the search strategy is triggered. For example, the template content could be "The following are the reasons why the current search matching results do not meet the response conditions. Determine the new search strategy action: Reason analysis: [Specific explanation of the unmet conditions], Original query statement: [User question], Current strategy: [The search strategy already used, such as vector matching or SQL query], Prompt requirements: Adjust the search granularity or switch the search database, and generate new strategy suggestions". By clearly guiding LLM to analyze the current strategy and generate optimized search actions, it is possible to trigger the action to determine the new search strategy. Based on the response prompt template, a response answer is constructed based on the search matching results. For example, the template content could be: "The following are the search matching results. Please construct a response answer based on these results and the user's query: Query: [User question], Search matching results: [Document fragments or structured data that meet the criteria], Construction requirements: The response must cover the core content of the query; If the matching results contain multiple related information, integrate them into a semantically coherent natural language expression." Through the semantic understanding and generation capabilities of LLM, the final answer is ensured to be semantically complete, logically clear, and highly relevant to the query.

[0048] Thus, by fully utilizing the natural language understanding capabilities of the large language model, diverse and dynamic task triggering and invocation were achieved.

[0049] It should be noted that the content format of each prompt template can be diverse, and the above description is for illustrative purposes only and should not be considered as a limitation of the present invention. Furthermore, the various prompt templates can also be integrated into a single unified template, thereby defining various operations in the document question-and-answer process using this unified template.

[0050] For example, the prompt template used to trigger NeuSym-RAG is as follows: "You are an intelligent agent skilled at retrieving useful context from the DuckDB database and Milvus vector store through SQL execution and similarity searches, and answering user questions. You are given a natural language question related to a PDF file, along with the schemas of the database and vector store. Your ultimate goal is to answer the input question according to a predefined answer format. The DuckDB database contains the parsed content of the original PDF file, while the Milvus vector store encodes the cell contents of specific columns in the database as vectors. You can generate executable actions, interact with the hybrid environment (including the database and vector store) across multiple rounds, and retrieve the necessary context until you are confident enough in your answer to the question."

[0051] ## Task Description Each input task contains the following parts: 1. [Question]: Natural language questions from users about PDF files, such as "Is there any...?" 2. [Answer Format]: Specifies the required format for the final answer, such as the answer should be "Yes" or "No" without punctuation.

[0052] 3. [Database Schema]: The detailed serialization schema of the DuckDB database, used for reference when generating SQL queries. It includes: - surface - Columns and their data types - Description of these pattern entries - Primary key and foreign key constraints 4. [Vectorstore Schema]: The detailed serialization schema of the Milvus vector store, used as a reference when generating executable retrieval operations with specific parameters. It includes: - Vector set - Fields - These vectors originate from relational databases containing (table, column) pairs. - The syntax of valid filtering rules. In some examples of embodiments of this application, the relational database further includes an element description summary determined based on non-text elements in the document, and / or the vector database further includes an element encoding vector determined based on non-text elements in the document.

[0053] Here, non-text elements in the document include, but are not limited to, images, charts, tables, formulas, and flowcharts.

[0054] In some implementations, semantic analysis or structured extraction (e.g., chart titles, table descriptions, formula context) is performed on non-text elements to generate structured summary information describing these elements, which is then stored in a relational database. Furthermore, non-text elements are vectorized using deep learning models (such as ResNet, CLIP, or BERT) to generate multimodal encoded vectors. This supports multimodal semantic expression and logical description of non-text elements in documents, leveraging the structured characteristics of relational databases and the semantic matching capabilities of vector databases to improve the retrieval efficiency of non-text elements and enhance the knowledge representation capabilities of document retrieval and question-answering systems.

[0055] Figure 4 A flowchart illustrating an example of constructing or updating a relational database according to an embodiment of this application is shown.

[0056] like Figure 4 As shown, in step S410, the metadata, text fragments and non-text elements corresponding to the target document are parsed.

[0057] It should be understood that the target document can be of various types, such as PDF, Word document, HTML document or other formats.

[0058] Meta-information can include basic document attributes such as title, author, keywords, publication date, version number, and document category, and can be extracted using various document format parsing tools. Text fragments can refer to information at different granularities within a document, such as chapters, paragraphs, and sentences. Non-text elements can be extracted from the document using image recognition technology, OCR technology, or LaTeX parsers to obtain charts, tables, or formulas.

[0059] In step S420, a multimodal language model is invoked to generate element description summaries corresponding to non-text elements.

[0060] Here, the types of multimodal language models can be diverse, such as CLIP (Contrastive Language–Image Pretraining), which is suitable for generating image or graph descriptions, or MathGPT, which specializes in handling mathematical symbols and formulas to generate descriptions with physical meaning and mathematical background.

[0061] In step S430, the metadata, text fragments, and element description summaries are stored in a structured manner to build or update the relational database.

[0062] Here, various database management systems (such as MySQL and PostgreSQL) can be used to define and manage relational database structures. Furthermore, data storage formats support JSON or SQL table structures, facilitating subsequent retrieval and manipulation. Thus, through structured storage, unified management of metadata, text fragments, and non-text elements is achieved, simplifying retrieval and analysis while supporting real-time updates and incremental database construction, maintaining data timeliness and integrity.

[0063] Figure 5 A flowchart illustrating an example of constructing or updating a vector database according to an embodiment of this application is shown.

[0064] like Figure 5 As shown, in step S510, at least one coded column is determined in the relational database, and the coded column contains multiple sequentially arranged table cells.

[0065] Here, coded columns can be content with certain semantic meaning or related to the query task. For example, by analyzing the data attributes and semantic characteristics of columns, coded columns can be filtered to focus on table data with high semantic value and avoid processing irrelevant or redundant information.

[0066] In step S520, for each coded column, the cell content of each table cell in the coded column is vector-encoded and associated with the corresponding cell index to build or update the vector database.

[0067] In some implementations, different encoding methods can be used for cells with different content types. For example, a pre-trained language model can be used to encode cell text, and numerical content can be normalized. Furthermore, the encoded vectors are associated with the corresponding cell indexes and stored in a vector database. This cell indexes enable the data source connection between the vector database and the relational database, which helps improve the efficiency and accuracy of cross-database collaborative retrieval.

[0068] Figure 6 A flowchart illustrating an example of cross-library association storage of non-text elements according to an embodiment of this application is shown.

[0069] like Figure 6 As shown, in step S610, the coordinates of the element region of non-text elements are extracted.

[0070] In some implementations, document parsing tools are used to identify the boundaries of non-text elements and extract their coordinate information. For example, by identifying the four boundary coordinates of a chart on the page, the physical location of a non-text element in the document can be determined, supporting data traceability.

[0071] In step S620, the element region coordinates are associated with the corresponding element description summary to update the relational database.

[0072] In step S630, the element encoding vector corresponding to the non-text element is determined based on the vector encoding model, and the corresponding element region coordinates are associated and recorded to update the vector database.

[0073] In some implementations, vectorized non-text elements are associated with their region coordinates using a unique identifier and stored in a vector database.

[0074] Through the embodiments of this application, the element region coordinates of non-text elements are used to store and associate the element description summaries and vectorized data corresponding to the non-text elements in relational databases and vector databases, forming a unified data storage structure. By locating the element region coordinates, the original position of non-text elements in a document can be quickly located. Accurate and reliable cross-database filtering can be performed based on the region information of non-text elements related to question-and-answer questions, improving the accuracy of cross-database collaborative retrieval results for documents involving non-text elements.

[0075] In some examples of embodiments of this application, to address the problem of missing symbol databases, semi-structured documents such as PDFs are used as external knowledge sources. By manually designing the database schema and utilizing a large number of open-source parsing tools, the original documents are divided into multiple views, hypertext elements are extracted, and they are written into the database in the form of a set of pipeline functions.

[0076] In addition, for the parsing of semi-structured documents, academic APIs are used to extract metadata, text parsing tools provide text content, and segmentation is performed based on different perspectives, such as chapters, paragraphs, and pages at different granularities. For non-text elements, the corresponding elements are extracted through an OCR model. Finally, a language or visual big data model is used to summarize the segmented and extracted elements.

[0077] In addition, the two retrieval modes (neural vector retrieval and symbolic language retrieval) are uniformly organized in the form of tool calls, allowing large model agents to interact with the external environment through generator function API actions in multiple iterations, thereby enabling them to autonomously retrieve content from databases or vector libraries and assist in the generation of the final answer.

[0078] Figure 7 A flowchart illustrating an example of a document question-answering method according to an embodiment of this application is shown, which describes a real-world simplified case from a labeled dataset (§3.1). "[T]", "[A]", and "[O]" represent thought, action, and observation, respectively.

[0079] like Figure 7As shown, the complete system flow (see diagram below) can be divided into three stages: Stage 1) Multi-view document parsing, Stage 2) Multimodal vector encoding, and Stage 3) Iterative agent interaction. Specifically, when a PDF of an academic paper is imported into the system: Phase 1) The parsed content will be populated into a relational database through the following four steps (the database schema will be manually designed in advance): Obtain paper metadata, such as author, conference, and citation information, through academic APIs; The text content of the PDF is obtained through text extraction tools, such as PyMuPDF, and the original text is segmented at different granularities according to different segmentation schemes, such as based on pages, chapters, and paragraphs. Use OCR visual element extraction tools, such as MinerU, to extract non-text elements from PDFs, such as images, tables, and formulas. Finally, using large language or visual models, corresponding summary descriptions are generated for longer and self-contained text fragments or charts, supplementing and enhancing the original content.

[0080] Phase 2) identifies enable columns in the database schema, such as longer text of type varchar (e.g., pages.page_summary) and columns recording the coordinates of the four corners of the visual region (images.bounding_box). The corresponding text content or captured image is then fed into the vector encoding model to obtain the corresponding multimodal vector codes. These vector codes will be stored in a vector library for subsequent retrieval.

[0081] Phase 3) Based on the constructed database and vector library, an active retrieval question-answering agent was built. It can interact with the real physical environment in real time. Based on the questions raised by the user, it predicts the executable actions, such as SQL programs to query the database and action APIs to retrieve the vector library. These actions will be executed in the corresponding environment, so that the agent can obtain the expected context in an iterative manner to answer professional and detailed questions.

[0082] Figure 8 A schematic diagram illustrating the effect of an example of iterative vector retrieval and iterative symbol retrieval is shown.

[0083] like Figure 8As shown, in iterative vector retrieval, the backend environment includes a vector database but not a relational database. It can handle queries involving semantic fuzzy matching and word form transformation, but it cannot handle problems such as aggregation calculations, comparisons, and operations very well. In iterative symbolic retrieval, the backend environment includes a relational database but not a vector database. It can handle problems such as aggregation calculations, comparisons, and operations, but it cannot handle semantic fuzzy matching very well.

[0084] Figure 9 A schematic diagram illustrating the effect of an example of a hybrid neural symbol retrieval enhancement system provided according to an embodiment of this application is shown.

[0085] like Figure 9 As shown, the hybrid neural symbolic retrieval enhancement system includes both vector and symbolic databases. By combining neural vector retrieval and symbolic language retrieval paradigms, a multi-view, multimodal document segmentation and encoding method is proposed, constructing an intelligent question-answering system capable of interacting with real physical environments. This aims to improve the accuracy of precise question answering and reduce the support of large models for long text inputs. With the development of artificial intelligence and the surge in the number of papers, this system helps researchers quickly obtain details of the latest papers, greatly improving the efficiency of scientific research.

[0086] It should be noted that semi-structured documents are widely used in various fields of industry. This system is not only suitable for academic paper retrieval and question-and-answering, but also has long-term research significance and value for other scenarios that rely on external knowledge documents, such as those in the medical, legal, and financial fields. Similar intelligent retrieval and question-and-answer systems can help relevant practitioners free themselves from massive amounts of literature and improve their productivity.

[0087] The following text will start with the task of fine-grained question answering based on semi-structured PDF documents, explore the application details of retrieval enhancement generation technology in this task, and elaborate on the technical solution.

[0088] This application proposes a hybrid neural symbol retrieval framework (NeuSym-RAG) for PDF question answering, combining two retrieval paradigms into an interactive workflow. In the preprocessing stage, for each paper's PDF file, metadata processing, raw text segmentation based on different views, and extraction of various embedded paratextual elements (such as tables and graphs) are performed through a series of parsing functions (§2.2). These identified elements are populated into a relational database specifically designed for semi-structured PDFs. Subsequently, encodeable column values ​​are selected from the populated database and stored in a vector store (§2.3). It is noteworthy that the vectorized unit values ​​of different columns provide diverse perspectives for interpreting the PDF content. Furthermore, a database schema diagram is used to organize these values ​​into a well-structured form.

[0089] When answering user questions, the LLM agent can adaptively predict executable actions (§2.4.1) and retrieve the necessary information from the backend environment (database or vector store) in multiple rounds. When the collected information is sufficient to answer the input question, the LLM agent will predict the termination action "GENERATE ANSWER" to generate the final answer.

[0090] Validating the NeuSym-RAG framework To verify the effectiveness of the NeuSym-RAG framework, this application establishes a complete PDF-based question-answering dataset (AIRQA-100) for artificial intelligence research papers. This dataset contains 100 real-world examples with instance-specific evaluation metrics (§3.1). Experiments on closed-source and open-source large language models (LLMs) show that NeuSym-RAG significantly outperforms classical neural retrieval methods (by more than 13 percentage points). Furthermore, a detailed ablation study highlights the following key points: 1) The necessity of iterative interaction; 2) The advantages of integrating neural and symbolic retrieval paradigms and multi-view segmentation methods; 3) Changes in different operation and observation formats; 4) It has low sensitivity to LLM selection in the pretreatment stage, but high sensitivity to subjective evaluation.

[0091] 2 NeuSym-RAG framework This section presents the complete framework of NeuSym-RAG, including three phases (see...). Figure 2 ).

[0092] 2.1 Overall Process The workflow for handling user issues in semi-structured PDF files is as follows: 1. Analysis Phase First, the original PDF file is input into a series of functional pipelines, segmented using a multi-view approach, non-text elements are extracted, and these elements are stored in a relational schema-restricted database (DB).

[0093] 2. Coding stage Next, the coded columns in the database are identified, and vectors of cell values ​​are obtained using embedding models for different modalities and inserted into the vector store (VS).

[0094] 3. Interaction Phase Finally, an iterative question-answering agent was constructed, which can predict executable actions, retrieve context from the backend environment (database or vector storage), and answer input questions.

[0095] 2.2 Multi-view document parsing At this stage, the objective of this application embodiment is to parse each incoming PDF file from multiple perspectives and transform it into the relational database DuckDB. The parsing pipeline includes the following steps (see reference). Figure 7 (the middle part) 1) Query academic APIs (such as dblp) to obtain author information and conference metadata by paper title, so as to support metadata-based filtering during the search process.

[0096] 2) Divide the text content according to different granularities, such as pages, chapters, and fixed-length continuous words.

[0097] 3) Use an OCR model to extract non-text elements (e.g., choose to use the tool MinerU).

[0098] 4) Use Large Language Models (LLMs) or Visual Language Models (VLMs) to generate brief summaries of parsed text, tables, and images.

[0099] The acquired metadata, parsed elements, and generated digests will all be populated into the symbol database. This application embodiment has a pre-designed database schema, which is carefully crafted and applicable to PDF documents (see...). Figure 7 (the middle part).

[0100] 2.3 Multimodal Vector Coding Figure 10 This illustration shows an example of how a coded unit value in a database is converted into a data entry in a vector storage according to an embodiment of this application.

[0101] After symbol processing is complete, the process moves on to neural encoding of the parsed elements. First, each column in the database schema is labeled as either "encodeable" or "non-encodeable".

[0102] - For text modalities, columns with data type "varchar" are marked as enable if their cell values ​​are relatively long (e.g., ...). Figure 10 (The column "pages.page_summary" in the table).

[0103] - For image modalities, the "bounding_box" column, which records the four coordinates of the graph or table, is used to extract rectangular regions from the PDF.

[0104] Subsequently, encoding models for text and image modalities are used to vectorize text fragments or cropped images.

[0105] To establish a one-to-one mapping between each cell value in the database and the neural vector in the Vector Store (VS), each data point in the VS is supplemented with its corresponding table name, column name, and primary key value for that row. This triple uniquely identifies each value in the database. Additionally, two extra fields, "paper_id" and "page_number," are added and included in a JSON dictionary to support metadata filtering. These data entries are then inserted into the VS and categorized into different sets based on the encoding model and modality.

[0106] Through the first two phases (§2.2 and §2.3), the various segmentation perspectives in VS are intrinsically linked through a structured database schema. Furthermore, long texts or visual bounding boxes in the database are vectorized and stored in VS to support fuzzy semantic matching.

[0107] 2.4 Iterative Agent Interaction With the database (DB) and vector storage (VS) already populated, a RAG agent can be built that can proactively retrieve contextual information from the DB and VS.

[0108] 2.4.1 Motion Space Design First, embodiments of this application define five parameterized actions that an intelligent agent can perform during interaction, including: - RETRIEVE FROM VECTOR STORE - RETRIEVE FROM DATABASE - VIEW IMAGE - CALCULATE EXPR - GENERATE ANSWER #### RETRIEVE FROM VECTOR STORE This action transforms classic static retrieval into real-time dynamic retrieval. It supports a variety of adjustable parameters (see code listing 1), such as the "query" parameter which encourages the LLM to more clearly rewrite the user intent, while the "table_name" and "column_name" parameters require the agent to choose an appropriate perspective for retrieval.

[0109] During execution, the system retrieves context related to "query" from the VS based on specified constraints (such as the "filter" parameter, like page_number=1) and returns the observations in tabular format.

[0110] Code Listing 1: The RETRIEVE FROM VECTOR STORE action and its function call format parameters.

[0111] #### RETRIEVE FROM DATABASE This action accepts a single parameter, "sql". By executing the provided SQL query, the agent retrieves a result table from a pre-processed database. Similar to ToolSQL, the agent can explore the database by predicting different SQL queries through iterative symbol retrieval and interpret PDF content using multi-view parsing.

[0112] #### VIEW IMAGE To address potential errors in the first stage (§2.2) of PDF parsing and to fully leverage the features of higher-level Visual Language Models (VLMs), this application embodiment designs an action to extract the cropped region (defined by "bounding_box") from the PDF page and return it to the agent as a base64 encoded image.

[0113] Specific coordinates can be retrieved from the database (DB) or visual representation (VS) during interaction. In this way, the agent can obtain the required image observations and perform inference based on them.

[0114] Code Listing 2: Parameters for the VIEWIMAGE action and its function call format.

[0115] CALCULATE EXPR This action, which accepts Python expressions (e.g., "2 + 3 * 4") and returns the calculation result, was integrated into preliminary studies to reliably reduce illusions in mathematical problems.

[0116] GENERATE ANSWER This is the termination action; the agent returns the final answer when it determines that the retrieved context is sufficient to solve the input problem. The only parameter of this action, "answer," can be of any type depending on the user's requirements.

[0117] 2.4.2 Hybrid Neural Symbol Retrieval In each round of interaction, the agent predicts an action, interacts with the environment, and obtains real-time observations. This embodiment employs the ReAct framework, requiring the agent to first output its thought process (see...). Figure 7 (Right side). This iterative retrieval method allows the agent to leverage the complementary advantages of two retrieval paradigms in successive steps.

[0118] For example, an agent can first filter relevant rows by executing SQL programs in the database. `RETRIEVE FROM DATABASE` performs particularly well in handling structured queries and metadata-based constraints. After extracting the primary key values ​​of the rows as intermediate output, these primary key values ​​can be further inserted into the "filter" parameter of the action `RETRIEVE FROM VECTOR STORE` for neural semantic matching. For example: `filter='primary_key in [the primary key value of the SQL execution result]'`.

[0119] Conversely, an agent can first use neural retrieval to query the VS to select the most relevant entries, and then treat this intermediate set as a temporary table or condition for subsequent SQL retrieval.

[0120] It should be noted that these tips provide a set of suggestions to guide the interaction process, emphasizing best practices for sequential decision-making in information retrieval, iterative optimization, and task solving. The tips are highly customizable and can be easily extended to address LLM prediction errors. Therefore, only one example tip needs to be provided for the complete NeuSym-RAG framework.

[0121] In addition, the task prompt defines the specific input content for the current user's question, and should include at least the following: 1) User input issues; 2) Answer format; 3) Database or vector storage architecture (if required).

[0122] Here, a code representation format can be used to define the database schema, and combined with a schema description to enhance schema linking functionality. For vector storage schemas, this application embodiment introduces the following: 1) All available sets; 2) Fields for each stored data entry; 3) Encoding (table, column) pairs from the corresponding DuckDB; 4) Filter the available valid operators in the conditions during vector search.

[0123] Specifically, the prompt template for triggering NeuSym-RAG has been described above, while task prompts for other methods can be easily derived based on whether the backend environment integrates a database or vector storage.

[0124] 3. Experiment This section introduces the human-annotated dataset AIRQA-100, the main experiments, and ablation studies.

[0125] 3.1 Question-answering datasets from artificial intelligence research papers Previous question-answering datasets for academic papers typically focused on simple questions based on a single page of a PDF or even just the title and abstract, which is far removed from real-world scenarios (see §4 for details). Therefore, this application's embodiments annotate a more complex question-answering dataset involving complete PDF documents, named AIRQA-100.

[0126] This dataset is based on 100 PDF files from ACL conference papers in 2023 and 2024. Six researchers with artificial intelligence (AI) backgrounds manually annotated 100 questions covering five categories: text, tables, images, formulas, and metadata. Table 1 shows an example for each category.

[0127] Table 1: Examples of each category in the AIRQA-100 dataset. It's worth noting that a question may belong to multiple categories simultaneously, requiring various skills to obtain the target answer.

[0128] Evaluation indicators While most long-text question-answering datasets rely on LLM or human experts for accuracy evaluation, this application proposes more flexible and accurate evaluation metrics. Specifically: 1) Use answer format restrictions and attach Python-style output restrictions next to the question (e.g., "Your answer should be a Python list containing two strings").

[0129] 2) Implement instance-based performance evaluation by designing 15 indicator functions with optional parameters. These functions are divided into subjective and objective categories, depending on whether LLM judgments are involved.

[0130] Unless otherwise stated, this application uses GPT-4o-2024-11-20 as an LLM-based subjective evaluation tool.

[0131] 3.2 Experimental Setup Different agent methods Here, six baseline methods for intelligent agents are provided; see details. Figure 6 : Classic RAG Retrieve question-related fragments from the vector storage and use them directly as context for the LLM to answer the question.

[0132] - Iterative Classic RAG LLM allows for repeated modification of the query text and iterative retrieval of fragments until an answer is obtained.

[0133] - Two-stage Neu-RAG The task is divided into two phases: in the first phase, LLM predicts a RETRIEVE FROM VECTOR STORE action; in the second phase, LLM outputs the answer using the retrieved context.

[0134] - Iterative Neu-RAG Supporting a trial-and-error mechanism, LLM can predict multiple parameterized actions and retrieve content from the vector store until the interaction trajectory is sufficient to answer the question.

[0135] Two-stage Sym-RAG The requirement is that the LLM first generates an SQL query, and then can only use the context of a single retrieval to predict the answer.

[0136] - Iterative Sym-RAG This is a multi-round version; LLM can interact with the symbol database multiple times to arrive at a better final answer.

[0137] LLM and Hyperparameters This application uses various LLM methods to evaluate the performance of the NeuSym-RAG method: - Closed-source models: including GPT-4omini-2024-07-18 and GPT-4-1106-vision-preview.

[0138] - Open source models: including Qwen2.5-72B-Instruct and Llama-3.3-70B-Instruct.

[0139] For GPT series models that support image input, the embodiments of this application also verify the effect of visual modality enhancement.

[0140] Regarding hyperparameter settings: - Set the temperature to 0.7; - top_p is fixed at 0.95; - The maximum length of text terms retrieved in each round is limited to 5000; - The maximum number of interaction rounds is 20.

[0141] To improve efficiency, this application embodiment limits the history window size to 5, that is, only the interaction records of the most recent 5 rounds are retained.

[0142] It should be noted that the available action types differ depending on the agent baseline: - Classic RAG: No action or lookout space cues; - Iterative Classic RAG: Only supports the RETRIEVE FROM VECTOR STORE and GENERATEANSWER actions; the former's view is fixed to "chunks.text_content"; - Two-stage Neu-RAG: The first stage only supports the RETRIEVE FROM VECTOR STORE action, while the second stage must be the GENERATE ANSWER action; - Iterative Neu-RAG: Supports all available action types, including [RETRIEVE FROM VECTORSTORE, CALCULATE EXPR, VIEW IMAGE, GENERATE ANSWER]; - Two-stage Sym-RAG: The first stage only supports the RETRIEVE FROM DATABASE action, while the second stage must be the GENERATE ANSWER action; - Iterative Sym-RAG supports all available action types, including [RETRIEVE FROM DATABASE, CALCULATE EXPR, VIEWI MAGE, GENERATE ANSWER].

[0143] 3.3 Main Experiments Table 2: Different LLM performances of Classic RAG and NeuSym-RAG on the AIRQA-100 dataset. "+vision" indicates that the LLM can retrieve visual context through image embedding and incorporate VIEWIMAGE actions into the action space.

[0144] Table 2 shows the performance of the method proposed in this application on different LLMs. A detailed analysis follows: 1) NeuSym-RAG is significantly superior to Classic RAG NeuSym-RAG improved performance by at least 13% across all LLMs. Through more customizable actions and multiple trials, NeuSym-RAG is more fault-tolerant in interacting with the backend environment and retrieves appropriate question-and-answer information through observation and learning.

[0145] 2) Visual modality further enhances performance. In image classification problems, giving agents the ability to directly access the original images significantly improves efficiency, and the addition of visual modalities has a significant enhancement to overall performance.

[0146] 3) The outstanding performance of open-source LLM Even open-source LLMs can handle such complex interactions in zero-sample settings. For example, NeuSym-RAG achieves a 22 percentage point improvement in overall accuracy on Qwen2.5-72B-Instruct compared to Classic RAG, and even surpasses the closed-source LLM GPT-4omini by as much as 11 percentage points, demonstrating NeuSym-RAG's versatility.

[0147] 4) Comparison of subjective and objective issues Compared to subjective questions that typically require long text answers, objective questions generally exhibit a higher level of task difficulty (as evidenced by accuracy). This phenomenon also exposes the limitations of previous precision question-answering efforts, namely that objective metrics cannot be entirely replaced by subjective assessments.

[0148] 3.4 Ablation Study For the sake of brevity, "GPT" will be used in this section to refer to GPT-4omini, and "Qwen" will be used to refer to Qwen2.5-72B-Instruct. Unless otherwise stated, experiments will be conducted primarily on Qwen.

[0149] ### Different Agent Approaches Table 3: Performance of different RAG methods on the AIRQA-100 dataset, where "# Interaction(s)" indicates the number of LLM calls for a single problem. For a detailed description of each RAG baseline method, see §3.2.

[0150] To analyze the contributions of each component in NeuSym-RAG, the performance of the agent method described in §3.2 was compared. Table 3 shows the results: 1) The two-stage retrieval scheme is superior to the Classic RAG baseline. Both neural and symbolic retrieval two-stage schemes showed consistent performance improvements. This can be attributed to the agent's ability to adaptively determine action parameters, such as SQL queries in a database or segmentation perspectives in neural retrieval.

[0151] 2) Iterative retrieval is superior to two-stage schemes. Through multiple rounds of interaction, the agent can explore the backend environment and select the most relevant information to answer questions.

[0152] 3) Different search paradigms perform well in specific categories. For example, in the metadata category, Sym-RAG is significantly better than Neu-RAG, demonstrating the advantage of symbol retrieval in structured knowledge.

[0153] 4) NeuSym-RAG outperforms all baseline methods overall. This verifies the complementary effect of multi-perspective retrieval and combining the two retrieval strategies.

[0154] ### Different prompt formats This application also investigates different formats for motion and view space in its embodiments. Based on the results in Table 4: Table 4: Ablation studies with different actions and observation formats (see Appendices A.2.2 and A.2.3 for details).

[0155] 1) The format of the action has a greater impact on the result. Compared to observation format, action format has a more significant impact on performance.

[0156] 2) The "markdown" action format performs best. This Python function call-like format performs best, likely because the other three action formats usually have stricter formatting requirements.

[0157] 3) Different LLMs have different preferences for observation formats. The performance of different observation formats may be affected by the LLM training corpus, but the performance difference is relatively small, especially for the two best-performing options.

[0158] ### Multi-view Segmentation In the Classic RAG baseline, the original document is segmented into 512 consecutive tokens. This application further explores the impact of different segmentation perspectives in its embodiments.

[0159] Table 5: Performance of the Classic RAG baseline on different search options on the AIRQA-100 dataset.

[0160] The results in Table 5 show that the classic segmentation strategy performs best overall. However, segmentation perspectives tailored to specific aspects may achieve the best results within their respective categories.

[0161] Details of preprocessing and evaluation To address the high cost of API calls, this application's embodiments explore the impact of open-source and closed-source LLMs in the following two aspects: 1) PDF pre-parsing 2) Subjective assessment Table 6: Performance of different LLM summarization models on the AIRQA-100 dataset.

[0162] As shown in Table 6, the element summaries generated by different LLMs are not sensitive to the experimental results. Therefore, the lower-cost open-source LLM is a feasible alternative to the closed-source LLM.

[0163] Figure 11 A simulation illustration shows an example of the consistency between human evaluation and LLM-based evaluation, where examples judged correctly by humans are classified as positive samples.

[0164] During the evaluation phase, AI researchers were invited to independently assess the semantic equivalence between the agent's predicted answer and the reference answer. Figure 11 This indicates that, in most cases, LLM assessments are highly consistent with human judgment.

[0165] It is worth noting that the closed-source GPT-4o significantly outperformed its open-source counterpart: it perfectly matched human judgment, while open-source LLMs occasionally appeared too strict or too lenient. Therefore, GPT-4o was chosen as the LLM evaluation tool in all experiments.

[0166] 3.5 Case Study This case study, using the AIRQA-100 dataset, demonstrates the interaction trajectories of three iterative methods in answering the same question. The question is: "Of the six different methods tested in the paper, how much higher is the WER value of DD2 compared to NV1 for ASR?" The detailed process for each method is as follows: ### Iterative Neu-RAG Round 1 - Reflection: I will try to search for relevant chapters by focusing on content that mentions WER values.

[0167] - Action: RETRIEVE FROM VECTOR STORE (Query content containing "WER values ​​for DD2 and NV1methods").

[0168] - Observation: The returned content mentions " Figure 3 and Figure 4 The WER and PER values ​​are displayed separately.

[0169] Round 2 - Reflection: Based on observation, the WER value difference between DD2 and NV1 is 0.57.

[0170] - Action: GENERATE ANSWER (Answer = 0.57).

[0171] Conclusion: The answer is inconsistent with reality because the graphic content was not searched more thoroughly.

[0172] ### Iterative Sym-RAG Round 1 - Reflection: I will check the "sections" table to see if the WER value is mentioned in the chapter.

[0173] - Action: RETRIEVE FROM DATABASE (SQL query "Retrieves all chapter contents from the sections table").

[0174] - Observation: Return to the content of chapter "5 Experiments".

[0175] Round 2 - Reflection: I will try to extract the WER value from the "5 Experiments" section.

[0176] - Action: Search the "5 Experiments" section again.

[0177] - Observation: No valid WER value information was retrieved.

[0178] Round 3 - Consider: Assume that the WER value of DD2 is 25.0 and NV1 is 10.0, and the difference is 15.0.

[0179] - Action: GENERATE ANSWER (Answer = 15.0).

[0180] Conclusion: Due to the lack of effective use of context, the assumed answer deviates significantly from reality.

[0181] ### NeuSym-RAG Round 1 - Thoughts: I will use vector storage for similarity search to find relevant context containing WER values.

[0182] - Action: RETRIEVE FROM VECTOR STORE (Query "WER values ​​for DD2 and NV1").

[0183] - Observation: Relevant context mentions " Figure 3 and Figure 4 The WER and PER values ​​are displayed.

[0184] Round 2 - Thinking: I will query the "figures" table to find figure captions that may contain WER values.

[0185] - Action: RETRIEVE FROM DATABASE (query) Figure 3 and Figure 4 (Caption).

[0186] - Observation: Return to the figure caption " Figure 3 "DD2 vs. other methods' WER scores."

[0187] Round 3 - Think about it: Try retrieving from the "images" table. Figure 3 Detailed information.

[0188] - Action: RETRIEVE FROM DATABASE Figure 3 (Graphical summary).

[0189] - Observation: The graphical summary shows the WER values ​​for the six methods, including DD2 (0.729) and NV1 (0.131).

[0190] Round 4 - Think about it: Calculate the difference in WER values ​​between DD2 and NV1.

[0191] - Action: CALCULATE EXPR(expression "0.729 - 0.131").

[0192] - Observation: The calculated result is 0.598.

[0193] Round 5 - Note: The difference in WER values ​​between DD2 and NV1 is 0.598.

[0194] - Action: GENERATE ANSWER (Answer = 0.598).

[0195] Conclusion: NeuSym-RAG achieved the correct answer by combining symbolic retrieval and neural vector retrieval, fully demonstrating the synergistic advantages of the two retrieval strategies.

[0196] Summarize: - Iterative Neu-RAG failed to utilize the graphical information in the database and gave incorrect answers based solely on fragment retrieval.

[0197] - The Iterative Sym-RAG, due to its lack of efficient processing of graphical data, makes assumptions that deviate from reality.

[0198] NeuSym-RAG successfully acquired and analyzed relevant graphic and textual content by combining vector storage and symbol database, providing an accurate answer (0.598).

[0199] In summary, the case study demonstrates the interaction trajectories of three iterative approaches to the same problem. NeuSym-RAG first predicts a RETRIEVE FROM VECTOR STORE action to search for fragments related to the "WER value". After detecting a relevant graph, it predicts a RETRIEVE FROM DATABASE action to access the graph's title and summary. In contrast, Iterative Neu-RAG fails to utilize the graph due to a lack of database, while Iterative Sym-RAG struggles to locate the relevant context.

[0200] In this case, the advantages of combining the two retrieval actions are clearly visible: VS helps the agent identify relevant context, while DB handles precise queries. This synergy achieves accurate answers that a single retrieval method cannot, highlighting the importance of the collaboration between the two retrieval strategies.

[0201] AIRQA-100 dataset ### Data Format This section briefly introduces the data format of the AIRQA-100 dataset. Each data instance is represented as a JSON dictionary containing the following fields.

[0202] - uuid: The globally unique UUID of the current task example.

[0203] - Question: A question about the given paper.

[0204] - answer_format: The LLM output format requirement so that the embodiments of this application can easily evaluate the answer.

[0205] - tags: A list of tags representing the different categories to which this task belongs.

[0206] - pdf_id: A list of PDF IDs used to answer a given question.

[0207] - conference: A list of conference names corresponding to the PDF.

[0208] - reasoning_steps: A series of manually annotated reasoning steps used to find the answer to a given question.

[0209] - `evaluator`: A dictionary with two fields, `eval_func` and `eval_kwargs`, used to define how to evaluate the model output. Specifically: - The `eval_func` field defines the name of a custom function (or metric) in this application embodiment, used to compare the predicted results with the expected gold standard results; The `eval_kwargs` field defines the parameters of the corresponding evaluation function, which typically contains the gold standard answer or a reference answer.

[0210] For example, the function "eval_scoring_points_with_llm" can be used to score the output of the LLM based on three given "scoring_points".

[0211] - state: A dictionary indicating whether the corresponding LLM can correctly answer the questions given the paper title and question. The prefix "`gui-`" indicates that a web chat interface is used instead of an API.

[0212] ###Statistical Data Table 8: Number of different types of examples in the AIRQA-100 dataset Table 8 presents the statistics for the AIRQA-100 dataset. Note that a task example can be labeled with multiple category labels (i.e., text, table, image, formula, and metadata).

[0213] Supplementary experiments and setup Figure 12 A simulation diagram illustrating an example of the frequency distribution of question length is shown. Table 9 presents some statistics on the iterative methods across different models. Overall, the GPT series of large language models (LLMs) completes the task in fewer interaction rounds, indicating that closed-source LLMs demonstrate greater capability and confidence. When using Qwen2.5-72B-InstructLLM, the NeuSym-RAG method requires more interaction rounds than the other two iterative methods because the NeuSym-RAG method is able to integrate contextual information from both retrieval paradigms.

[0214] Table 9: Interaction count, cumulative number of prompt lexical units and completed lexical units, time consumption, and LLM cost statistics for different models and RAG methods in the AIRQA-100 dataset.

[0215] Table 10: Performance of NeuSym-RAG with different text encoding models on the AIRQA-100 dataset. Experiments were conducted using the open-source LLM Qwen2.5-72B-Instruct.

[0216] Table 10 shows the performance of NeuSym-RAG based on Qwen2.5-72B-Instruct LLM when applying different text modality sets (i.e., encoding models) in vector storage. Experimental results reveal that a single BM25 text set is sufficient to achieve optimal performance. This application's embodiments speculate that this phenomenon may have two reasons: 1) Existing LLMs are still unable to effectively explore the retrieved context from multiple encoding models and select the optimal one. This behavior pattern requires deep reasoning capabilities. Embodiments of this application plan to verify this hypothesis by employing supervised fine-tuning with agent trajectories in future work.

[0217] 2) Another potential insight is that, for different coding models, the best practice is to directly choose the model that performs best, rather than leaving the selection burden to the agent.

[0218] For closed-source GPT models, this application embodiment directly uses their API services. For open-source models, this application embodiment deploys them on an NVIDIA A800 Tensor Core cluster and uses vLLM 2 (Kwon et al., 2023) for assistance.

[0219] ### Encodeable Columns in Database Architecture and Vector Storage Figure 13 A schematic diagram of an example of a complete and general database architecture for storing parsed elements of each PDF file is shown, visualized using the free online tool drawSQL. The table “chunks” stores the segmented text fragments using a traditional continuous lexicalization strategy (512 lexicals). Note that the actual data types of the columns “bounding_box”, “authors”, “tags”, and “page_numbers” are INT[4], VARCHAR[], VARCHAR[], and INT[], respectively, because the visualization tool drawSQL cannot display the “ARRAY” data type.

[0220] Specifically, all coded columns (the sources of their cell values) provide multiple perspectives for interpreting PDF content. These different perspectives are essentially interconnected through subgraphs of the original database architecture (e.g., Figure 14 (As shown).

[0221] Figure 14 A schematic diagram of an example of all coded columns in the database is shown, which are essentially interconnected through a schema subgraph. Note that the actual data types of the columns “images.bounding_box” and “tables.bounding_box” are INT[4] because the visualization tool drawSQL cannot display the “ARRAY” data type.

[0222] 4. Related Work #### Structured and Iterative Retrieval Existing literature has proposed various RAG methods for structured knowledge sources: - GraphRAG (Edge et al., 2024) summarizes parts and the final answer by constructing a knowledge graph (KG).

[0223] - StructRAG (Li et al., 2024) refactors documents to fit specific task structures.

[0224] HybridRAG (Sarmah et al., 2024) integrates KG and vector-based retrieval methods, focusing on financial documents.

[0225] FastRAG (Abane et al., 2024) combines text retrieval and KG to extract structured data through pattern and script learning.

[0226] - Blended RAG (Sawarkar et al., 2024) uses a hybrid query strategy to achieve semantic retrieval.

[0227] Furthermore, some work has demonstrated that agents can benefit from iterative retrieval: - IRCoT (Trivedi et al., 2023) handles multi-step question answering by combining retrieval with mind chaining.

[0228] - FLARE (Jiang et al., 2023) guides retrieval iteratively by predicting sentences and optimizes low-confidence results.

[0229] - Iter-RetGen (Shao et al., 2023) iteratively combines retrieval and generation to integrate knowledge.

[0230] - CoA (Zhang et al., 2024) uses multiple agents to handle long-context tasks, with a management agent integrating the results.

[0231] Based on the NeuSym-RAG provided in the embodiments of this application, neural and symbolic retrieval are combined, and multi-view context is obtained through multi-turn interaction.

[0232] #### Questions and Answers about PDF Documents Existing question-answering datasets for PDF documents often neglect their underlying structure and layout: - BioMRC (Pappas et al., 2020) and QASPER (Dasigi et al., 2021) generate questions based solely on the title and abstract.

[0233] - FinQA (Chen et al., 2021) targets single-page question and answer.

[0234] - QASA (Lee et al., 2023) retains the chapter structure but lacks key elements such as graphics and tables.

[0235] - Recent Visconde (Pereira et al., 2023) and SCIPIP (Wang et al., 2024b) focus on long text subjective tasks using neural retrieval.

[0236] This study addresses these shortcomings by processing PDF files of complete AI research papers and incorporating objective evaluations.

[0237] 5. Conclusion In this study, this application proposes a hybrid neural symbol retrieval framework (NeuSym-RAG) for question answering tasks on semi-structured PDFs. By parsing the original PDF file into a structured database, this application can obtain multi-perspective content that surpasses traditional fixed-length segmentation. Simultaneously, NeuSym-RAG treats contextual retrieval as an iterative interactive process, where an agent based on LLM or VLM actively predicts different types of executable actions to query the backend database or vector storage. Experiments on the human-annotated AI research question-answering dataset AIRQA-100 show that NeuSym-RAG significantly improves accuracy by more than 13 percentage points compared to the classic RAG baseline method.

[0238] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0239] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions, which can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform any of the document question-and-answer methods described above.

[0240] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform any of the above-described document question-and-answer methods.

[0241] In some embodiments, this application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a document question-and-answer method.

[0242] Figure 15 This is a schematic diagram of the hardware structure of an electronic device for executing a document question-and-answer method according to another embodiment of this application, such as... Figure 15 As shown, the device includes: One or more processors 1510 and memory 1520, Figure 15 Take the 1510 processor as an example.

[0243] The device for performing the document question-and-answer method may further include an input device 1530 and an output device 1540.

[0244] The processor 1510, memory 1520, input device 1530, and output device 1540 can be connected via a bus or other means. Figure 15 Taking the example of a connection between China and Israel via a bus.

[0245] The memory 1520, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the document question-and-answer method in the embodiments of this application. The processor 1510 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 1520, thereby implementing the document question-and-answer method of the above-described method embodiments.

[0246] Memory 1520 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. Furthermore, memory 1520 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 1520 may optionally include memory remotely located relative to processor 1510, and these remote memories may be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0247] Input device 1530 can receive input digital or character information and generate signals related to user settings and function control of the electronic device. Output device 1540 may include display devices such as a display screen.

[0248] The one or more modules are stored in the memory 1520, and when executed by the one or more processors 1510, they execute the document question-and-answer method in any of the above method embodiments.

[0249] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0250] The electronic devices in this application embodiments exist in various forms, including but not limited to: (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily aim to provide voice and data communication. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0251] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, possessing computing and processing capabilities, and generally also have mobile internet access features. These terminals include: PDAs, MIDs, and UMPCs, etc.

[0252] (3) Portable entertainment devices: These devices can display and play multimedia content. This category includes audio and video players, handheld game consoles, e-book readers, as well as smart toys and portable car navigation devices.

[0253] (4) Other airborne electronic devices with data interaction capabilities, such as vehicle-mounted systems installed on vehicles.

[0254] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0255] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0256] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A document question-and-answer method, comprising: The content of the first document database is retrieved based on the query statement to determine the corresponding intermediate search results; Based on the query statement and the intermediate search results, content retrieval is performed on the second document database to determine the search matching results. The first document database and the second document database are databases constructed based on at least one document. The first document database is a vector database and the second document database is a relational database, or the first document database is a relational database and the second document database is a vector database. Detect whether the search matching results meet the response conditions of the query statement; When the search matching result meets the response condition, a response answer is constructed based on the search matching result.

2. The method according to claim 1, further comprising: When the search matching result does not meet the response condition, the search strategy action corresponding to the search matching result is determined to iteratively search the first document database or the second document database for content until the search matching result re-determined through iterative retrieval meets the response condition.

3. The method according to claim 2, wherein, The method is applied to large language models, and the method further includes: Based on the first search suggestion template, a content search is triggered against the first document database; Based on the second search suggestion template, a content search is triggered for the second document database; Based on the inspection prompt template, the detection of the aforementioned response condition is triggered; Based on the action prompt template, the action to determine the retrieval strategy is triggered; Based on the reply prompt template, a reply answer is constructed based on the search matching results.

4. The method according to claim 1 or 2, wherein, The relational database also contains element description summaries determined based on non-text elements in the document, and / or the vector database also contains element encoding vectors determined based on non-text elements in the document.

5. The method according to claim 1 or 2, further comprising: Parse the metadata, text fragments, and non-text elements corresponding to the target document; The multimodal language model is invoked to generate a summary of the element descriptions corresponding to the non-text elements; The metadata, text fragments, and element description summaries are stored in a structured manner to construct or update the relational database.

6. The method according to claim 5, further comprising: Identify at least one coded column in the relational database; The coded column comprises multiple sequentially arranged table cells; For each of the coded columns, the cell content of each table cell in the coded column is vector-encoded and associated with the corresponding cell index to construct or update the vector database.

7. The method according to claim 6, further comprising: Extract the coordinates of the non-text element region; The coordinates of the element region are associated with the corresponding element description summary to update the relational database; The element encoding vector corresponding to the non-text element is determined based on the vector encoding model, and the corresponding element region coordinates are recorded in association to update the vector database.

8. A storage medium having a computer program stored thereon, wherein, When executed by a processor, the program implements the steps of the method described in any one of claims 1-7.

9. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1-7.

10. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-7.

Images