Multi-source complex table-oriented trusted question and answer agent construction method and system

Abstract

The invention discloses a multi-source complex table-oriented trusted question and answer agent construction method and system, a question and answer agent is constructed based on a ReAct framework, at least a code interpreter tool and a document reading tool are used in the ReAct framework, and the multi-source complex table-oriented trusted question and answer agent construction method comprises the following steps: constructing a multi-mode complex table analysis tool; the analysis module is used for analyzing a non-editable unstructured table file into an editable structured table text; constructing a single-mode complex table cleaning tool for performing abstract modeling, identification extraction and structure reconstruction on a complex data structure of the structured table text to generate a normalized table which is simpler than an original table in structure; and constructing an illusion detection and repair tool for performing real-time illusion verification on the output content of the question and answer agent, and triggering a fallback regeneration mechanism when illusion is detected. According to the invention, through tool set expansion and multi-tool cooperation, an efficient and credible table question and answer agent is constructed.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for constructing a trustworthy question-answering agent for complex multi-source tables. Background Technology

[0002] In the era of data-driven analytical decision-making, businesses and individuals have an ever-growing need for efficient analysis and processing of structured data. Spreadsheets (such as Excel and CSV files) serve as the core carrier of structured data, with extremely wide-ranging applications: they can precisely meet the data organization and calculation needs of individuals or families, and also support large, medium, and small enterprises in building large-scale data ecosystems. Leveraging the advanced features of professional software such as Excel and WPS, including pivot tables, formulas, functions, and VBA programming, in-depth and professional data analysis can be conducted using spreadsheets, meeting the demands of businesses and individuals for efficient processing of structured data.

[0003] While software like Excel offers convenience for analyzing and processing spreadsheet data, the use of advanced functions has a certain barrier to entry, requiring users to be proficient in various formulas and VBA syntax. The tedious and repetitive operations also lead to low efficiency in data analysis. Furthermore, the comprehensiveness of data analysis is limited by individual analytical thinking, making it difficult to fully uncover the hidden value within the data. In recent years, with the breakthroughs achieved by large-scale pre-trained language models (LLMs) in natural language understanding and reasoning, intelligent question-answering systems based on LLMs have provided the technological possibility for building lightweight spreadsheet data processing and analysis tools. After uploading spreadsheet files such as Excel, users can quickly perform operations and in-depth analysis of the spreadsheet data simply by asking questions through a dialog box. This changes the traditional interaction mode that relies on manual operation, lowers the barrier to entry, and improves the efficiency of data processing and analysis. In addition, LLMs can try hundreds of analytical dimensions and combinations in a short time, proactively discovering implicit correlations and deep attributions that are difficult for humans to detect. This allows data analysis systems to evolve from tools that require "passive operation" into collaborators that can "actively communicate, think, and offer suggestions."

[0004] Currently, existing intelligent question-answering systems based on tables mainly follow two technical solutions: Option 1: Intelligent Agent Solution. This solution is based on paradigms such as Reasoning-and-Action (ReAct) or Plan-and-Execute. After receiving user instructions, it performs task planning and reasoning through LLM and calls tools such as code interpreters to query and process data, thereby obtaining question answers or analysis conclusions.

[0005] Option 2: An end-to-end LLM generation or document reading solution. This solution uses tabular data as additional input, directly serializing the table and concatenating it with the user's question before inputting it into the LLM to obtain the answer or analysis conclusion. When the tabular data is too long and exceeds the maximum input limit of the LLM, the input range is further narrowed by retrieving relevant data fragments. Unlike the agent-based solution that uses external tools to process tabular data, this solution relies entirely on the model's own computational and reasoning capabilities, and the model's output efficiency is low when the input is too long.

[0006] Specifically, Solution 2 can be encapsulated as a document reading tool and integrated into Solution 1 for autonomous invocation: upon receiving the table and question, the agent can autonomously decide whether to generate code and call a code interpreter tool to process the table data, or to directly concatenate the table sequence and input it into the LLM to obtain the answer. This approach combines the advantages of both solutions. When the table data is small or the question is simple (such as a simple query), the document reading method is more efficient; when the table data is too long, the code generation and execution method is more efficient and provides more comprehensive data processing.

[0007] The highly flexible use of spreadsheets has also led to diverse data structures. Beyond the standard two-dimensional row-column structure, there are also tables that use merged cells to represent hierarchical relationships between headers and data rows, and summary rows to represent various computational logics between data rows. These tables are not merely data arrangements but also exhibit nested and interconnected structures, presenting multidimensional structures and features on a two-dimensional plane, termed "complex tables and complex data structures." Furthermore, tabular data is widely distributed in uneditable, unstructured files such as PDF documents and JPG / PNG / BMP images, exhibiting multimodal characteristics. Therefore, the two aforementioned solutions, as well as the integrated solution, still have significant shortcomings when dealing with complex, multimodal tabular question-answering scenarios, mainly in the following aspects: 1. Lack of deep understanding and reasoning regarding complex table data structures. Agent-based solutions typically guide the model to understand and simplify table structures through prompt word engineering; however, more complex tables often imply more complex prompt words. End-to-end generation or document reading solutions suffer from low understanding efficiency and easy loss of relationships when faced with complex tables. Some methods propose using quintuples to describe complex structures, but this can easily increase the understanding cost for the agent.

[0008] 2. Lack of accurate parsing of complex multimodal tables. Although existing research uses table parsing technology to convert tables (i.e., table images) in PDF documents, JPG / PNG / BMP files into editable Excel files (i.e., table text) and use them as a tool for intelligent agents to autonomously invoke, existing table parsing methods, whether designing multiple small models to separately recognize table structure and text and then combining them, or outputting the HTML sequence of tables end-to-end based on a large multimodal model, can only handle relatively simple tables. For complex tables in industry, there are still problems such as structural parsing errors and missing key data.

[0009] 3. Lack of effective control mechanisms for "model illusions" in LLM outputs. The probabilistic prediction-based output mechanism of large models can easily lead to agents generating content that is inconsistent with reality or context, i.e., model illusions. For example, the model may generate tool inputs (such as code) that do not match the actual data, resulting in execution errors or incorrect results. It may also output inference plans or conclusions that are inconsistent with the results of tool calls (such as running a code interpreter) or actual tabular data. Tabular data analysis often requires high accuracy and needs to produce accurate and reliable analytical results. Summary of the Invention

[0010] To overcome the shortcomings of the above-mentioned technologies, this invention provides a method and system for constructing a trustworthy question-answering agent for complex multi-source tables. This invention is an optimization and innovation based on the core technology framework of the ReAct agent. The ReAct framework provides a more flexible processing mechanism compared to Plan-and-Execute through an iterative loop of "Reasoning-Action-Observation," performing better on complex tasks and suitable as a basic paradigm for question answering complex tables. Building upon the ReAct framework, in addition to basic code interpreter tools and document reading tools, this invention addresses the challenges of multi-source complex tables and trustworthiness by systematically enhancing the technologies in key aspects of the ReAct framework through toolset expansion and multi-tool collaboration, thereby constructing an efficient and trustworthy table question-answering agent.

[0011] The technical solution adopted by this invention to overcome its technical problems is: A method for constructing a trustworthy question-answering agent for complex multi-source tables, wherein the question-answering agent is built based on the ReAct framework, which utilizes at least a code interpreter tool and a document reading tool. The method for constructing a trustworthy question-answering agent for complex multi-source tables includes: Build a multimodal complex table parsing tool to parse uneditable, unstructured table files into editable, structured table text; A single-modal complex table cleaning tool was developed to abstract, model, identify, extract, and reconstruct the complex data structure of structured table text, generating a normalized table that is simpler than the original table structure. A hallucination detection and repair tool is built to perform real-time hallucination verification on the output of the question-answering agent and trigger a rollback and regeneration mechanism when a hallucination is detected.

[0012] Preferably, the multimodal complex table parsing tool of this invention is a model fine-tuning-based multimodal complex table parsing tool, employing an end-to-end multimodal large model based on the Transformer architecture. This large multimodal model includes a visual encoder, an image-text mapping module, and a sequence decoder. The multimodal complex table parsing tool constructs an industry table dataset and efficiently fine-tunes some parameters of the large multimodal model, enhancing its ability to understand the structure and semantics of complex table images within the industry. Ultimately, this results in a table parsing tool that can unify the data modality of tables from different sources and can be autonomously invoked at the input side of the question-answering agent. The industry table dataset includes both real and synthetic data.

[0013] Furthermore, the tool for constructing multimodal complex table parsing specifically includes: Establish a labeled data format such as <image, HTML sequence>, where the image represents the original table image as input during fine-tuning of the multimodal large model, and the HTML sequence represents an editable HTML format sequence of the table as output during fine-tuning of the multimodal large model; Collect and process multimodal tabular data from various industry sectors, construct image and HTML sequence sample pairs, and form an industry tabular dataset. Editable table templates are extracted from collected real table data, and more complex tables than the original tables are synthesized through structural transformation operations to improve the industry table dataset. Freeze all parameters of the visual encoder in the multimodal large model, and update the parameters of the image-text mapping module and the sequence decoder; A multi-task training framework is adopted to jointly train a large multimodal model based on an improved industry table dataset.

[0014] Furthermore, the training tasks of the multi-task training framework include the following three types: 1) Task 1: Decode the original table image as input to generate the corresponding HTML sequence; 2) Task 2: Clear the text content in the table and keep only the structural information to obtain an empty table, and decode the image of the empty table to generate the corresponding HTML sequence; 3) Task 3: Perform Optical Character Recognition (OCR) on the text content in the table.

[0015] Task 1 is the main task, while Tasks 2 and 3 are auxiliary tasks. Through joint training of multiple tasks, while ensuring the analytical capabilities of the multimodal large model, the multimodal large model's ability to understand table structure and content is enhanced, thereby improving the analytical accuracy and generalization performance of the multimodal large model for complex multimodal table data.

[0016] Preferably, the single-modal complex table cleaning tool of this invention is a knowledge graph-based single-modal complex table cleaning tool. This tool targets structured table text after data modality unification, utilizing the powerful knowledge representation capabilities of knowledge graphs to abstractly model all possible complex structures in the table. For specific tables, it leverages the understanding capabilities of large-scale dialogue language models with larger parameters and stronger capabilities, as well as large-scale reasoning language models with numerical reasoning abilities, to identify and extract specific complex structures in the table, such as multi-level headers and multi-level data rows, and represents them specifically based on the knowledge graph. Finally, it restructures and organizes the original table according to the knowledge graph, generating a standardized table (such as a one-dimensional table or a two-dimensional table) with a simpler structure than the original table, providing reliable data preprocessing operations for subsequent task planning and tool invocation by the question-answering agent.

[0017] Furthermore, the construction of the single-modal complex table cleaning tool specifically includes: Summarize the structural features of different types of tables, predefine the knowledge graph hierarchy structure of "worksheet, table area, table header and data row, data cell", establish graph table nodes, sub-table nodes, table header nodes and data row nodes respectively, and add corresponding node descriptions for different hierarchical relationships and calculation logic; Based on a predefined knowledge graph hierarchy, the specific structural information of tables is identified and extracted by combining large-scale models of conversational and reasoning languages, and knowledge graph instances of corresponding worksheets are generated. Based on the knowledge graph instance, a large dialogue language model is invoked, and a hierarchical flattening strategy is adopted to restructure and organize the original table, ultimately generating a simple and standardized one-dimensional or two-dimensional table with complete semantic logic, which helps the question-answering agent understand and process the post-question-answering link.

[0018] Preferably, the hallucination detection and repair tool of this invention is a knowledge distillation-based hallucination detection and repair tool. This tool uses knowledge distillation technology to compress a large dialogue language model with hallucination detection capabilities into a lightweight hallucination monitor. This ensures the discrimination capability of the large dialogue language model while reducing its computational cost and latency. Simultaneously, the lightweight hallucination monitor is embedded in the output link of the question-answering agent. Without interfering with the original parameters of the large dialogue language model, it performs real-time verification of its output content, identifying anomalies such as inconsistencies in context or input-output discrepancies. When an anomaly is detected, a rollback and regeneration mechanism is triggered to repair the hallucination, thereby suppressing error propagation during the execution of the question-answering agent and improving the overall credibility of the agent's output.

[0019] Furthermore, the construction of the hallucination detection and repair tool specifically includes: A large-scale conversational language model with hallucination detection capabilities is used as the teacher model. A discriminative dataset is constructed as the training sample for knowledge distillation. Each data sample in the discriminative dataset includes LLM input, LLM output, and hallucination confidence score. During distillation training, the LLM input and LLM output are concatenated into a sequence and input into the student model to train the student model to fit the hallucination confidence score of the teacher model. The core of knowledge distillation training is to enable the student model with fewer parameters to learn to fit the hallucination judgment results of the teacher model with more parameters and stronger capabilities, thereby obtaining a lightweight hallucination monitor. Embed the hallucination monitor into the output link of the question-answering agent for real-time hallucination detection; When the output of the question-answering agent is detected to be illusory, a rollback and regeneration mechanism is triggered to regenerate the output content.

[0020] Preferably, in this invention, the LLM input refers to a user-uploaded tabular data fragment or the output result after being invoked by a question-and-answer intelligent agent tool, such as the result of a code interpreter running. The LLM output refers to the textual content (i.e., the planning text or summary text output by the large model in the question-answering agent) or the parameter content (i.e., the input parameters required by the large model in the question-answering agent during the tool invocation phase) generated by the question-answering agent. The large model used by the question-answering agent is usually a large model of conversational language. The hallucination confidence score refers to a score in the range of 0 to 1 output by the teacher model, where 1 indicates that there is no hallucination and 0 indicates that there is a hallucination. Compared with directly outputting 0 or 1, the score is a more refined "soft label" that represents the uncertainty in the teacher model's judgment.

[0021] Preferably, in this invention, several hallucination monitors are embedded in the output chain of the question-answering agent. The hallucination monitors are generally embedded in key output nodes that are prone to hallucination, including at least the task planning stage, the tool invocation stage, and the conclusion generation stage.

[0022] This invention also discloses a trusted question-answering agent system for multi-source complex tables. The question-answering agent is built based on the ReAct framework. The trusted question-answering agent system for multi-source complex tables also includes: A multimodal complex table parsing tool used to parse uneditable, unstructured table files into editable, structured table text; A single-modal complex table cleaning tool is used to abstract, model, identify, extract, and reconstruct the complex data structure of structured table text, generating a normalized table that is simpler than the original table structure. The hallucination detection and repair tool is used to perform real-time hallucination verification on the output of the question-answering agent and trigger a rollback and regeneration mechanism when a hallucination is detected.

[0023] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for constructing a trustworthy question-answering agent for multi-source complex tables as described above.

[0024] The present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for constructing a trustworthy question-answering agent for multi-source complex tables as described above.

[0025] The beneficial effects of this invention are: 1. Compared to existing table-based question-answering agents, this invention innovatively introduces a multimodal complex table parsing tool, a unimodal complex table cleaning tool, and an illusion detection and repair tool into the ReAct framework. The multimodal complex table parsing tool parses uneditable, unstructured table files into editable, structured table text. The unimodal complex table cleaning tool abstracts, models, identifies, extracts, and reconstructs the complex data structure of the structured table text, generating a standardized table with a simpler structure than the original. This significantly improves the question-answering agent's understanding and reasoning ability regarding tables from multiple sources (such as images and text) and with complex data structures. Furthermore, the illusion detection and repair tool effectively controls illusion problems in the agent's output, thereby improving the overall efficiency, accuracy, and reliability of the agent's table-based question-answering capabilities.

[0026] 2. The application of this invention facilitates the automated processing and question-and-answer analysis of complex multi-source tabular data within the industry, significantly reducing manual operation costs and the technical threshold for data analysis, and providing efficient and reliable data decision support for diverse business scenarios such as market operations, engineering cost estimation, and financial management. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating the principle of a table-based question-answering intelligent agent system in the prior art.

[0028] Figure 2 This is a schematic diagram illustrating the principle of the trusted question-answering intelligent agent system for multi-source complex tables as described in an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the basic architecture and training task of the multimodal large model in the multimodal complex table parsing tool described in the embodiments of the present invention.

[0030] Figure 4 This is a schematic diagram of a complex table type according to an embodiment of the present invention.

[0031] Figure 5 For knowledge graph-based representation Figure 4 A schematic diagram of the complex data structure of the table.

[0032] Figure 6 for Figure 4 A simplified table diagram after cleaning the complex table. Detailed Implementation

[0033] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following are merely exemplary and do not limit the scope of protection of the present invention.

[0034] The prior art discloses a table-based question-answering intelligent agent system, which is built on the ReAct framework, such as... Figure 1As shown, the question-answering agent first receives an Excel or CSV file (file extensions such as .xlsx, .xls, or .csv) uploaded by the user, along with processing instructions or questions for the table. Then, based on the question and a preliminary understanding of the table (e.g., reading the first few rows of data), it performs task planning, determining the initial steps needed to solve the problem, and clarifying ambiguous user intentions through follow-up questions if necessary. Each step involves the large-scale dialogue language model determining whether to invoke a tool to solve the current problem: if so, it autonomously selects a suitable tool (e.g., a code interpreter) from the toolkit and generates the necessary input parameters (e.g., code), then runs the tool to obtain the execution result (e.g., code execution result); if not, the large-scale dialogue language model reasons and considers how to solve the current problem. Finally, the result of the tool execution or the result of the dialogue language model's reasoning returns to the main process, where the dialogue language model determines whether the user's problem has been solved. If solved, a conclusion is output; otherwise, task planning continues to the next step. Through the iterative loop mechanism of "reasoning (planning) - action (a certain step) - result" described above, intelligent question answering based on tabular data is achieved. Besides code interpreter tools and document reading tools, common external tools for question answering agents based on the ReAct framework include web search, knowledge base search, and various APIs.

[0035] This invention provides a method and system for constructing a trustworthy question-answering agent for complex multi-source tables, such as... Figure 2 As shown, based on the ReAct framework-based question-answering agent, three tools were constructed: a multimodal complex table parsing tool, a single-modal complex table cleaning tool, and an illusion detection and repair tool. These tools address the problems of existing table question-answering agents' inaccurate understanding of multimodal and complex structured tables in the industry, as well as the lack of detection and control of "illusions" on the output side. Figure 2 In Chinese, "table parsing" is short for multimodal complex table parsing tools, and "table cleaning" is short for single-modal complex table cleaning tools. Specifically, it includes the following technical solutions: S1. Build a multimodal complex table parsing tool to parse uneditable unstructured table files into editable structured table text.

[0036] Specifically, in this embodiment, a multimodal complex table parsing tool is constructed based on efficient fine-tuning of a large multimodal model. For example... Figure 2 As shown, the multimodal complex table parsing tool will be automatically invoked when the question-answering agent receives a multimodal table file. It is used to convert table images in PDF documents or images (at least including JPG, PNG, BMP) into editable Excel text, enabling the question-answering agent to be compatible with table data from different sources and in different modalities.

[0037] like Figure 3 As shown, this embodiment adopts an end-to-end multimodal large model based on the Transformer architecture. The multimodal large model is based on the visual Transformer to encode the global features of the table image in one go, i.e., the visual encoder. Then, the encoded high-dimensional features are nonlinearly transformed by the multilayer perceptron (MLP), i.e. the image-text mapping module. Finally, the input language Transformer decodes and outputs the editable HTML sequence corresponding to the table, i.e. the sequence decoder, and then automatically converted into an Excel file by the conversion function.

[0038] In this embodiment, the construction of the multimodal complex table parsing tool specifically includes: S11. Establish a labeled data format such as <image, HTML sequence>, which is the training data format for the multimodal large model. Here, the image represents the original table image, which is used as the input when fine-tuning the multimodal large model, and the HTML sequence represents the editable HTML format sequence of the table, which is used as the output when fine-tuning the multimodal large model.

[0039] S12. Collect and process multimodal tabular data from various industry sectors. Specifically, convert the multimodal tabular data into images and editable HTML sequences to construct training sample pairs of <images, HTML sequences>, forming an industry tabular dataset. In this embodiment, the multimodal tabular data specifically includes: A) Collect tabular data used in actual production processes within the industry sector, especially tables with complex data structures, including non-editable PDF documents, JPG, PNG, or BMP image files, and editable Excel and CSV files; among which: a) For editable spreadsheet files (such as Excel and CSV), use the built-in export function of office software (such as Excel and WPS) to batch convert them into images and HTML sequences; b) For non-editable tabular data (such as PDF documents, JPG or PNG or BMP images), first convert them into image formats, then use existing open-source parsing methods to identify and parse the tables in the images to generate an initial HTML sequence, and then have it manually verified and corrected by professionals to finally form an accurate HTML sequence annotation. B) Collect labeled <images, HTML sequences> samples from various open-source evaluation datasets to further supplement and fine-tune the dataset. The open-source evaluation datasets include at least the tabular data from MMTab, IWuDao Financial Annual Report, and arXiv papers.

[0040] S13. Based on the collected real table data, extract editable table templates from the table HTML sequences, and synthesize tables that are more complex than the original tables through structural transformation operations. In this embodiment, structural transformation operations include merging cells, splitting cells, etc. The synthesized table will generate corresponding image data, constructing <image, HTML sequence> sample pairs to improve the industry training dataset.

[0041] S14. Set the trainable parameters of the multimodal large model, such as... Figure 3 As shown, in this embodiment, all parameters of the visual encoder in the multimodal large model are frozen, and only the parameters of the image-text mapping module and the sequence decoder are updated.

[0042] S15. A multi-task training framework is adopted to jointly train the multimodal large model based on the improved industry table dataset. For example... Figure 3 As shown, in this embodiment, the training tasks of the multi-task training framework include the following three types: 1) Task 1: Table HTML Sequence Generation. Input the original table image and generate a complete HTML sequence containing all text content and structural tags using the multimodal large model. This task will serve as the main training objective for optimizing and fine-tuning the multimodal large model.

[0043] 2) Task 2: Table skeleton structure prediction. Remove the text content from the table and input it as an image into the multimodal large model. The model is required to predict and generate the corresponding empty table HTML sequence (i.e., the table skeleton information). This task will serve as an auxiliary training objective to enhance the multimodal large model's ability to understand table structure.

[0044] 3) Task 3: Table Text OCR. The multimodal large model is required to recognize and output all the text content contained in the table in row and column order. This task will serve as an auxiliary training objective to improve the recognition accuracy of the multimodal large model for text within the table.

[0045] The three training tasks work together to ensure the general recognition capability of the multimodal large model and promote its fine-tuned table parsing capability.

[0046] S2. Construct a single-modal complex table cleaning tool to abstract, model, identify, extract, and reconstruct the complex data structure of structured table text, generating a normalized table that is simpler than the original table structure.

[0047] Specifically, in this embodiment, a single-modal complex table cleaning tool is constructed based on knowledge graph technology. For example... Figure 2As shown, the single-modal complex table cleaning tool will be invoked by the question-answering agent after receiving a complex text table (an Excel file directly uploaded by the user or the result of parsing an image table uploaded by the user). The tool will preprocess the table, organizing the complex table data with complex data structure and free format into simple tables with simple data structure and standardized format (such as one-dimensional tables and two-dimensional tables). This will help the question-answering agent better understand the relationship between the data in the table and better execute subsequent table-based task planning and tool invocation (such as code interpreters).

[0048] In this embodiment, the construction of the single-modal complex table cleaning tool specifically includes: S21. Summarize the structural characteristics of different types of tables, predefine the knowledge graph hierarchy of "worksheet (i.e., sheet page), table area, table header and data row, and data cell", and establish graph table nodes, sub-table nodes, table header nodes, and data row nodes respectively, adding corresponding node descriptions for different hierarchical relationships and calculation logic. This embodiment will be explained with specific table examples, such as... Figure 4 The diagram shown is a schematic of a complex table type as described in this embodiment. Since this embodiment is only intended to demonstrate the conversion of a complex table structure into a simple table structure, ... Figure 4 No specific data was filled in. Specifically, this includes: a) Type 1 ( Figure 4 (Blue highlighter): This refers to situations where multiple data ranges exist within the same worksheet, i.e., sub-tables, for example... Figure 4 There are two sub-tables. First, the worksheet (i.e., the page) is used as the root node, and then the sub-table nodes are set according to the data range.

[0049] b) Type II Figure 4 (High-brown highlighted section): For tables where headers and data rows occupy multiple rows and columns, and the hierarchical relationship within the headers or data rows is represented by merged cells, multiple header nodes and data row nodes will be set up, and the hierarchical relationship between them will be established. The data relationships are typically hierarchical, such as... Figure 4 In this context, sales revenue = home appliances + clothing + pet supplies.

[0050] c) Type 3 ( Figure 4 (Highlighted in red): For operations such as addition, subtraction, multiplication, division, weighting, and percentage calculations between data rows, a hierarchical relationship is also established between data row nodes. This is further differentiated by noting the calculation logic of the corresponding child nodes in the parent node's annotation. For example... Figure 4 In this context, profit = sales revenue - cost items, and cost items = advertising expenditure + inventory loss.

[0051] d) Type Four ( Figure 4(Green indicator): When dealing with descriptive text content such as table titles and metadata, a descriptive attribute is added to the sub-table node for representation.

[0052] S22. Based on a predefined knowledge graph hierarchy, and combining large-scale models of conversational and reasoning languages, identify and extract specific structural information of tables, and integrate them to generate knowledge graph instances for corresponding worksheets, such as... Figure 5 As shown, this is a knowledge graph-based representation. Figure 4 A schematic diagram of the complex data structure of the table. Figure 5 The colors in Figure 4 The colors in the text correspond to specific meanings. Due to space limitations, Figure 5 Only the knowledge graph of sub-table 1 is shown. Specifically, it includes: For the graph hierarchy structures of Type 1, Type 2 and Type 4 mentioned above, this embodiment will adopt a more efficient large-scale dialogue language model. Taking the HTML sequence of the table as input, it will locate different data areas respectively, output the row and column coordinates of the four corner boundaries of each sub-table, and identify the data hierarchy relationship represented by merged cells. When the sub-table contains descriptive text content such as titles and meta-information, it will be extracted as descriptive attributes of the sub-table nodes.

[0053] In practical implementation, the data structures corresponding to Type 1 and Type 4 are uniformly extracted using the instructions (instructions are prompts given to the large model to complete the corresponding tasks) provided in one of the optional examples below. After recognition, the original table is divided according to the coordinates of the four corner boundaries to obtain sub-table data, and subsequent processing is performed based on the sub-tables. It should be noted that in this embodiment, the instructions (prompts) adopt Markdown format; for example, the number of "#" indicates different heading levels, and "-" indicates that the data is an unordered list.

[0054] ## Character Setting You are a senior Excel structure analysis expert, specializing in analyzing complex worksheet layouts based on HTML sequences. Your core competency lies in identifying multiple independent data subsets (sub-tables) within the same worksheet and accurately extracting their boundaries and associated text information.

[0055] ## Table HTML Sequence (

[0056]

[0057]

[0058]

[0059]

[0060]

[0061]

[0062]

[0063] (Elements and internal structure) {table_data} ## Core Tasks Please analyze the Excel worksheet sequence provided above and complete the following two core tasks according to the following steps: 1. Identification and Location: Find all independent sub-tables in the worksheet and output their precise cell coordinate range; 2. Content Analysis: Separate and extract the data area and descriptive attributes (title, metadata, etc.) for each sub-table. ### Step 1: Scanning and Identifying Sub-tables You need to divide the worksheet into multiple logically independent sub-table units based on the following characteristics: #### Characteristic 1: Physical Separation Markers: - Empty Rows / Columns: Areas significantly separated by consecutive blank rows or columns (usually ≥1 row / column); - Border Variations: Areas with independent outer borders or borders disconnected from surrounding areas. #### Feature 2: Logical Structure Markers: - Independent Header: Has its own independent header row or column, and is not continuous with the header content of adjacent areas; - Abrupt Change in Data Pattern: The number of columns and data types (e.g., a sudden change from numerical tables to text descriptions) of adjacent areas undergo a fundamental change; - Summary Row Identifier: Ends with "Subtotal," "Total," "Grand Total," etc., and begins a completely new topic after it. ### Step 2: Extract Sub-Table Nodes and Description Attributes For each sub-table within the identified boundaries, perform internal structure analysis to distinguish between the "Description Area" and the "Data Area": ​​- Description Area: Usually located at the top or left of the sub-table, containing the following: - Main title, subtitle; - Meta-information such as table creation time, unit, and compiler; - Explanatory paragraphs for the data below; - Characteristics: Mostly merged cells, text format, and no strict row and column data logic. - Data Area: - Has a clear row and column structure (the first row is often the column header, and the first column is often the row identifier); - The core numerical and text data body; - May contain internal "total" rows. ### Step 3: Please strictly follow the following JSON format to output a structured object for each sub-table node: - Output format: {"sub_tables: [{"subtable_id": ST1, / / Assign a unique ID to each identified sub-table (e.g., ST1, ST2) "boundary": (start row, end row, start column, end column), / / Four corner boundary coordinates "description": "Meta-information text", / / Combine all text content in the description area into a concise descriptive text. If there is no obvious description area, leave it blank or provide a brief summary based on the first row header of the data area.} / / ... (Other sub-tables)]}### Special Case Handling Guidelines - Nested Tables: If a large table contains completely independent small tables (such as a summary table with a comment table), recognize them as two sub-tables; - Single Row / Single Column: If a row or column of data is logically a self-contained unit (such as an independent summary conclusion row) and is surrounded by whitespace, it should also be recognized as a sub-table; - Uncertain: If you encounter ambiguous boundaries, give your best judgment and a note. The data structure corresponding to Type 2 can be extracted using the following optional example instruction: ## Role Setting You are an Excel table structure analysis expert, specializing in parsing complex tables with merged cells (rowspan, colspan) to extract the hierarchical relationships and organizational structure between data. ## Table HTML Sequence (.

[0064] (Elements and internal structure) {table_data} ## Core Task: Based on the HTML table sequence provided above, follow these steps to extract the hierarchical relationship between the header and data rows from the merged cells and construct a structured representation. ### Step 1: Grid Reconstruction Analysis

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078] Figure 6

[0079]

[0080]

[0081]

[0082]

[0083] Figure 2

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] Step 1: Expand rowspan and colspan to construct a two-dimensional cell grid, recording the original content and span of each cell. Cells with rowspan > 1 are marked as occupying multiple rows vertically, and cells with colspan > 1 are marked as occupying multiple columns horizontally. Step 2: Extract the column hierarchy (header tree) to analyze the top rows of the table (usually the header). Use a top-down approach to analyze the column hierarchy, where a cell spanning multiple columns (colspan > 1) is the parent node of the cells below it. Step 3: Extract the row hierarchy (grouping tree) to analyze the rows of the data area. Use a left-to-right approach to analyze the row hierarchy, where a cell spanning multiple rows on the left (rowspan > 1) is the grouping label of the row to its right. Step 4: Establish mappings for key data cells, describing their complete column path (from root category to specific field) and row path (from top-level grouping to the current row). ### Step 5: Please strictly follow the following JSON format for output: {"columns_group": [{"parent": "cross-column grouping label", / / generally the header "children": ["child node column 1", "child node column 2", ...]}],"row_group":[{"parent": "cross-row grouping label", / / generally the data row "children": ["child node row 1", "child node row 2", ...]}]} For various operational logics existing in the three-graph hierarchical structure of the aforementioned type, such as using identifiers like ordinal numbers, "subtotal", and "total" to describe the total score relationship between data rows, not using obvious identifiers to represent the total score relationship between data, or having complex logical operations such as addition, subtraction, multiplication, division, weighting, average, and percentage between data, this embodiment will uniformly call a large-scale inference language model to derive numerical relationships and output the calculation logic; and when there are calculation formulas in the original data, the consistency of the data operation logic will be checked and verified in conjunction with the formulas. Here is an optional example of a specific instruction: ## Role Setting You are a data analysis expert, proficient in Excel data structure, formula logic, and numerical relationship derivation. Your task is to deeply analyze the data relationships in the table, identify explicit and implicit calculation relationships, and systematically verify the consistency of data logic. ## Table Data (and the formulas behind the data, if any) {table_data} ## Core Task Please analyze the Excel table data (or data fragments) provided above according to the following steps: ### Step 1: Understanding the Data Structure - Identify the table header fields, data range, and key columns (such as serial number, item name, numerical column, identifier column, etc.); - Locate all keywords that may identify summary relationships (such as "subtotal", "total", "sum", "cumulative", etc.); - Mark the cells or rows and columns that have clearly marked calculation formulas (e.g., containing formulas such as =SUM(), =A2+B2, etc.).### Step 2: Identify Numerical Relationship Types #### Type A: Explicit Total-to-Small Relationship - If a data row contains identifiers such as "subtotal" or "total," deduce its summation relationship with the corresponding detailed rows; - Output the corresponding calculation logic (e.g., "Total Sales = Sales Revenue of Product A + Sales Revenue of Product B + Sales Revenue of Product C"). #### Type B: Implicit Logical Relationship - If a data row has no explicit identifier, but there may be potential total-to-small, grouped summaries, or proportional relationships between the values, deduce their logic. For example: if a row's value equals the sum of the values ​​in the previous rows, but is not labeled "total," deduce the implicit summarization rule. #### Type C: Complex Operational Relationship - If there are complex operations such as addition, subtraction, multiplication, division, weighting, average, and percentage between the data, deduce their calculation formulas. For example: "Net Profit = Total Revenue - Total Cost - Taxes," or "Percentage = Partial Value / Total Value." ### Step 3: Logical Derivation and Verification #### Sub-step 3.1: Verification Based on Existing Formulas - If the original data provides a calculation formula, parse the formula and verify whether it matches the actual value. For example, if a cell's formula is =SUM(B2:B10), check whether the cell value is indeed equal to the sum of B2 to B10. #### Sub-step 3.2: Reasoning Without Formulas - If no formula is provided, construct possible calculation logic through numerical comparison and reasoning; - It is recommended to use the reverse reasoning method: try to derive the summary value from the detailed data, or deduce the components from the summary data. #### Sub-step 3.3: Consistency Verification - Compare whether the derived logic matches the actual data; - If inconsistencies are found (e.g., "Subtotal" value ≠ Sum of Details), clearly mark the abnormal location, the difference value, the possible cause, and the correction suggestions. ### Step 4: Based on the reasoning results from the above steps, output the logical calculation and analysis results: Please strictly adhere to the following JSON format for output: {"relations": [{"parent": "Parent node row","children": ["Child node row 1", "Child node row 2", ...],"expression": "Calculation logic expression (e.g., parent node = child node 1 + child node 2 +..).","consistency": "The model reasoning logic is consistent with the actual data logic | The model reasoning logic is inconsistent with the actual data logic, and the sum of the summary row values ​​and the sub-item sums are inconsistent. This may be due to an error in the formula. Please refer to the sub-item calculation results for subsequent processing."}]} Wherein, "table_data" represents the original table data and the corresponding HTML sequence. If there are formulas behind the data, the formulas will be noted in the form of "()" after the specific values. Integrate the JSON data output from the above steps and generate corresponding knowledge graph instances for each worksheet according to the abstract modeling logic of the knowledge graph.S23. Based on the knowledge graph instance described in step S22, the large-scale dialogic language model is invoked, and a hierarchical flattening strategy is adopted to restructure and organize the original table. Specifically, this includes: a) Connecting the multi-level table headers and data row content with hierarchical relationships in the knowledge graph using "_" symbols according to their hierarchical order to generate composite table header fields and data row fields, realizing the semantic integration and flattened representation of multi-level information. b) For data rows with calculation logic, the rows containing the calculation results (such as "subtotal" and "total") are removed from the original data, and natural language descriptions are generated for their corresponding non-leaf data row nodes to clarify their calculation logic and data relationships. c) For different sub-tables (i.e., different data areas) identified in the same worksheet, they are split into different independent worksheets. Finally, the complex data structure of the table is transformed into a standardized table with a simple data structure and complete semantics, such as a one-dimensional or two-dimensional table, as shown. In addition to the data, the table is accompanied by the following descriptive information: "The title of this table is <Revenue Data of Northbound Business in the Past 3 Years>, and there is a description <Unit: RMB 10,000>, where (1) Profit = Sales Amount - Cost Items; (2) Sales Amount = Home Appliances + Clothing + Pet Supplies; (3) Cost Items = Advertising Placement + Goods Loss". The prompt words for the dialogue language model to "flatten" the table based on the table structure graph are executed according to the following optional example: ## Role Setting You are a table structure flattening expert, responsible for reconstructing and organizing the original table based on the provided table structure knowledge graph using the "hierarchical relationship flattening" strategy, and converting the complex multi-dimensional table into a standardized table with a simple structure. ## Table Structure Knowledge Graph {table_KG} ## Original Table Data {table_data} ## Core Task: Please flatten the table according to the following steps: ### Step 1: Generate Composite Fields - Based on the table structure graph, connect the multi-level headers and multi-level data rows in the table according to their hierarchical order using "_"; - Method: Connect level by level from the root node to the leaf node; - Example: ["2024", "First Quarter", "January"] → "2024_First Quarter_January" ### Step 2: Process Calculation Result Rows - Only retain the data nodes in the table structure graph, and remove calculation result rows such as "Subtotal" and "Total" from the original table; - Generate descriptions for the removed calculation result rows (i.e., non-leaf nodes), and write their corresponding calculation logic in text form below the reconstructed table as explanatory information; - Example: "

Calculation Explanation

Claims

1. A method for constructing a trustworthy question-answering agent for complex multi-source tables, wherein the question-answering agent is built based on the ReAct framework, which uses at least a code interpreter tool and a document reading tool, characterized in that... Methods for constructing trustworthy question-answering agents for complex multi-source tables include: Build a multimodal complex table parsing tool to parse uneditable, unstructured table files into editable, structured table text; A single-modal complex table cleaning tool was developed to abstract, model, identify, extract, and reconstruct the complex data structure of structured table text, generating a normalized table that is simpler than the original table structure. A hallucination detection and repair tool is built to perform real-time hallucination verification on the output of the question-answering agent and trigger a rollback and regeneration mechanism when a hallucination is detected.

2. The method according to claim 1, characterized in that, The multimodal complex table parsing tool is built on a multimodal large model, which includes a visual encoder, an image-text mapping module, and a sequence decoder. The tool for constructing multimodal complex table parsing specifically includes: Establish a labeled data format such as <image, HTML sequence>, where image represents the original table image and HTML sequence represents an editable HTML format sequence of the table; Collect and process multimodal tabular data from various industry sectors, construct image and HTML sequence sample pairs, and form an industry tabular dataset. Editable table templates are extracted from collected real table data, and more complex tables than the original tables are synthesized through structural transformation operations to improve the industry table dataset. Freeze all parameters of the visual encoder in the multimodal large model, and update the parameters of the image-text mapping module and the sequence decoder; A multi-task training framework is adopted to jointly train a large multimodal model based on an improved industry table dataset.

3. The method according to claim 2, characterized in that, The training tasks of the multi-task training framework include: Generate the corresponding HTML sequence based on the input original table image; The text content in the table is cleared, leaving only the structural information to obtain an empty table, and the corresponding HTML sequence is generated based on the image of the empty table; Perform optical character recognition on the text content in the table.

4. The method according to claim 1, characterized in that, The single-modal complex table cleaning tool is built based on a knowledge graph; The tool for constructing a single-modal complex table cleaning system specifically includes: Summarize the structural features of different types of tables, predefine the knowledge graph hierarchy structure of "worksheet, table area, table header and data row, data cell", establish graph table nodes, sub-table nodes, table header nodes and data row nodes respectively, and add corresponding node descriptions for different hierarchical relationships and calculation logic; Based on a predefined knowledge graph hierarchy, the specific structural information of tables is identified and extracted by combining large-scale models of conversational and reasoning languages, and knowledge graph instances of corresponding worksheets are generated. Based on the knowledge graph instance, a large dialogue language model is invoked, and a hierarchical flattening strategy is used to restructure and organize the original table, ultimately generating a one-dimensional or two-dimensional standardized table.

5. The method according to any one of claims 1-4, characterized in that, The hallucination detection and repair tool is built based on knowledge distillation technology. The aforementioned hallucination detection and repair tool specifically includes: A large-scale conversational language model with hallucination detection capabilities is used as the teacher model. A discriminative dataset is constructed as the training sample for knowledge distillation. Each data sample in the discriminative dataset includes LLM input, LLM output, and hallucination confidence score. During distillation training, the LLM input and LLM output are concatenated into a sequence and input into the student model to train the student model to fit the hallucination confidence score of the teacher model, thereby obtaining a lightweight hallucination monitor. Embed the hallucination monitor into the output link of the question-answering agent for real-time hallucination detection; When the output of the question-answering agent is detected to be illusory, a rollback and regeneration mechanism is triggered to regenerate the output content.

6. The method according to claim 5, characterized in that, The LLM input refers to the table data fragments uploaded by the user or the output results after being invoked by the question-answering intelligent agent tool; The LLM output refers to the text-based or parameter-based content generated by the question-answering agent to detect whether hallucinations exist. The hallucination confidence score refers to the score in the range of 0 to 1 output by the teacher model, where 1 indicates that there is no hallucination and 0 indicates that there is a hallucination.

7. The method according to claim 5, characterized in that, Several illusion monitors are embedded in the output chain of the question-answering agent. The locations where the illusion monitors are embedded include at least the task planning stage, the tool invocation stage, and the conclusion generation stage.

8. A trusted question-answering agent system for multi-source complex tables, wherein the question-answering agent is built based on the ReAct framework, characterized in that, Trusted question-answering agent systems for complex multi-source tables also include: A multimodal complex table parsing tool used to parse uneditable, unstructured table files into editable, structured table text; A single-modal complex table cleaning tool is used to abstract, model, identify, extract, and reconstruct the complex data structure of structured table text, generating a normalized table that is simpler than the original table structure. The hallucination detection and repair tool is used to perform real-time hallucination verification on the output of the question-answering agent and trigger a rollback and regeneration mechanism when a hallucination is detected.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes a computer program, it implements the method as described in any one of claims 1-7.

Images