Table data processing method and system based on knowledge graph and large language model

Abstract

The invention discloses a table data processing method and system based on a knowledge graph and a large language model. The method comprises the following steps: performing structured analysis on an original table, and identifying, filling and merging cells to generate a complete structured record; an enhanced knowledge graph is constructed based on the record, an atomic problem library associated with graph nodes is established, and semantic abstract texts are generated by the nodes and associated edges of the knowledge graph through a large language model; when a user question is responded, background information is generated based on the knowledge graph and directly answered, if the information is insufficient, the question is decomposed into atomic sub-questions, and after an atomic question library is matched, a final answer is generated based on correlation graph node reasoning. According to the method, the problem of data missing of a table containing merged cells is effectively solved, and the accuracy, interpretability and illusion resistance of table questions and answers are remarkably improved through double mechanisms of the semantic enhanced knowledge graph and the atomic question library.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and data processing technology, and more specifically, to a method and system for processing tabular data based on knowledge graphs and large language models. Background Technology

[0002] In numerous fields such as business operations, financial analysis, and scientific research management, large amounts of structured or semi-structured data exist in the form of spreadsheets (such as Excel and CSV files). How to efficiently and accurately extract information from these spreadsheets and interact with them using natural language question answering (Table Question Answering, Table QA) has become a key requirement for human-computer interaction and intelligent decision support.

[0003] Existing table-based question answering technologies mainly fall into two categories. The first category is based on end-to-end deep learning models, such as serializing the table and inputting it into a Large Language Model (LLM) to directly generate answers. While this type of method performs reasonably well for simple queries, it has significant drawbacks: First, for complex tables containing merged cells, the model struggles to correctly understand the semantic attribution of the data. Merged cells result in numerous empty data positions, preventing the model from establishing correct row and column correspondences, thus leading to incorrect factual associations. Second, this type of method lacks interpretability; its answering process is like a "black box," where users cannot trace the specific source of the answer or the reasoning logic, which is unacceptable in fields with stringent accuracy requirements such as finance and healthcare.

[0004] The second type of approach attempts to introduce knowledge graphs (KG) to enhance the structured representation of tabular data. The basic idea is to transform the rows, columns, or cells of a table into nodes and edges in a graph. However, existing technologies still have significant shortcomings in constructing such graphs. On the one hand, during the preprocessing stage, they typically assume the table is regular and fail to effectively fill and parse merged cells, resulting in data gaps and semantic breaks in the constructed graph from the outset. On the other hand, the constructed knowledge graphs often only contain raw numerical values ​​or text, lacking rich semantic descriptions. The relationships between nodes and edges are too simplistic (e.g., only recording "Product A - Sales Revenue - 1 million"), failing to leverage the capabilities of large language models to inject contextual semantics (e.g., "1 million is the total sales revenue of Product A in the first quarter of 2024"), which greatly limits the understanding depth and generalization ability of subsequent question-answering systems.

[0005] More importantly, regardless of the method described above, when dealing with complex user questions (such as those involving multi-hop reasoning, comparison, or aggregation), there is a general lack of a reliable, verifiable unit of fact as the basis for reasoning. When large language models encounter queries with insufficient or ambiguous information, they are prone to "hallucination," i.e., fabricating seemingly reasonable but actually incorrect answers. Existing technologies fail to provide a mechanism to decompose complex user questions into a series of independently verifiable atomic questions that strictly correspond to tabular facts, thereby constraining the model's generation process and ensuring the accuracy and reliability of the final answer.

[0006] In summary, the existing technology faces at least three interrelated technical problems that urgently need to be addressed: (1) It cannot effectively process original tables containing merged cells, resulting in incomplete structured data records; (2) The constructed knowledge graph is semantically poor and lacks deep semantic descriptions of nodes and edges enhanced by large language models; (3) The lack of an atomic question library closely tied to the facts in the table makes it impossible to provide an interpretable and verifiable reasoning path for complex question answering. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this application provides a method and system for tabular data processing based on knowledge graphs and large language models.

[0008] The first aspect of this application provides a method for processing tabular data based on knowledge graphs and large language models, comprising the following steps: The original table is parsed in a structured manner to identify merged cells in the original table. The contents of the merged cells are then filled into all the missing data positions within the area covered by the merged cells to generate multiple structured records. Each structured record contains the values ​​corresponding to each data column of the table. A knowledge graph is constructed based on multiple structured records, and an atomic question database is established, including: The values ​​of each data column in each structured record are used as graph nodes, and a language model is called to generate semantic summary text describing the graph nodes. Based on the header structure relationship between each data column, establish the associated edges between graph nodes, and generate semantic summary text of the associated edges based on the structured records corresponding to the graph nodes; Storing graph nodes, associated edges, and corresponding semantic summary text in a graph database forms a knowledge graph; For each structured record, the language model is invoked to generate one or more atomic problems involving the structured record, and the atomic problems are associated with the graph nodes corresponding to the structured records and stored in the atomic problem library; When responding to user questions, question-answering processing is performed based on the knowledge graph and atomic question library: Retrieve graph nodes and associated edges related to the user's question, and generate optimal background information based on the graph nodes and associated edges; The large language model is invoked to generate answers to user questions based on optimal background information; If the answer contains a numerical value, name, or status information corresponding to the user's question, then the answer is output; If the answer contains preset keywords indicating missing or uncertain information, the user's question is broken down into atomic sub-questions. These sub-questions are then matched with atomic questions in the atomic question library. Finally, reasoning is performed based on the graph nodes associated with the successfully matched atomic questions to generate the final answer.

[0009] This application ensures the integrity of structured data by accurately filling merged cells; secondly, it uses a large language model to inject deep semantics into the nodes and edges of the knowledge graph, constructing an enhanced knowledge graph rich in business meaning; finally, by constructing an atomic question library strictly bound to the graph nodes and adopting a dual-path question-answering mechanism of "direct answering" and "decomposition-matching-reasoning", it effectively constrains the generation behavior of the large language model and significantly improves the accuracy, robustness and interpretability of complex table question answering.

[0010] A second aspect of this application provides a tabular data processing system based on knowledge graphs and large language models, comprising: The table parsing module is used to perform structured parsing on the original table, identify merged cells in the original table, and fill the content of the merged cells into all the empty data positions in the area covered by the merged cells to generate multiple structured records. Each structured record contains the values ​​corresponding to each data column of the table. The knowledge graph and atomic problem library building module is used for: The values ​​of each data column in each structured record are used as graph nodes, and a language model is called to generate semantic summary text describing the graph nodes. Based on the header structure relationship between each data column, establish the associated edges between graph nodes, and generate semantic summary text of the associated edges based on the structured records corresponding to the graph nodes; Storing graph nodes, associated edges, and corresponding semantic summary text in a graph database forms a knowledge graph; For each structured record, the language model is invoked to generate one or more atomic problems involving the structured record, and the atomic problems are associated with the graph nodes corresponding to the structured records and stored in the atomic problem library; The question-and-answer processing module is used to respond to user questions and perform the following operations: Retrieve graph nodes and associated edges related to the user's question, and generate optimal background information based on the graph nodes and associated edges; The language model is invoked to generate answers to user questions based on optimal contextual information. If the answer contains a numerical value, name, or status information corresponding to the user's question, then the answer is output; If the answer contains preset keywords indicating missing or uncertain information, the user's question is broken down into atomic sub-questions. These sub-questions are then matched with atomic questions in the atomic question library. Finally, reasoning is performed based on the graph nodes associated with the successfully matched atomic questions to generate the final answer.

[0011] Through modular design, the system solidifies the method of this application into a deployable software architecture, which is easy to integrate into various data analysis platforms or intelligent customer service systems, providing stable and efficient intelligent question and answer services. Attached Figure Description

[0012] To more clearly illustrate the technical solution of the present invention, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is an overall flowchart of a table data processing method based on knowledge graphs and large language models provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the process of performing structured parsing on an original table containing vertical, horizontal, and block-merged cells and generating multiple structured records in one embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the process of constructing an enhanced knowledge graph and establishing an atomic question library based on structured records in one embodiment of the present invention; Figure 4 This is a flowchart illustrating the dual-path logic of question-answering processing based on an enhanced knowledge graph and an atomic question base in one embodiment of the present invention. Figure 5 This is a structural block diagram of a tabular data processing system based on knowledge graphs and large language models, provided in an embodiment of the present invention. Detailed Implementation

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, a brief introduction to this application will be given below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the descriptions of these embodiments are intended to help understand this application, but do not constitute a limitation on this application.

[0014] Please see Figure 1 In the tabular data processing method provided in this embodiment of the invention, the following steps are first performed: S1: Perform structured parsing on the original table, identify merged cells in the original table, and fill the content of the merged cells into all empty data positions within the area covered by the merged cells to generate multiple structured records.

[0015] Specifically, in step S1, the system receives a raw table as input. This raw table can originate from an Excel file (.xlsx or .xls format), a CSV file, an HTML webpage table, or a table structure extracted from a PDF document using an OCR engine. Regardless of the source, the system will invoke the underlying table recognition algorithm to parse its cell layout, row and column coordinates, and content attributes to obtain complete table structure information.

[0016] Please refer to the following: Figure 2 . like Figure 2 As shown on the left, the original table is a typical business report—a "Quarterly Product Sales Statistics Table." Its header uses a multi-level structure: the first row is the year "2024," and the second row is the quarter "Q1," "Q2," "Q3," and "Q4." In the data area, "Product A" is vertically merged in rows 4–7, column 1, and "Product B" is vertically merged in rows 8–11, column 1. This merging operation results in the cells within the merged area (such as A3 and A4) being empty in the original data.

[0017] To address this issue, the system first accurately identifies all merged cells in step S1. For Excel or HTML tables, this is achieved by reading their built-in merge attributes (such as Excel's MergeCells tag or HTML's colspan / rowspan attributes); for tables output by OCR, the merge status is inferred by analyzing the geometric coordinates and adjacency relationships of the cells.

[0018] Subsequently, the system performs a content filling operation: for each identified merged cell, its content (usually stored in the top-left cell of the merged area) is copied to all empty data positions within the entire rectangular area covered by that merged cell, excluding the original content location. For example: For the vertically merged "Product A" (covering B4–B7), the system will fill "Product A" into B4–B7 respectively; For the horizontally merged "2024" (covering D2–G2), the system will fill "2024" into D2, E2, F2, and G2 respectively.

[0019] After the above structured parsing and filling process, the original table is transformed into a well-organized data matrix without null values. The system traverses this matrix row by row, extracting each row of data as a structured record. The final generated structured records include: Record 1: ["Product A", "2024", "Q1", "1 million yuan"] Record 2: ["Product A", "2024", "Q2", "1.2 million yuan"] Record 3: ["Product B", "2024", "Q1", "800,000 RMB"] Record 4: ["Product B", "2024", "Q2", "900,000 RMB"] Each structured record contains complete values ​​corresponding to each data column in the table, and there is no longer any data loss due to merged cells. These high-quality structured records form a reliable data foundation for the subsequent construction of the knowledge graph and atomic question base.

[0020] S3. Construct a knowledge graph based on multiple structured records and establish an atomic question database.

[0021] In step S3, the system constructs a knowledge graph based on the multiple structured records generated in step S1. This step is the core of the invention, aiming to transform flat, tabular data into a structured knowledge representation rich in semantics and relationships. Figure 3 As shown, step S3 specifically includes the following sub-steps: S301: Take the values ​​of each data column in each structured record as graph nodes, and call the language model to generate semantic summary text describing the graph nodes.

[0022] The system iterates through each structured record and creates an independent graph node in memory for each data column value in the record (e.g., "Product A", "Q1 2024", "1 million yuan"). Each graph node initially contains only its original value, but to overcome the shortcomings of traditional knowledge graphs, such as the single semantic meaning of nodes and the lack of contextual understanding, this invention introduces a semantic enhancement mechanism.

[0023] Specifically, the system invokes a pre-trained Large Language Model (LLM) to construct a structured prompt by combining the value of the graph node with the column name of its corresponding data column. For example, for a field with the value "1 million yuan" and the column name "sales revenue", the system's constructed input context could be: "Column name: sales revenue; Value: 1 million yuan". Based on this context, and combined with its understanding of business terminology and numerical semantics, the LLM generates a concise, accurate natural language description that reflects the meaning of the node in a specific business scenario, serving as the semantic summary text for that graph node.

[0024] Typical output examples include: "This value represents the total sales revenue of product A in the first quarter of 2024, in RMB ten thousand." or "RMB 1 million is the sales revenue generated by product A during Q1 2024." Such summaries not only preserve the accuracy of the original data, but also supplement key contextual information such as time, subject, and unit of measurement, significantly improving the semantic richness and interpretability of the nodes.

[0025] The generated semantic summary text is then bound as a property to the corresponding graph node and stored in the graph database along with the node. Through this mechanism, each graph node is no longer an isolated data fragment, but a semantic entity carrying a complete business context, providing a high-quality semantic foundation for subsequent natural language-based retrieval, matching, and reasoning.

[0026] S303: Based on the header structure relationship between the data columns, establish the associated edges between the graph nodes, and generate the semantic summary text of the associated edges based on the structured records corresponding to the graph nodes.

[0027] While creating graph nodes, the system simultaneously analyzes the header structure of the original table to construct a knowledge graph with correct semantic topology. Specifically, the system parses the row structure of the header area to identify whether a multi-level header layout exists. A typical multi-level header is characterized by the first row containing parent group labels (such as "2024"), and the second row containing child field labels (such as "Q1", "Q2", "Q3", "Q4"), which visually and logically constitute a hierarchical grouping relationship.

[0028] The system automatically infers the hierarchical dependency structure between data columns by comparing the relationship between each column in different header rows. For example, if column C belongs to "2024" in the first header row and "Q1" in the second header row, the system determines that the column represents the composite semantic dimension of "2024 Q1".

[0029] Based on this hierarchical dependency, the system establishes association edges between graph nodes that originate from the same structured record and whose corresponding data columns are in the same logical group or have a parent-child hierarchical relationship. For example, in the record "Product A, 1 million yuan", an association edge will be established between the "Product A" node and the "1 million yuan" node, which implicitly expresses the fact that "Product A's sales in Q1 of 2024 were 1 million yuan".

[0030] To further enhance the semantic expressive power of the associated edges, the system again invokes the large language model, taking the values ​​of the two graph nodes connected by the associated edge, their corresponding column names, and the complete structured record to which they belong as context input, to generate a natural language description as the semantic summary text of the associated edge. For example, the model might output: "Product A's sales revenue in the first quarter of 2024 was 1 million RMB." The semantic summary text is then bound as an attribute of the associated edge and stored in the graph database along with the edge. Through this mechanism, each relationship in the knowledge graph not only records the connection between entities but also carries complete business semantics, significantly improving the graph's reasoning and interpretation capabilities in complex question-answering scenarios.

[0031] S305: Store the graph nodes, associated edges, and corresponding semantic summary text in a graph database to form the knowledge graph.

[0032] Before step S305, the system can further perform entity resolution: based on vector similarity calculation or predefined synonym rules, it identifies graph nodes with the same semantics but different expressions (such as "product A1" and "product A1"), merges them into a unified entity, and updates the relevant association edges. This operation significantly improves the consistency and query robustness of the knowledge graph.

[0033] In sub-step S305, the system not only stores the graph nodes and edges into the graph database, but also embeds each entity and relation text (including its semantic summary) into vectors (e.g., using the Sentence-BERT model), and stores the resulting vectors into a vector database (such as FAISS, Milvus).

[0034] Finally, after constructing the nodes and edges of a single structured record, the system batch-writes all newly created graph nodes, associated edges, and their corresponding semantic summary texts into a graph database (e.g., Neo4j, JanusGraph) in standard graph data formats (such as attribute graph models). Through this operation, all structured records are integrated into a unified, global, enhanced knowledge graph. This graph not only preserves the fact triples of the original tables but also achieves deep semantic enhancement through semantic summary text, providing a high-quality knowledge foundation for subsequent complex queries and reasoning.

[0035] S307. For each of the structured records, call the language model to generate one or more atomic problems involving the structured record, and store the atomic problems in the atomic problem library after associating them with the graph nodes corresponding to the structured records.

[0036] When processing the same structured record, the system simultaneously performs the construction of the atomic problem library. Specifically, the system uses all field contents of the structured record (e.g., "Product A", "Q1 2024", "Sales Revenue", "1 million RMB") as context input, calls the large language model, and prompts it to generate one or more atomic problems.

[0037] The "atomic questions" mentioned here refer to natural language questions that involve only a single, independent data fact. Their semantics focus on a specific information point within a structured record, without involving cross-record comparisons (e.g., "Is the sales revenue of product A higher than that of product B?"), numerical aggregation (e.g., "What was the total sales revenue in Q1 2024?"), or multi-hop logical reasoning (e.g., "Which product had the highest sales revenue in Q1 2024?"). Typical examples of atomic questions include: "What was the sales revenue of product A in the first quarter of 2024?", "To which business line did product A belong in Q1 2024?", etc. These questions are characterized by the fact that their answers can be directly and uniquely extracted from a single structured record, possessing clear factual boundaries and verifiability.

[0038] After receiving structured records, the large language model automatically generates several atomic questions that meet the above characteristics based on its understanding of natural language and tabular semantics. To ensure the diversity and coverage of the questions, the system can adopt a few-shot prompting strategy, embedding positive examples of multiple atomic questions in the prompt words to guide the model to output questions with consistent format and clear semantics.

[0039] After generating atomic problems, the system immediately performs an association and binding operation: establishing a precise index mapping relationship between each atomic problem and one or more graph nodes it depends on. For example, for the problem "What was the sales revenue of product A in Q1 2024?", the system will associate it with a graph node representing "product A", a time node representing "Q1 2024", and a numerical node representing "1 million yuan". This mapping can be achieved by storing unique identifiers (such as node IDs) of the relevant graph nodes in the atomic problem record.

[0040] Ultimately, all generated atomic questions and their corresponding graph node index mappings are uniformly written into a dedicated storage module (such as a relational database or document database), forming a structured atomic question library. This question library is not only a collection of questions, but also a queryable and traceable fact-question mapping network, providing crucial support for initiating reliable reasoning paths based on atomic questions in the subsequent question-answering stage when information is insufficient.

[0041] S5. When responding to user questions, perform question-answering processing based on the knowledge graph and atomic question library.

[0042] See Figure 4 The method provided in this application includes the following sub-steps: S501. Retrieve graph nodes and associated edges related to the user's question, and generate optimal background information based on the graph nodes and associated edges.

[0043] In step S501, the system responds to the user's natural language question by first performing a relevant knowledge retrieval operation, which aims to extract the most relevant contextual information to the user's question from the constructed enhanced knowledge graph.

[0044] In step S501, the "related knowledge retrieval operation" specifically includes: BM25 keyword recall (based on text fields) and vector similarity recall (retrieving the entity / relational vector that is most similar to the user's question embedding in the vector database), and the results of the two are fused and used for subgraph expansion.

[0045] Specifically, the system performs natural language processing on the user-input question text, including word segmentation, named entity recognition (NER), and keyword extraction. For example, for the question "What was the sales revenue of product A in Q1 2024?", the system can identify the key entities "product A", "Q1 2024", and the intent keyword "sales revenue".

[0046] Subsequently, based on this extracted information, the system performs a relevant subgraph retrieval in the graph database. In a preferred embodiment, this retrieval employs a hybrid strategy combining keyword matching and graph traversal: first, it quickly recalls graph nodes containing entities such as "Product A" or "Q1 2024" through keyword indexing; then, starting from these nodes, it performs a limited-depth neighborhood traversal (e.g., 1–2 hops) in the knowledge graph to obtain other nodes and related edges that are directly or indirectly associated with them. This hybrid strategy ensures both a high recall rate for explicit entities and the ability to uncover implicit semantic relationships, avoiding the omission of key background information.

[0047] Finally, the system combines all retrieved relevant graph nodes and their connected edges into a local subgraph, and further integrates the semantic summary text carried by each node and edge to generate a coherent and structured optimal background information. This background information serves as the context for the subsequent language model to generate the answer, and its quality directly determines the success rate of the first path (direct answer).

[0048] S503. Invoke the language model to generate the answer to the user's question based on the background information.

[0049] Specifically, the system uses a preset prompt template, placing background information in the context and user questions in the instruction position. For example, the prompt content can be organized as follows: "You are an intelligent form question-and-answer assistant. Please answer questions based only on the background information provided below, and do not fabricate content that is not mentioned."

[0050] Background information: [Insert a collection of semantic summary text generated by S501 here, such as "Product A's sales revenue in the first quarter of 2024 was RMB 1 million."] Question: What was the sales revenue of Product A in Q1 2024? answer:" By designing such restrictive prompts, the system guides the large language model to strictly rely on the retrieved facts to answer questions, effectively suppressing its "illusion" tendency. Based on this prompt, the large language model outputs a response in natural language form.

[0051] It's important to note that the goal of this step is not to force the model to provide a final answer, but rather to provide a direct response using existing background information. If the background information is sufficient and the question is simple (e.g., a single-fact query), the model can typically generate an accurate, complete, and valid answer. If the question is complex or the background information is incomplete (e.g., involving unretrieved records or multi-hop reasoning), the model may return a vague, uncertain, or explicitly "cannot answer" response. These different types of outputs will serve as the key basis for determining whether to initiate the second path (atomic problem decomposition) in the next step (S505).

[0052] S505. If the answer contains a numerical value, name, or status information corresponding to the user's question, then output the answer.

[0053] In step S505, the system performs a validity assessment on the response generated by the large language model in step S503 to determine whether to use the response as the final output.

[0054] Specifically, the system analyzes the answer text returned by the model to determine whether it constitutes a clear, complete, and context-informed affirmative answer. The criteria for judgment include, but are not limited to: (1) Whether the answer contains specific numerical values, names or states, rather than expressions such as "I don't know", "no relevant information provided", or "cannot be determined" that indicate missing or uncertain information; (2) Whether the answer content is semantically consistent with the optimal background information provided in step S501, and whether it introduces new facts not mentioned in the background; (3) Whether the answer directly addresses the core needs of the user's question without deviation or generalization.

[0055] If all the above conditions are met, the system determines the answer to be valid. At this point, the process reaches the end of the first path, and the system directly outputs the answer as the final result to the user, completing this question-and-answer interaction. This method is suitable for scenarios with sufficient background information and relatively simple questions, and has the advantages of fast response speed and low resource consumption.

[0056] Conversely, if the model-generated answer is determined to contain preset keywords indicating missing or uncertain information (such as expressions like "no relevant data found" or "cannot answer based on existing information," or the answer content is empty or vague), the system will not accept the answer and will trigger the second path processing flow, i.e., proceed to step S507.

[0057] S507. If the answer contains preset keywords indicating missing or uncertain information, the user question is decomposed into atomic sub-questions, the atomic sub-questions are matched with the atomic questions in the atomic question library, and reasoning is performed based on the graph nodes associated with the successfully matched atomic questions to generate the final answer.

[0058] In step S507, for user questions identified as "insufficient information" in step S505, the system initiates a reliable reasoning path based on an atomic question library. The core idea of ​​this path is to break down complex user questions into several verifiable atomic sub-questions, locate precise factual evidence by matching them against a pre-built atomic question library, and then perform controlled reasoning based on these facts to generate an accurate and interpretable final answer.

[0059] Specifically, this step includes the following sequential operations: First, break down the user's problem.

[0060] The system invokes a large language model to perform deep semantic parsing on the original user question, identifying multiple implicit sub-query intents and generating a corresponding atomic sub-question for each intent. For example, for the complex question "What were the sales revenues of product A and product B in Q1 2024?", the system can decompose it into two atomic sub-questions: "What were the sales revenues of product A in Q1 2024?" and "What were the sales revenues of product B in Q1 2024?". In a preferred embodiment, to improve the accuracy of decomposition, the system can first perform entity linking and intent recognition on the user question based on a knowledge graph before decomposition, clarifying the entities mentioned in the question (such as "product A") and their corresponding nodes in the graph, thereby guiding the model to generate more accurate atomic sub-questions.

[0061] Secondly, the atomic problem matching.

[0062] The system matches each generated subproblem against the subproblem database. Matching can be performed using semantic similarity calculations (such as vector embedding comparison) or keyword exact matching. Since each problem in the subproblem database is indexed and mapped to a graph node corresponding to one or more structured records, once a match is successful, the system can immediately locate the specific graph node supporting the answer to that subproblem.

[0063] Secondly, reasoning is performed based on graph nodes.

[0064] The system collects graph nodes associated with all successfully matched sub-problems, forming a set of factual evidence. In one implementation, if multiple sub-problems are involved (such as comparison problems), the system will associate and integrate these graph nodes to construct a temporary reasoning context. For example, the nodes "Product A: 1 million yuan" and "Product B: 800,000 yuan" are juxtaposed to provide a basis for generating subsequent comparison conclusions.

[0065] Finally, the final answer is generated.

[0066] The system uses the integrated graph node set as context and calls the large language model again, prompting it to generate the final answer based on these verified facts. Since the input comes entirely from reliable graph nodes and the model is constrained to use only these facts to answer, the generated answer has high accuracy and traceability, effectively avoiding the "illusion" problem that is prone to occur in traditional end-to-end models.

[0067] Through step S507, the present invention realizes the transformation from "black box generation" to "white box reasoning", which is especially suitable for application scenarios such as finance, auditing, and government affairs where the reliability of the answer is extremely important.

[0068] When step S505 determines that the answer contains preset keywords indicating missing or uncertain information, the system enters the second path processing flow. This invention provides several optional implementation methods, all based on a constructed enhanced knowledge graph and atomic question base, but employing different collaborative mechanisms to generate the final answer.

[0069] Example 1: Atomic Problem Matching and Graph Fact Reasoning (Main Path) In this embodiment, the system first invokes a large language model to perform semantic parsing on the user's question, identifies the multiple sub-query intents contained therein, and generates corresponding atomic sub-questions for each intent. For example, for the question "What were the sales revenues of product A and product B in Q1 2024?", the system generates two atomic sub-questions: "What were the sales revenues of product A in Q1 2024?" and "What were the sales revenues of product B in Q1 2024?".

[0070] Subsequently, the system performs semantic matching (e.g., through vector similarity calculation) on each atomic sub-problem against the atomic problem database, locating one or more graph nodes associated with the successfully matched atomic problem. The system collects all matching results; if multiple facts are involved (e.g., comparison, aggregation), it determines their composability based on entity relationships, time dimension, or category labels in the knowledge graph, and calls the language model to generate the final answer based on these verified facts. This approach ensures that every step of reasoning is verifiable and is suitable for scenarios with high reliability requirements.

[0071] Example 2: Hybrid Search and Joint Suggestion Engineering In this embodiment, instead of explicitly decomposing the user question, the system performs two types of searches in parallel to build an enhanced context: First, implement a dual-path recall: BM25 Keyword Recall: Segment and identify entities in user questions, extract keywords (such as "Product A" and "sales revenue"), and perform sparse retrieval in the node text fields of the graph database; Vector semantic recall: Convert user questions into vectors using an embedding model (such as Sentence-BERT), retrieve the most similar graph nodes or associated edge vectors from a vector database, and return Top-K results.

[0072] Secondly, perform a second-order neighbor expansion: Starting with the nodes obtained from the dual-path recall, a 1-2 hop neighborhood traversal is performed in the knowledge graph to obtain directly connected or indirectly related nodes and edges, forming a local subgraph.

[0073] Next, perform a reordering: The system inputs the initial recall results into a re-ranking model, which comprehensively evaluates the following dimensions: Core topic matching degree (relevance between user question topic and node semantic summary); Information relevance (the importance of a node in the column of the original table); Data timeliness (e.g., the recentity of the time field); and output a finely ranked list of results sorted in descending order of relevance.

[0074] Finally, construct a joint hint: The system integrates the reordered semantic summaries of the graph nodes, the descriptions of associated edges, and several most similar atomic problems (as in-context examples) into a structured context, which is then input into a large language model to generate the answer. For example: Please answer the questions based on the following information: Fact 1: Product A had sales of 1 million yuan in Q1 2024.

[0075] Fact 2: Product B's sales revenue in Q1 2024 was 800,000 yuan.

[0076] Example question: 'What was the sales revenue of Product A in Q1 2024?' → '1 million yuan' Question: What were the sales revenues of Product A and Product B in Q1 2024? answer:" This approach uses atomic problems as high-quality examples to guide the model, significantly improving its generalization ability on unseen problems.

[0077] Example 3: Atomic problems as model fine-tuning data In this embodiment, the primary purpose of the atomic question library is to fine-tune the large language model offline before system deployment. Specifically, atomic questions and their corresponding structured records (or graph triples) are constructed into (question, answer) pairs as a supervised fine-tuning (SFT) dataset. After this fine-tuning, the language model in step S503 has a stronger ability to follow tabular facts and can extract answers more accurately from optimal background information, thereby reducing the probability of triggering S507.

[0078] Even though it still requires entering S507, the fine-tuned model performs better when performing problem decomposition (Example 1) or context understanding (Example 2). This approach transforms the role of the atomic question library from a "runtime retrieval source" to a "training signal source," enabling knowledge internalization and making it suitable for real-time question answering systems sensitive to response latency.

[0079] Through the three embodiments described above, this invention demonstrates how to flexibly utilize enhanced knowledge graphs and atomic question libraries to achieve high-precision, low-illusion table-based question answering under different resource constraints and performance requirements. Regardless of the approach taken, the core idea is to anchor the generative behavior of the large language model to pre-validated structured facts, thereby achieving a balance between open generation and factual constraints.

[0080] A second aspect of this application also provides a tabular data processing system based on knowledge graphs and large language models, see [link to relevant documentation]. Figure 5 The system includes: a table parsing module, a knowledge graph and atomic question database construction module, and a question-answering processing module.

[0081] The table parsing module receives raw table input and performs structured parsing. Specifically, this module calls the underlying table recognition algorithm to parse the table structure output from Excel, HTML, or OCR, identify vertical, horizontal, and block-merged cells, and fill the content of the merged cells into all missing data positions within their covered areas, thereby generating multiple complete structured records. For example, for a quarterly sales table containing "Product A" merged across rows and "2024" merged across columns, this module can output well-organized records such as ["Product A", "2024", "Q1", "1 million yuan"].

[0082] The knowledge graph and atomic question base construction module is coupled to the table parsing module and is used to construct an enhanced knowledge graph and establish an atomic question base based on the structured records. This module further includes: Node and edge building unit: Create graph nodes for each column value in each structured record, and establish the associated edges between nodes according to the table header hierarchy; Semantic summarization generation unit: invokes a large language model to generate semantic summary text in natural language form for each node and associated edge; Entity ablation and vectorization unit: Normalizes and merges entities with similar representations, and embeds and vectorizes the semantic summaries of all nodes and edges; Storage unit: Store the graph structure in a graph database (such as Neo4j) and the vector representation in a vector database (such as FAISS); Atomic question generation unit: Based on structured records, it uses preset prompts to guide the large language model to generate atomic questions that involve only a single fact, and establishes an index mapping between the atomic questions and the corresponding graph nodes, and stores them in the atomic question library.

[0083] The question-answering module is coupled to the knowledge graph and atomic question library construction module, and is used to respond to user questions and generate answers. This module includes: Hybrid retrieval unit: After receiving a user's question, it simultaneously performs BM25 keyword recall and vector semantic recall, and expands the recall results using second-order neighbors; Re-ranking unit: Refines the recall results based on core topic matching degree, information relevance, and data timeliness; Dual-path reasoning unit: (1) First, the large language model is invoked to answer directly based on background information; (2) If the answer contains preset keywords indicating missing or uncertain information, the second path is initiated—either decompose the question to match the atomic question library, or use the atomic questions as examples to provide joint prompts, or call the model finely tuned by the atomic questions to perform reasoning, and finally generate a reliable answer based on the graph nodes.

[0084] Through the collaborative work of the above modules, this system can efficiently process complex tables containing merged cells, build a semantically rich knowledge base, and flexibly respond to diverse user question-and-answer needs while ensuring factual accuracy, which is significantly better than existing end-to-end table question-and-answer systems.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0086] Finally, it should be noted that the above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for processing tabular data based on knowledge graphs and large language models, characterized in that: Includes the following steps: The original table is parsed in a structured manner to identify merged cells in the original table. The contents of the merged cells are then filled into all the missing data positions within the area covered by the merged cells to generate multiple structured records. Each structured record includes the values ​​corresponding to each data column of the table. A knowledge graph is constructed based on multiple structured records, and an atomic question database is established. When responding to user questions, question-answering processing is performed based on the knowledge graph and atomic question library; The construction of a knowledge graph and establishment of an atomic question base based on multiple structured records includes: The values ​​of each data column in each structured record are used as graph nodes, and a language model is called to generate semantic summary text describing the graph nodes. Based on the header structure relationship between the data columns, establish the associated edges between the graph nodes, and generate semantic summary text of the associated edges based on the structured records corresponding to the graph nodes; The graph nodes, associated edges, and corresponding semantic summary texts are stored in a graph database to form the knowledge graph. For each of the structured records, a language model is invoked to generate one or more atomic problems involving the structured record, and the atomic problems are associated with the graph nodes corresponding to the structured records and stored in the atomic problem library; When responding to user questions, question-answering processing is performed based on the knowledge graph and atomic question library, including: Retrieve graph nodes and associated edges related to the user's question, and generate background information based on the graph nodes and associated edges; The language model is invoked to generate an answer to the user's question based on the background information. If the answer contains a value, name, or status information corresponding to the user's question, then the answer is output; If the answer contains preset keywords indicating missing or uncertain information, the user question is decomposed into atomic sub-questions, the atomic sub-questions are matched with the atomic questions in the atomic question library, and reasoning is performed based on the graph nodes associated with the successfully matched atomic questions to generate the final answer.

2. The tabular data processing method as described in claim 1, characterized in that, The merged cells include vertically merged cells spanning multiple rows, horizontally merged cells spanning multiple columns, or block-shaped merged cells spanning both multiple rows and multiple columns simultaneously. The step of filling the merged cell's content into all empty data positions within the area covered by the merged cell includes: Copy the contents of the merged cell to the data fields of all cells within the rectangular area covered by the merged cell, excluding the original content location.

3. The tabular data processing method as described in claim 1, characterized in that, The step of establishing the association edges between the graph nodes based on the header structure relationship between the data columns includes: Identify the hierarchical dependencies of the data columns in the multi-level header, and establish associated edges between the graph nodes corresponding to data columns at the same level or with parent-child relationships.

4. The tabular data processing method as described in claim 1, characterized in that, The step of establishing the association edges between the graph nodes based on the header structure relationship between the data columns includes: The values ​​and column names of the data columns corresponding to the graph nodes are input into the language model to generate a natural language description that reflects the business meaning of the graph nodes.

5. The tabular data processing method as described in claim 1, characterized in that, The atomic problem is a natural language question that involves only a single data fact.

6. The tabular data processing method as described in claim 1, characterized in that, The step of associating the atomic problem with the corresponding graph node of the structured record and storing it in the atomic problem library includes: An index mapping is established between the atomic problem and one or more graph nodes corresponding to the structured record, so that the graph node corresponding to the user problem can be directly located through the atomic problem; The step of decomposing the user problem into atomic subproblems includes: The large language model is invoked to perform semantic parsing on the user question, identify multiple sub-query intents contained in the user question, and generate corresponding atomic sub-questions for each sub-query intent.

7. The tabular data processing method as described in claim 1, characterized in that, If the answer contains preset keywords indicating missing or uncertain information, the following steps are also included: Before decomposing user questions into atomic sub-questions, entity linking and intent recognition are performed on user questions based on the knowledge graph to improve the accuracy of atomic sub-question generation.

8. The tabular data processing method as described in claim 1, characterized in that, The retrieval of graph nodes and associated edges related to the user's question includes: Keywords are extracted from user questions, and a hybrid search based on keywords and graph traversal is performed in the graph database to obtain relevant subgraphs.

9. The tabular data processing method as described in claim 1, characterized in that, The process involves reasoning based on the graph nodes associated with the successfully matched atomic questions to generate the final answer, including: The graph nodes corresponding to multiple successfully matched atomic problems are associated and integrated, and the language model is invoked to generate the final answer based on the integrated graph node set.

10. A tabular data processing system based on knowledge graphs and large language models, characterized in that: include: The table parsing module is used to perform structured parsing on the original table, identify merged cells in the original table, and fill the content of the merged cells into all the empty data positions in the area covered by the merged cells to generate multiple structured records. Each structured record contains the values ​​corresponding to each data column of the table. The knowledge graph and atomic problem library building module is used for: The values ​​of each data column in each structured record are used as graph nodes, and a language model is called to generate semantic summary text describing the graph nodes. Based on the header structure relationship between the data columns, establish the associated edges between the graph nodes, and generate semantic summary text of the associated edges based on the structured records corresponding to the graph nodes; The graph nodes, associated edges, and corresponding semantic summary texts are stored in a graph database to form a knowledge graph. For each of the structured records, a language model is invoked to generate one or more atomic problems involving the structured record, and the atomic problems are associated with the graph nodes corresponding to the structured records and stored in the atomic problem library; The question-and-answer processing module is used to respond to user questions and perform the following operations: Retrieve graph nodes and associated edges related to the user's question, and generate optimal background information based on the graph nodes and associated edges; The language model is invoked to generate an answer to the user's question based on the background information. If the answer contains a value, name, or status information corresponding to the user's question, then the answer is output; If the answer contains preset keywords indicating missing or uncertain information, the user question is decomposed into atomic sub-questions, the atomic sub-questions are matched with the atomic questions in the atomic question library, and reasoning is performed based on the graph nodes associated with the successfully matched atomic questions to generate the final answer.

Images