Intelligent question and answer method and system for table-containing document

Abstract

The invention discloses an intelligent question and answer method and system for a table-containing document, and the method comprises the steps: firstly obtaining a document, carrying out the data preprocessing of the document, and obtaining a preprocessed table D and text data F; secondly, processing the table data through a large language model to respectively obtain table nodes and table entity nodes; and then the text data F is cut according to paragraphs, each text segment is embedded by embedding the large language model B, text nodes are generated, and a knowledge graph is constructed through recursion based on the text nodes, table nodes and table entity nodes. And finally, according to the knowledge graph, optimizing user query and maximizing retrieval information based on iterative retrieval. And constructing related texts and user queries into structured information, returning the structured information to the large model M, and generating answers. According to the method, documents in various forms can be better processed, and the precision is higher when intelligent question answering is carried out on the document containing the table.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent question answering technology, and in particular relates to an intelligent question answering method and system for documents containing tables. Background Art

[0002] Retrieval-augmented generation (RAG) has achieved remarkable success in addressing the challenges faced by large language models without requiring retraining. By referencing external knowledge bases, RAG is able to improve the output of LLMs, effectively alleviating issues such as "hallucinations," lack of domain-specific knowledge, and outdated information. However, the complex relational structure between different tables poses challenges for RAG systems. Current RAG performance is often suboptimal when answering documents containing tables, primarily due to the following issues:

[0003] (1) The structure of complex tables is not well parsed. When encountering more complex table types such as nested tables, directly converting them into text and processing them for the large language model will cause the large language model to incorrectly process related data, resulting in incorrect results. (2) The retrieved table text fragments are often isolated and lack contextual connections, making it difficult to provide a global understanding of the entire document, making it difficult for the large language model to effectively integrate this information to generate accurate answers. (3) Traditional retrieval-augmented generation (RAG) methods have significant limitations when dealing with complex problems, especially multi-hop problems. These methods usually rely on single-round retrieval, that is, only using user queries or questions as input to retrieve knowledge. They often fail when faced with multi-hop problems that require multi-step reasoning and complex information integration. For example, for problems that require extracting information from multiple documents and performing comprehensive analysis, single-round retrieval methods may not provide enough contextual information to generate accurate answers.

[0004] While existing multi-round retrieval methods can gradually collect information through multiple retrievals, these methods typically require multiple calls to large language models to generate new queries or perform inference, which increases computational cost and latency. Summary of the Invention

[0005] To address the aforementioned issues, this method proposes an intelligent question-answering method and system for documents containing tables. After acquiring a table document, it is converted into an image. Using OCR technology to identify the table and text in the document, the text is segmented into blocks, and their coordinates are recorded. A large model is used to identify the table's structural information and convert it into Markdown format, generating a summary for the entire table. The table's structural information is analyzed, and the table is classified according to rules. The table is preprocessed by category and then handed over to a large language model to identify the columns containing entities. A text block summary is generated for each entity column. Entity nodes are clustered with the table node as the root node, and the clusters are summarized. This process is repeated recursively to ultimately construct a tree-like knowledge graph. In the retrieval enhancement phase, an iterative retrieval query optimizer is introduced to generate new queries and filter out irrelevant text blocks without requiring multiple LLM calls, significantly reducing system hallucinations and improving accuracy.

[0006] The optimizer consists of an extractor, an annotator, and a reorganizer fine-tuned based on modernBert. The principle is that after receiving a user query, the relevant embedding model is used to generate the corresponding embedding. The retriever searches the knowledge graph, and the retrieved text blocks and query are input into the extractor and annotator for extraction and annotation. The extracted and annotated text blocks and query are then input into the reorganizer to generate a new query. The above process is repeated until the iteration limit is reached or no new text blocks are retrieved. It stops and all this information is passed to the final generator to obtain the final response. This method can efficiently generate new queries for subsequent retrieval rounds, thereby improving the quality and accuracy of the final retrieval enhancement generated answers. This method eliminates the need to call the LLM multiple times for query generation, thereby improving performance when processing complex queries.

[0007] An intelligent question-answering method for a document containing a table, comprising the following steps:

[0008] Step 1: Prepare document data: obtain a document, which includes two parts: text data and table data.

[0009] Step 2: Data preprocessing: Perform data preprocessing on the document to obtain preprocessed table data D and text data F.

[0010] The document is converted into an image, and the image is converted from the BGR color space to a grayscale image to facilitate subsequent edge detection operations. A Gaussian filter is used to perform noise reduction on the grayscale image to reduce the interference of noise on edge detection. The Canny algorithm is applied to the preprocessed image for edge detection, and appropriate low threshold and high threshold parameters are set to obtain edge information in the image. Contour detection uses the findContours function of OpenCV to extract all contours, and polygonal approximation is performed on the detected contours one by one. At the same time, the contour area is calculated and rectangles with an area greater than the set threshold are screened out to determine the boundary of the table. Hough transform is performed on the edge image detected by the Canny algorithm to detect straight lines in the image. The detected straight lines are traversed and the polar coordinate parameters (ρ and θ) of the straight lines are used to determine whether they are horizontal or vertical lines, thereby determining the row and column boundaries of the table. The position of the table cell is determined based on the intersection of the detected horizontal and vertical lines. Each cell area is segmented, and the text in the cell is recognized using OCR technology. The recognized table content is stored in markdown data format to obtain the preprocessed table D. For the rest, the text detection model is used to detect the text position, and the OCR technology is used to recognize the text to obtain the preprocessed text data F.

[0011] Step 3: Table data processing: The table data D is processed through the large language model M to obtain table nodes and table entity nodes respectively.

[0012] First, the preprocessed table D is identified as a table type according to preset rules. After the identification is completed, the large language model M is obtained to identify the table header column and identify the column where the entity is located; the large language model M is used to generate a summary for the entire table D and generate a table node; the content of the cell where the entity is located and the content of its adjacent cells are spliced into a piece of text; the model M generates a summary based on the input text; the summary is embedded by embedding the large language model B to generate a table entity node.

[0013] Step 4: Text data processing: Cut the text data F into paragraphs, and embed each paragraph into the large language model B to generate an embedding to generate a text node.

[0014] The text is divided into paragraphs, and each paragraph is embedded using the embedding model B. A text block node is generated.

[0015] Step 5: Based on text nodes, table nodes, and table entity nodes, construct the knowledge graph through recursion.

[0016] Using the document as the root node and each table node as a child node, we select text nodes close to the table and table entity nodes under the table node for clustering using the DBSCAN algorithm. For each cluster generated, we call the large language model M to summarize the text blocks within the cluster and generate a text summary for that cluster.

[0017] The generated text summaries are re-input into embedding model B to obtain new embedding vectors. Based on these new embedding vectors, DBSCAN clustering is performed again to generate higher-level text summaries. This process is repeated until further clustering is impossible or the generated summaries reach a predetermined level of abstraction. In this way, a multi-layered tree-like knowledge graph is constructed from the bottom up, with nodes at each layer representing different levels of summary information.

[0018] Step 6: Based on the knowledge graph, optimize user queries and maximize retrieved information based on iterative retrieval.

[0019] Fine-tune the extractor, tagger and reorganizer based on the modernBert model; first obtain the user question Q to be answered, and obtain the embedding E of the question based on the embedding model; retrieve relevant nodes from the knowledge graph based on similarity, starting from the root node, recursively retrieve its child nodes until reaching the bottom node, and also perform a search for all leaf nodes. For all retrieved nodes, use the extractor to extract the text in the node to extract the text related to the user query; use the tagger to annotate the retrieved nodes. If the node text is relevant to the user query and additional information is required, it is marked as a continue retrieval node. If it is not related to the user or no additional information is required, it is marked as an end node. If all nodes are marked as end nodes or the iteration limit is reached, the retrieved nodes and queries are input into the next step; if there is a continue retrieval node or the iteration limit is not reached, the relevant text and query extracted by the extractor are used to generate a new query, and the new query is used as the input for the next round of retrieval. Repeat this process until the iteration limit is reached or all nodes are marked as end nodes.

[0020] Step 7: Perform retrieval enhancement generation: Construct relevant text and user queries into structured information and return it to the large model M to generate answers.

[0021] This application also provides an intelligent question-answering system for documents containing tables, including the following modules:

[0022] Document import module, used to import document data.

[0023] The data preprocessing module is used to preprocess the imported document data to obtain preprocessed document text data and document table data.

[0024] The table data processing module processes the table data using the large model to obtain the corresponding table nodes and table entity nodes.

[0025] The text data processing module embeds the text data into a large language model to generate text nodes.

[0026] The knowledge graph construction module is used to construct a knowledge graph using preprocessed text data and table data to obtain a tree-like knowledge graph containing document table summaries, entity summaries and embedded information.

[0027] The retrieval module, based on the knowledge graph, uses the modernBert model to iteratively retrieve and optimize user queries and maximize retrieved information.

[0028] The answer module constructs structured text based on the iteration results, and the large model generates answers.

[0029] Beneficial effects of the present invention: Compared with the existing technology, the above method can better process documents in various forms, such as document data containing tables and text, and has higher accuracy when conducting intelligent question-answering for documents containing tables.

[0030] When facing documents containing tables, compared with existing retrieval enhancement generation methods, this method can more effectively answer questions about documents containing tables because it analyzes the table structure and extracts entities from the table information before constructing a knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the process of the present invention;

[0032] Figure 2 Build a schematic diagram for the knowledge graph;

[0033] Figure 3 Schematic diagram of iterative query process;

[0034] Figure 4 Schematic diagram of the knowledge graph structure. DETAILED DESCRIPTION

[0035] In order to make the technical solutions and advantages of the present invention more clear, the present invention will be further described below with reference to the accompanying drawings.

[0036] Figure 1 This is a flow chart of an intelligent question-answering method for documents containing tables according to the present invention. The intelligent question-answering method for documents containing tables mainly includes the following steps:

[0037] First, step S1 is executed to obtain a high-temperature resistant material design specification document with a table and convert it into a corresponding image.

[0038] When designing and selecting materials for brackets, the first thing to consider is temperature characteristics. High-temperature resistance is usually a consideration. The sources of high temperature are usually: Ambient high temperature, such as near the engine area, APU compartment area, etc.; High temperatures generated by the pipelines and equipment themselves; Some environmental control pipelines and inerting systems dissipate heat outward, resulting in high surface temperatures on the pipelines. This is a key consideration when selecting bracket materials. Consider changes in bracket material capabilities based on the temperature field on and near the high-temperature pipeline surface. For brackets with high-temperature resistance requirements, the following materials are commonly available: titanium alloys, stainless steel, and other high-temperature resistant materials. Commonly used high-temperature resistant metal materials for bracket design are shown in Table 1.

[0039] Table 1 Selection of commonly used high temperature resistant materials

[0040]

[0041] Then execute step S2 and use OpenCV's cv2.imread() function to load the image containing the table. Convert the image from BGR color space to grayscale to facilitate subsequent edge detection operations. Use a Gaussian filter to denoise the grayscale image to reduce the interference of noise on edge detection. Use the cv2.Canny() function to perform edge detection on the preprocessed image, and set appropriate low threshold and high threshold parameters to obtain edge information in the image. Call the cv2.findContours() function to extract all contours in cv2.RETR_LIST mode, and use the cv2.CHAIN_APPROX_SIMPLE method to simplify contour points. Perform polygon approximation on each detected contour, and use the cv2.approxPolyDP() function to determine whether the contour is a rectangle (i.e., it has 4 vertices). At the same time, calculate the contour area and filter out rectangles with an area greater than the set threshold to determine the boundary of the table. Use the cv2.HoughLines() function to perform Hough transform on the edge image detected by the Canny algorithm to detect straight lines in the image. Traverse the detected straight lines and use the polar coordinate parameters (ρ and θ) of the straight lines to determine whether they are horizontal or vertical lines, thereby determining the row and column boundaries of the table. Determine the position of the table cells based on the intersection of the detected horizontal and vertical lines. Segment each cell area and use OCR technology to recognize the text in the cell. Use the cv2.drawContours() or cv2.rectangle() function to draw the rectangular boundary of the table on the original image. Store the recognized table content in markdown data format to obtain preprocessed table data F. For the remaining text data that does not exist in the table, use OCR technology to identify it, segment it into natural segments, record their distance from the nearest table, and obtain text data D.

[0042] Then, step S3 is executed to identify the arrangement of the table and data by judging the cells in the first row and first column of the table according to the rules. Nested cells are then processed and split into simple cells. The large language model GPT-4o is then used to first generate a brief summary of the preprocessed table data F. This summary is then embedded using the Sentence-BERT embedding model to form a table node. The embedding model is then used to embed the summary for each row of the table, generating a summary for each row of data to form a table entity node.

[0043] The rules are judged by cell spacing, text content consistency, and header features. First, the horizontal and vertical spacing between cells in the table are calculated. If the average value of the horizontal spacing is significantly greater than the vertical spacing, it tends to be judged as a vertical arrangement; otherwise, it is a horizontal arrangement. Then check whether the text format of cells in the same row or column is consistent (such as font, font size, alignment, etc.). If the text format of the same column is highly consistent, it is more likely to be a vertical arrangement; if the text format of the same row is highly consistent, it is more likely to be a horizontal arrangement. Finally, analyze the content and position of the header cells. If the header cells are concentrated in the first column and the content is a category label, it tends to be arranged vertically; if the header cells are concentrated in the first row and the content is a time or category label, it tends to be arranged horizontally.

[0044] The table is comprehensively evaluated according to the above rules, the weight of each rule is calculated and weighted summed up, and finally it is determined whether the table is arranged vertically or horizontally.

[0045] Then, step S4 is executed to form a summary for each text block, and then the embedding model is used to generate an embedding for the summary, and a text block node is generated for each text block.

[0046] Then execute step S5, as Figure 2 As shown in the figure, with the document as the root node and the table node as the child node, the text content in the table document is divided into multiple text blocks according to a certain length. The text blocks adjacent to the table node are selected, and each text block is embedded using a pre-trained language model such as SBERT to obtain a vector representation of the text block. The DBSCAN algorithm is used for clustering to cluster similar text blocks with table entity nodes, and a summary text is generated for each cluster. The pre-trained language model (GPT-4o) is used to summarize the text blocks in the cluster. Repeat the above embedding, clustering and summarizing process, convert all nodes into child nodes of the table node, and gradually build a tree structure from the bottom text block to the high-level summary text to form a knowledge graph.

[0047] Then, step S6 is executed to obtain the user question Q to be answered, generate the embedding corresponding to the question using the embedding model, and process the user query using the extractor, tokenizer and reorganizer fine-tuned based on MordenBert, such as Figure 3 As shown. Relevant nodes are retrieved from the knowledge graph based on the similarity with the user query. The knowledge graph structure is as follows Figure 4 As shown. Starting from the root node, for nodes with high relevance, recursively retrieve their child nodes until reaching the bottom node, and perform a search for all leaf nodes, and select nodes with high similarity. For all retrieved nodes, use the extractor to extract the text in the node, and extract the text related to the user query. Use the tagger to tag the retrieved nodes. If the node text is related to the user query and requires additional information, it is marked as a node that needs to be retrieved. If it is not related to the user or does not require additional information, it is marked as an end node. If all nodes are marked as end nodes or the iteration limit is reached, the retrieved nodes and queries are input into the next step. If there are nodes that need to be retrieved or the iteration limit is not reached, the relevant text and query extracted by the extractor are used to generate a new query, and the new query is used as the input for the next round of retrieval. This process is repeated until the iteration limit is reached or all nodes are marked as end nodes.

[0048] The extractor, tagger, and reorganizer are all fine-tuned from the MordenBert model. Training data uses the existing multi-hop question answering dataset 2Wiki-MultihopQA as the underlying data source. Based on the characteristics of each sub-model, corresponding annotated datasets are constructed, and then multiple rounds of training are performed using contrastive learning.

[0049] Then, step S7 is executed to reorganize the information of the relevant table nodes and text block nodes obtained above into structured text, and the large language model GPT-4o is used to generate an answer to the question and return the answer.

[0050] The present invention provides another solution, an intelligent question-answering system for documents containing tables, comprising the following modules:

[0051] Document import module, used to import document data.

[0052] The data preprocessing module is used to preprocess the imported document data to obtain preprocessed document text data and document table data.

[0053] The table data processing module processes the table data using the large model to obtain the corresponding table nodes and table entity nodes.

[0054] The text data processing module embeds the text data into a large language model to generate text nodes.

[0055] The knowledge graph construction module is used to construct a knowledge graph using preprocessed text data and table data to obtain a tree-like knowledge graph containing document table summaries, entity summaries and embedded information.

[0056] The retrieval module, based on the knowledge graph, uses the modernBert model to iteratively retrieve and optimize user queries and maximize retrieved information.

[0057] The answer module constructs structured text based on the iteration results, and the large model generates answers.

Claims

1. An intelligent question-answering method for documents containing tables, characterized in that: The following steps are involved: Step 1: Obtain a document, which includes text data and table data; Step 2: Preprocess the document to obtain preprocessed table data D and text data F; Step 3: Process the table data D through the large language model M to obtain table nodes and table entity nodes respectively; Step 4: Split the text data F into paragraphs, and generate text nodes for each paragraph by embedding the large language model B; Step 5: Based on text nodes, table nodes, and table entity nodes, construct the knowledge graph through recursion; Step 6: Based on the knowledge graph, optimize user queries and maximize retrieved information based on iterative retrieval; construct relevant text and user queries into structured information and return it to the large model M to generate answers.

2. The intelligent question-answering method for documents containing tables according to claim 1, characterized in that: The data preprocessing specifically includes the following operations: Convert the document into an image, convert the image from BGR color space to grayscale, perform noise reduction on the grayscale image using a Gaussian filter, perform edge detection on the pre-processed image using a Canny algorithm, set a threshold parameter, and obtain edge information in the image; Contour detection performs polygonal approximation on each detected contour, calculates the contour area, and selects rectangles with an area greater than a set threshold to determine the boundaries of the table. Hough transform is performed on the edge image detected by the Canny algorithm to detect straight lines in the image. The detected straight lines are traversed and the polar coordinate parameters of the straight lines are used to determine whether they are horizontal or vertical lines, thereby determining the row and column boundaries of the table. According to the intersection of the detected horizontal and vertical lines, the position of the table cells is determined, each cell area is segmented, and the text in the cell is recognized using OCR technology. The recognized table content is stored to obtain the preprocessed table D; For the rest, the text detection model is used to detect the text position, and the OCR technology is used to recognize the text to obtain the preprocessed text data F.

3. The intelligent question-answering method for documents containing tables according to claim 2, characterized in that: The specific implementation process of step 3 is as follows: First, the preprocessed table D is identified as a table type according to preset rules. After the identification is completed, the large language model M is obtained to identify the table header column and identify the column where the entity is located; the large language model M is used to generate a summary for the entire table D and generate a table node; the content of the cell where the entity is located and the content of its adjacent cells are spliced into a piece of text; the model M generates a summary based on the input text; the summary is embedded by embedding the large language model B to generate a table entity node.

4. The intelligent question-answering method for documents containing tables according to claim 3, characterized in that: The pre-set rules are specifically as follows: first calculate the horizontal and vertical spacing between cells; if the average value of the horizontal spacing is greater than the vertical spacing, it is judged to be arranged vertically; otherwise, it is arranged horizontally; then check whether the text format of the cells in the same row or column is consistent. If the text format of the same column is consistent, it is arranged vertically; if the text format of the same row is consistent, it is arranged horizontally; finally, analyze the content and position of the header cells. If the header cells are concentrated in the first column and the content is a category label, it is arranged vertically; if the header cells are concentrated in the first row and the content is a time or category label, it is arranged horizontally.

5. The intelligent question-answering method for documents containing tables according to claim 4, characterized in that: The specific implementation process of constructing the knowledge graph is as follows: With the document as the root node and each table node as a child node, the text nodes close to the table and the table entity nodes under the table node are selected for clustering using the DBSCAN clustering algorithm. For each cluster generated by clustering, the large language model M is called to summarize the text blocks within the cluster to generate a text summary for the cluster; The generated text summary is re-input into embedding model B to obtain a new embedding vector; then, based on the new embedding vector, DBSCAN clustering is performed again to generate a text summary; this process is repeated until clustering is no longer possible or the generated summary reaches the predetermined level of abstraction, thus constructing a tree-like knowledge graph with a multi-layer structure, in which the nodes of each layer represent summary information at different levels.

6. The intelligent question-answering method for documents containing tables according to claim 5, characterized in that: The specific implementation process of optimizing user queries and maximizing retrieved information based on iterative retrieval is as follows: Fine-tune the extractor, tagger, and reorganizer based on the modern Bert model. First, obtain the user question Q to be answered and obtain the question embedding E based on the embedding model. Then, retrieve relevant nodes from the knowledge graph based on similarity. Starting from the root node, recursively retrieve its child nodes until reaching the bottom node. All leaf nodes are also retrieved once. For all retrieved nodes, use the extractor to extract the text in the nodes to extract the text related to the user query. Use the tagger to annotate the retrieved nodes. If the node text is relevant to the user query and requires additional information, it is marked as a continue retrieval node. If it is not relevant to the user or does not require additional information, it is marked as an end node. If all nodes are marked as end nodes or the iteration limit is reached, the retrieved nodes and queries are input into the next step; if there are continue retrieval nodes or the iteration limit has not been reached, a new query is generated using the relevant text and query extracted by the extractor, and the new query is used as the input for the next round of retrieval. This process is repeated until the iteration limit is reached or all nodes are marked as end nodes.

7. An intelligent question-answering system for documents containing tables, used to implement the intelligent question-answering method according to any one of claims 1 to 6, characterized in that: Includes the following modules: Document import module, used to import document data; The data preprocessing module is used to preprocess the imported document data to obtain preprocessed document text data and document table data; The table data processing module processes the table data using the large model to obtain the corresponding table nodes and table entity nodes; The text data processing module embeds the text data into a large language model to generate text nodes; The knowledge graph construction module is used to construct a knowledge graph using preprocessed text data and table data, and obtain a tree-like knowledge graph containing document table summaries, entity summaries, and embedded information; The retrieval module, based on the knowledge graph, uses the modern Bert model to iteratively optimize user queries and maximize retrieved information; The answer module constructs structured text based on the iteration results, and the large model generates answers.

Images