Complex table text conversion method based on tree structure

Abstract

The invention discloses a complex table text conversion method based on a tree structure. The method comprises the following steps of: firstly, converting a table image into structured JSON (JavaScript Object Notation) data; then, performing type judgment on cells in the JSON data by utilizing a prompt word normal form of LLM, and supplementing cell type information; performing row / column position matching and connectivity verification according to two basic principles of a title-data relationship and a row-column position relationship, thereby automatically constructing two subtrees of a row tree and a column tree; and finally, performing row and / or column-to-dimension text conversion based on a sub-tree structure by utilizing a tree traversal strategy assisted by a queue and a stack. The problem that a traditional table serialization mode cannot process hierarchical tables, merge cells and nesting relations is solved, and meanwhile the problem that a table conversion mode does not consider content semantic relations is solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of natural language processing, and particularly relates to a complex table text conversion method based on a tree structure, which is used to enable a large model to understand a complex table. BACKGROUND

[0002] In the digital era, the tide of data is surging, among which table data occupies an important position in its unique structured form. From the detailed financial statements of enterprises in the financial field to the complex project data in the engineering field, and various types of statistical information in various industries, table data is ubiquitous and widely permeates every corner of society. Currently, table data is growing exponentially, especially in the fields of finance, medicine, and the Internet of Things, and the amount of data in a single table has exceeded the million-row level, and the demand for table processing has sharply increased.

[0003] With the expansion and deepening of businesses in various fields, the form of tables has gone beyond the simple structured mode. In the medical field, the hierarchical examination index table structure is complex and has multiple layers of nesting. In the field of academic research, a large number of tables in IEEE journals have cross-page image annotation associations, significantly increasing the difficulty and complexity of data processing. In the legal field, the clause list of documents often uses recursive hierarchical coding, and at least six levels of clause nesting are required to locate the desired analysis. Overall, there is a trend from the traditional row-column alignment structure to complex forms containing irregular nested relationships, such as data block distribution across rows and columns, and multi-level header nesting.

[0004] Under this background, how to process various complex tables and improve the analysis efficiency of table data has become an important problem to be solved. In recent years, large language models have shown excellent capabilities for data analysis and processing in many fields. Given this, we can apply large language models to the processing of table data. However, most large language models currently can only process text modalities, which requires us to convert table data into text that large language models can understand and process. At the same time, the current table text conversion method cannot well handle complex tables containing irregular nested relationships.

[0005] The present application explores an effective complex table text conversion method using the natural hierarchy of tree structure and large language models in the field of natural language processing. This method can accurately and efficiently convert various types of tables into text, providing a new solution for efficient analysis and utilization of complex table data in various fields. SUMMARY

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for converting complex table text based on a tree structure. This method solves the problems of traditional table serialization methods being unable to handle hierarchical tables, merged cells, and nested relationships, while also addressing the issue that table conversion methods fail to consider semantic relationships within the content. Furthermore, it enables large language models to understand complex tables, thus providing a superior preprocessing method for combining large language models with table processing. This provides valuable reference for practical applications such as data mining, information retrieval, and intelligent question answering.

[0007] The objective of this invention is achieved through the following technical solution: a method for converting complex table text based on a tree structure, comprising the following steps: (1) Convert the table image into structured JSON data; (2) Use the prompt word paradigm of LLM to determine the type of cells in JSON data and supplement the cell type information; the cell type includes row header cells, column header cells and data cells; (3) Based on the two basic principles of title-data relationship and row-column position relationship, the row / column position matching and connectivity are verified, thereby automatically constructing two subtrees: row tree and column tree; (4) A tree traversal strategy assisted by queues and stacks is used to perform row and / or column-to-dimensional text conversion based on subtree structures. Furthermore, for multimodal documents containing tables, the layout of the multimodal document is first analyzed, and then the table images are extracted and converted into structured JSON data.

[0008] Furthermore, the layout of the multimodal document is first analyzed, then table images are extracted and converted into structured JSON data, including: First, the multimodal document is inferred using a finely tuned LayoutLMv3 model for layout detection, thereby locating different elements in the document: including images, tables, text, headings, and formulas; then, for all table elements... TableMaster is used to define the table boundaries, and then PaddleOCR is used to recognize the table text content, ultimately identifying all table elements. Convert the table image to JSON spreadsheet format. This facilitates further data processing and analysis, where n is the number of table elements.

[0009] Furthermore, step (2) specifically includes the following sub-steps: JSON format for each table element ,1≤ i ≤ nFirst, the LLM prompt word paradigm is used to determine the type of each cell in the JSON data based on its three functional roles: column header, row header, and data. Then, after obtaining the type determination result for each cell, auxiliary code is used to rewrite the JSON file and supplement the corresponding cell types, finally obtaining the supplemented JSON format. .

[0010] Furthermore, step (3) specifically includes the following sub-steps: (3.1) First, construct a column tree. Find all column header cells based on the header-data relationship, and verify their column position matching and connectivity according to their row-column position relationship in the table, and connect them reasonably into the tree structure. Then, treat other types of cells that have a connection relationship with the column header cells as data at the same level, and then integrate them as child nodes of the column header cells at the leaf node position according to the row-column position relationship, and finally construct the column tree. ; (3.2) Finally, construct the row tree. Find all row title cells based on the title-data relationship, and verify their row position matching and connectivity according to their row-column position relationship in the table, and connect them reasonably into the tree structure. Then, treat other types of cells that have a connection relationship with the row title cells as data at the same level, and then integrate them as child nodes of the row title cells at the leaf node position according to the row-column position relationship, and finally construct the row tree. .

[0011] Furthermore, step (4) specifically involves: For column trees To perform column-to-column transformation, firstly, a depth-first traversal algorithm is used to traverse the column tree, outputting one or more column headers sequentially in a queue. Secondly, for each data cell, its corresponding row header information is determined by tracing the source row tree, and one or more row headers are stored using a stack. Then, the collected row headers are traced back through the source column tree to find the corresponding column header, again using a stack, and finally output in order. The final result is the transformed text. ; For row trees To perform row-to-row conversion, firstly, a depth-first traversal algorithm is used to traverse the row tree, outputting one or more row headers sequentially in a queue. Secondly, for each data cell, its corresponding column header information is determined by tracing the source column tree, and one or more column headers are stored using a stack. Then, the collected column headers are traced back through the source row tree to find the corresponding row headers, using a stack, and finally, they are output in order. The final result is the row-to-row converted text. .

[0012] Furthermore, if step (4) involves performing a two-dimensional text conversion of rows and columns, it also includes: optimizing the two-dimensional text of rows and columns through LLM prompt word engineering and generating an optimized natural language description.

[0013] Furthermore, LLM cue word engineering is used to optimize the row and column-dimensional text and generate an optimized natural language description, including: Text results for two different strategies and The large model is used to evaluate the text from three perspectives: information presentation logic, text readability, and information completeness. The text content is then re-output after the expression is optimized in a more concise and logically clear direction.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for converting complex table text based on a tree structure.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for converting complex table text based on a tree structure.

[0016] The beneficial effects of this invention are: for multimodal documents, it can more accurately classify document content, quickly capture table data for subsequent operations, and improve the efficiency of data analysis. Compared with traditional table processing task systems, this invention performs hierarchical structure processing and optimizes the expression method for more complex tables, improving the understanding ability of large models for complex tables; and better utilizes the efficiency of large language models, providing a more intelligent table expression form, and proposing a better preprocessing method for various table tasks. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a method for converting complex table text based on a tree structure, provided in an embodiment of the present invention.

[0019] Figure 2 The flowchart for determining the cell type in step (2) is as follows.

[0020] Figure 3 Example diagram for automated tree structure construction in step (3).

[0021] Figure 4 Example diagram of the tree traversal strategy using queues and stacks in step (4).

[0022] Figure 5 Example diagram of the two-dimensional text evaluation in step (5). Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0024] This invention proposes a method for converting complex table text based on a tree structure. Taking a multimodal PDF as an example in a specific implementation scenario: first, the multimodal PDF is subjected to layout detection inference using a finely tuned LayoutLMv3 model; then, for all table elements... A table recognition model is used to convert the data into JSON format. Then, a large-scale model prompt word project combined with auxiliary code is used to supplement the cell types in the JSON file. Next, the matching and connectivity of row / column positions are verified using two basic principles: title-data relationship and row-column position. This automatically constructs two subtrees: a row tree and a column tree. A queue and stack-assisted tree traversal strategy is then used to output the text representing the table's row and / or column transformations. Preferably, the queue and stack-assisted tree traversal strategy is used to output the text representing the table's row and column transformations. These two text outputs are then input into a large language model for evaluation and sentence optimization.

[0025] like Figure 1 As shown, this invention provides a method for converting complex table text based on a tree structure, including the following steps: (1) Perform layout analysis on the multimodal document, extract table images and convert them into structured JSON data.

[0026] The specific steps (1) are as follows: First, the multimodal document is inferred using a finely tuned LayoutLMv3 model for layout detection, thereby locating different elements in the document: including images, tables, text, headings, and formulas; then, for all table elements... TableMaster is used to define the table boundaries, and then PaddleOCR is used to recognize the table text content, ultimately identifying all table elements. Convert the table image to JSON spreadsheet format. This facilitates further data processing and analysis, where n is the number of table elements. The multimodal document is in PDF format and contains text, tables, and images.

[0027] (2) Use the prompt word paradigm of LLM to determine the type of cells in JSON data and supplement the cell type information.

[0028] See Figure 2 Step (2) specifically includes the following sub-steps: (2.1) JSON format for each table element ,1≤ i ≤ n This method utilizes the LLM prompt word paradigm to determine the type of cells in JSON data based on their three main functional roles: column headers, row headers, and data. Column headers (columns_header) are titles used to uniquely identify the content of certain columns in a table or data structure, summarizing or describing the theme or information category of those columns; there may be multiple column headers. Row headers (row_header) are titles used to uniquely identify the content of certain rows in a table or data structure, summarizing or describing the theme or information category of those rows; there may be multiple row headers. Data is the core component of the table, consisting of cells containing the actual information.

[0029] (2.2) After obtaining the type determination result of each cell, auxiliary code is used to rewrite the specific JSON file and supplement the corresponding cell types to reduce illusions and improve efficiency. Finally, the supplemented JSON format is obtained. .

[0030] In step (2), the cell type determination prompt used by the large language model is: (3) Based on the two basic principles of title-data relationship and row-column position relationship, the column position matching and connectivity are verified, thereby automatically constructing two subtrees: row tree and column tree.

[0031] Step (3) specifically includes the following sub-steps: (3.1) JSON format for each table Based on two basic principles and different strategies, automation is carried out, and a tree structure is constructed according to the relative relationships.

[0032] (3.2) First, construct a column tree. Based on the header-data relationship, find all cells with the attribute "columns_header" and verify their column position matching and connectivity according to their row-column position in the table, connecting them appropriately into the tree structure. Then, treat other attribute cells (cells with attributes other than "columns_header") that are related to the column header cells as data at the same level. Then, based on the row-column position relationship, integrate them as child nodes of the cell with the attribute "columns_header" and located at the leaf node position, finally constructing the tree. This process must strictly follow specific rules: cells whose attribute is not "columns_header" must not have child nodes, thus effectively avoiding potential multi-level nesting issues and ensuring the accuracy and rationality of the tree structure construction.

[0033] The following is a pseudocode example of the algorithm in section (3.2). (3.3) Finally, construct the row tree. Based on the header-data relationship, find all cells with the attribute "row_header" and verify their row position matching and connectivity according to their row-column position in the table, connecting them appropriately into the tree structure. Then, treat other attribute cells (cells with attributes other than "row_header") that are related to the row header cell as data at the same level. Based on the row-column position relationship, integrate them as child nodes of the cell with the attribute "row_header" and located at the leaf node position, ultimately constructing the tree. .

[0034] For example, see Figure 3 During the construction of the row tree, after finding all cells with the attribute "row_header", when constructing child nodes based on the row-column position relationship, there is no connection between all column header cells and row header cells. Therefore, in its construction, data cells are only integrated as child nodes of cells with the attribute "row_header" and located at the leaf node position based on the row-column position relationship.

[0035] (4) Implement a two-dimensional text conversion based on subtree structure by using a tree traversal strategy assisted by queues and stacks.

[0036] See Figure 4 Step (4) specifically involves: (4.1) For the obtained column tree and rows of trees The text will be transformed using two strategies: column-oriented transformation and row-oriented transformation.

[0037] (4.2) For column trees Perform column transformation, and use a depth-first traversal algorithm to traverse the column tree, visiting one or more column headers. Enqueue items in the order of access. Output results That is, a queue Output the elements in order. Then, for each data cell... (Located at level x in the column tree, and level y in the row tree), along the row tree To trace back to all its corresponding ancestor line header nodes ∈ and the line header nodes obtained from tracing the source Push them onto the stack in source order (from leaf to root). Then, the collected line headers Source tree Find all corresponding ancestor column headers (line headings) (corresponding to y column headers), each traced column header node Push them onto the stack in the order of their origin. In the end, we obtained , , ..., Finally, the stack... and The elements in the stack are arranged and output in the order they are popped (i.e., from root to leaf) to form the cell. The corresponding complete title information fragment .Then ,Right now All data cells This is combined with its corresponding title information fragments. Ultimately, the desired result is... .

[0038] The following is a pseudocode example of the algorithm for step (4.2). (4.3) For row trees Perform a row direction transformation, and use a depth-first traversal algorithm to traverse the row tree, visiting one or more row titles. Enqueue items in the order of access. Output results That is, a queue Output the elements in order. Then, for each data cell... (Located at level x in the column tree, and level y in the row tree), along the column tree To trace back to all its corresponding ancestor column header nodes ∈ and the column header nodes obtained through tracing Push them onto the stack in source order (from leaf to root). Then, the collected column headers Tracing the source tree in sequence Find all corresponding ancestor line headers (Column headings) (corresponding to y line headers), each traced line header node Push them onto the stack in the order of their origin. In the end, we obtained , , ..., Finally, the stack... and The elements in the stack are arranged and output in the order they are popped (i.e., from root to leaf) to form the cell. The corresponding complete title information fragment .Then ,Right now All data cells This is combined with its corresponding title information fragments. Then, the desired result is obtained. Ultimately, two texts with different expression strategies were obtained. and .

[0039] It should be noted that if the first a rows do not have row headers, when tracing upwards to the column header node for a data cell, there is no need to continue tracing the row tree; if the first b columns do not have column headers, when tracing upwards to the row header node for a data cell, there is no need to continue tracing the column tree.

[0040] For example, see Figure 3 Since there are no row headers in the first two rows, when tracing upwards to the column header node for a data cell, there is no need to continue tracing the row tree.

[0041] (5) Optimize the row and column two-dimensional text through LLM prompt word engineering and generate optimized natural language description.

[0042] Step (5) specifically includes the following sub-steps: See Figure 5 For the text results of two different strategies and We use large models (such as GPT-5) to evaluate the quality of information presentation from three perspectives: logicality, readability, and completeness. After optimizing the expression to be more concise and logically clear, we re-output the text content.

[0043] In step (5), the large language model uses the following text selection hints: The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method for converting complex table text based on a tree structure.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for converting complex table text based on a tree structure.

[0045] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0047] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0048] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A complex table text conversion method based on a tree structure, characterized by, The method comprises the following steps: (1) converting a table image into structured JSON data; (2) using the prompt word paradigm of an LLM to determine the type of a cell in the JSON data and supplement cell type information; the cell type includes a row title cell, a column title cell and a data cell; (3) verifying the matching and connectivity of row and column positions according to the two basic principles of title-data relationship and row-column position relationship, thereby automatically constructing two sub-trees of a row tree and a column tree; (4) using a queue and a stack to assist in a tree traversal strategy to implement row and column dimension text conversion based on the sub-tree structure.

2. The method according to claim 1, wherein, For a multi-modal document containing a table, first, the multi-modal document is analyzed, then the table image is extracted and converted into structured JSON data.

3. The method according to claim 2, wherein, First, the multi-modal document is analyzed, then the table image is extracted and converted into structured JSON data, comprising: First, the multi-modal document is subjected to layout detection inference using a fine-tuned LayoutLMv3 model to locate different elements in the document: images, tables, text, titles, formulas; then, for all table elements TableMaster is used to define the boundaries of the table, and then PaddleOCR is used to recognize the table text content. Finally, all table elements of the table image are converted into a JSON spreadsheet format , facilitating further data processing and analysis, where n is the number of table elements.

4. The method according to claim 1, wherein, The step (2) specifically comprises the following sub-steps: For each table element corresponding to the JSON format , 1≤ i ≤ n , first, the cell in the JSON data is determined by the prompt word paradigm of the LLM column title, row title and data three functional roles, then after obtaining the type determination result of each cell, the specific JSON file is rewritten and supplemented with the corresponding cell type by using the auxiliary code, and finally the supplemented JSON format is obtained.

5. The method of claim 1, wherein the method further comprises: The step (3) specifically comprises the following sub-steps: (3.1) Firstly, the column tree is constructed. All column header cells are found according to the title-data relationship, and the column position matching and connectivity are verified according to the row-column position relationship in the table, and they are reasonably connected to the tree structure. Subsequently, other types of cells associated with the column header cells are uniformly regarded as data at the same level, and then they are integrated as the child nodes of the column header cells at the leaf node position according to the row-column position relationship, and finally the column tree is constructed ; (3.2) Finally, the row tree is constructed. All row header cells are found according to the header-data relationship, and the row position matching and connectivity are verified according to the row-column position relationship in the table, and they are reasonably connected to the tree structure. Then, other types of cells that have a connection relationship with the row header cells are uniformly regarded as data at the same level, and then they are integrated as the child nodes of the row header cells at the leaf node position according to the row-column position relationship, and finally the row tree is constructed .

6. The method of claim 1, wherein the method further comprises: The step (4) is specifically: For column tree The column conversion is performed by first traversing the column tree using a depth-first traversal algorithm, outputting one or more layers of column titles in a queue form; secondly, for each data cell, the corresponding row title information is determined by tracing the row tree, one or more layers of row titles are stored in a stack operation mode, and then the collected row titles are traced to find the corresponding column titles by using the stack operation mode, and finally the ordered output is obtained; and finally, the column conversion text is obtained ; For row tree The row conversion is performed by first traversing the row tree using a depth-first traversal algorithm, and outputting one or more layers of row titles in a queue form; secondly, for each data cell, the corresponding column title information is determined by tracing the column tree, and one or more layers of column titles are stored in a stack operation mode, and then the collected column titles are traced back to the row tree to find the corresponding row title, and finally output in an ordered manner; and finally, the text after row conversion is obtained .

7. The method of claim 1, wherein the method further comprises: determining a tree structure of the complex table text; and determining a tree structure of the complex table text. If the step (4) performs row and column two-dimensional text conversion, it further comprises: optimizing the row and column two-dimensional text through the LLM prompt word engineering, and generating an optimized natural language description.

8. The method according to claim 7, wherein, Optimizing the row and column two-dimensional text through the LLM prompt word engineering, and generating an optimized natural language description, comprising: Text results for two different strategies and , using large models to make judgments from the logicality of information presentation, text readability, and information integrity, and optimizing the expression in the direction of more concise and logically clear, and then output the text content again.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the complex table text conversion method based on the tree structure according to any one of claims 1-8.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the complex table text conversion method based on the tree structure according to any one of claims 1-8.

Images