Complex table understanding method based on large language model

Abstract

The invention discloses a complex table understanding method based on a large language model. The complex table understanding method comprises the following steps: inputting a user question, an original table and an alternative operation pool into an LLM (Logical Language Model); selecting a series of operations required for generating answers from the alternative operation pool, generating and executing an overall planning path based on LLM, generating an execution result, and performing confidence self-evaluation on the generated LLM; if the score of the LLM confidence coefficient self-evaluation is greater than a set threshold value, the operation of the step is valid, if execution is finished, an answer is output and returned to the user, and if all the steps of planning are not executed, the next operation in the path planning is continued to be executed; and if the current operation is smaller than the set threshold value, adding the current operation and the execution result into the existing context prompt, and returning to the previous operation for re-planning. According to the method, the reasoning and planning capabilities of LLM are fully utilized, and the advantages of overall generation planning and dynamic generation planning are combined, so that efficient understanding, reasoning and generation of complex table data are realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a complex table understanding method based on a large language model. Background Art

[0002] Safety and risk management system audits are a crucial component of the evaluation criteria for power grid safety production risk management systems. Enterprise-level assessments are a core task of these audits, requiring the understanding of large amounts of complex spreadsheet data. Currently, enterprises primarily maintain, monitor, and analyze management results against benchmarks using Excel. This is hindered by the high workload and time-consuming nature of data statistics and maintenance, making it difficult to meet regular monitoring requirements. Furthermore, errors are prone to occur during the statistical, calculation, and aggregation processes of data-related indicators. With the increasing scale and intelligence of power systems, the emergence of risk identification and early warning capabilities based on deep learning technologies is providing reliable support for more efficient management. However, training deep learning models requires a large amount of domain data, and the power industry has accumulated a large amount of sensitive data. Due to data security concerns, this data is difficult to obtain, which limits algorithm performance. Furthermore, algorithm developers must possess a solid foundation of knowledge related to power grid safety management, a demanding requirement. In recent years, the emergence and development of large language models (LLMs) has made efficient management possible. LLM is trained based on large-scale data from various fields and demonstrates extraordinary advantages in semantic understanding and language generation. In terms of semantic understanding, it can capture subtle differences in text and understand both daily conversations and complex technologies in professional fields. In terms of content generation, it can output logically clear text to meet user needs in different scenarios and improve management efficiency.

[0003] The development of LLM-based table comprehension tasks has made it possible to efficiently process table comprehension tasks, relieving some workload on domain experts. While LLMs excel in table comprehension tasks, they lack real-time access to the latest knowledge about the power grid domain and are prone to generating "hallucinations" (i.e., generating content that is inconsistent with the facts). Furthermore, due to the large number of training parameters required for LLMs, retraining these models to incorporate the latest knowledge about the power grid domain is both expensive and impractical. Recently, several hinting methods have demonstrated excellent results in LLM-based table comprehension tasks. The accuracy of LLMs in answering table questions is not only related to their inherent reasoning capabilities but also, to a certain extent, influenced by their planning methods. These methods can be categorized into large-model holistic planning methods and large-model dynamic planning methods based on their different planning methods. Large-model holistic planning methods generate a complete table comprehension plan all at once upon receiving a user question, while large-model dynamic planning methods generate the next step of the plan incrementally based on the user question and current state feedback.

[0004] The GPT4Table method is a table comprehension approach for holistic planning. It breaks the table comprehension task into seven subtasks, including table size detection, cell lookup, and reverse lookup. The valid table structure and content information obtained in these subtasks is used as context before being added to the table and answering the question, forming a holistic planning process. LLM follows this process step by step to arrive at the final answer. This holistic planning approach, GPT4Table, generates a complete optimal path from the starting point to the end point. Its planning logic is relatively simple, making it easy to understand and implement. The resulting planning is relatively stable and suitable for static environments. However, it lacks adaptability and flexibility in the face of sudden changes in the environment. Furthermore, once the plan is complete, each step is executed according to the predetermined planned path, resulting in a large amount of unnecessary computation and increased computational cost. Therefore, if holistic planning is to be used to solve problems in real-world scenarios, careful consideration should be given to factors such as cost and efficiency.

[0005] The chain-of-table method dynamically generates plans. For a table comprehension problem, it employs a multi-round reasoning approach and leverages the LLM's planning capabilities to adaptively generate reasoning chains based on the user's input question. An operation pool is designed to assist in reasoning, containing a series of operations such as adding columns and selecting rows. The original table, the question, and an initialized operation chain are input. The operation chain is used to store the historical operations selected in each previous step of reasoning. At each step, the LLM is prompted to dynamically plan the next predefined operation from the operation pool based on the question, the current table content, and the historical operation chain. The operation is then executed on the current table, updating the table and operation chain, and then the next round of iterations is performed until the answer to the question is found. This multi-step iterative sampling approach progressively processes the table, effectively converting large and complex tables into tables specifically tailored to the problem, thereby obtaining the correct answer.

[0006] The ReAcTable table comprehension framework, which employs a similar concept to the chain-of-table approach, also dynamically generates plans. ReAcTable feeds the original table and user questions into the LLM, instructing it to think through the questions step by step. The LLM then determines whether to generate SQL queries or Python code related to the table, or directly answer the question. It then calls an external code executor to process the table and generate an intermediate result table. The SQL or Python code generated at each step of reasoning, as well as the latest intermediate result table, are added to the previous prompts and fed back to the LLM for the next round of reasoning iterations, ending when the LLM can answer the question. This iterative reasoning approach to transforming table data allows the LLM to undergo an observation-thinking-action process, enhancing its planning and analytical capabilities.

[0007] Both the chain-of-table and ReAcTable methods are table comprehension methods for dynamically generating plans for large models, which make up for the shortcomings of the overall generation planning method for large models. By generating plans step by step, LLM can dynamically generate the next plan based on real-time feedback results, which can respond quickly in dynamically changing scenarios and is highly flexible. However, designing a dynamic planning framework requires a complex dynamic adjustment mechanism, and each planning step relies on feedback. If a plan step in the dynamic generation plan produces an illusion, there is a risk of generating a local optimal path, which may make it difficult to achieve the global optimal solution. It may also gradually deviate from the correct planning path, and it is impossible to obtain an accurate answer.

[0008] Both holistic and dynamic generative planning have their own advantages and disadvantages. Large-model holistic planning is more suitable for static or less dynamic scenarios, providing a stable global optimal solution. However, it suffers from poor adaptability and flexibility. Large-model dynamic generative planning is suitable for dynamically changing environments, allowing for real-time adjustments to the planned path. This provides strong adaptability, but the planning process is complex and may expose the risk of reaching a local optimum. To achieve effective results in real-world scenarios, using either holistic or dynamic planning alone has certain drawbacks.

[0009] In view of this, a complex table understanding method based on a large language model is needed. Summary of the Invention

[0010] To address the problems existing in the prior art, the present invention provides a complex table understanding method based on a large language model. This method reduces unnecessary computation by generating a holistic plan and avoids repeated computation by dynamically generating a plan. In a dynamic environment, dynamic planning can quickly respond to environmental changes and adjust the planning path, thereby improving the efficiency of LLM table understanding. The specific technical solution is as follows:

[0011] A complex table comprehension method based on a large language model, including:

[0012] Input user questions, original forms, and alternative operation pools into LLM;

[0013] Select a series of operations required to generate an answer to the user question from the candidate operation pool, and generate an overall planning path at one time based on the LLM;

[0014] LLM executes each step of the overall planning path one by one, generates execution results, and performs LLM confidence self-assessment on the generated results;

[0015] If the LLM confidence self-assessment score is greater than the set threshold, it means that the result of this step is reliable and effective for subsequent planning, and it is determined whether all steps of the planning have been executed. If so, the answer is output and returned to the user. If not, the next step in the planning path is continued;

[0016] If the LLM confidence self-assessment score is less than the set threshold, the current operation and execution result are added to the existing context prompt, and the previous operation is returned to re-plan.

[0017] Preferably, the operation types in the candidate operation pool include table transposition, question rewriting, table compression, table conversion, table expansion and table structure enhancement prompts.

[0018] Preferably, the table transposition operation includes:

[0019] Enter the first row and first column of the original table into LLM;

[0020] Prompt LLM to determine whether the header of the original table is in the first row or the first column to determine the location of the header and constrain the output of LLM;

[0021] If the table header is in the first column, LLM outputs "Table needs to be transposed". If the table header is in the first row, LLM outputs "Table does not need to be transposed".

[0022] Preferably, the question rewriting operation includes question rewriting and RAG retrieval;

[0023] The question is rephrased to include:

[0024] Convert user questions into semantic vectors, perform semantic similarity search on the fragments in the vectorized external knowledge base, extract the fragments with high similarity and input them into the LLM together with the original user questions;

[0025] Prompt LLM to rewrite the question based on the fragment and output the original user question and the rewritten question;

[0026] The RAG search includes:

[0027] The rewritten question is vectorized and encoded, and the cosine similarity is calculated with the content fragments in the vectorized external knowledge base. All the similarity results are ranked, and the text fragment corresponding to the first-ranked value is extracted as the context information for answering the question.

[0028] Preferably, the table compression operation includes a task of retrieving relevant columns and a task of retrieving relevant rows;

[0029] The task of retrieving related columns includes:

[0030] Input all the column names of the table into the LLM, treat the relevance of each column to the problem as a binary classification task, ask the LLM whether the column is relevant to the problem, delete irrelevant columns, output the column names relevant to the problem, the LLM outputs all the retrieved column names, calls the external Python executor to perform operations on the entire table, and updates the intermediate table after column reduction to the next task;

[0031] The task of retrieving related rows includes:

[0032] Input all the column names of the table into the LLM, treat the relevance of each column to the problem as a binary classification task, ask the LLM whether the column is relevant to the problem, delete irrelevant columns, output the column names relevant to the problem, the LLM outputs all the retrieved column names, calls the external Python executor to perform operations on the entire table, and updates the intermediate table after column reduction to the next task;

[0033] The task of retrieving related rows includes:

[0034] The intermediate table obtained after retrieving the relevant columns and the question are input into LLM, prompting LLM to retrieve the row index output related to the question.

[0035] Preferably, the table conversion operation includes a table grouping task and a table row sorting task:

[0036] The table grouping task includes:

[0037] Group data according to the value of a column to perform aggregate statistics;

[0038] The table row sorting task includes:

[0039] For problems that require numerical calculations and statistical analysis, the LLM is prompted to determine whether it is necessary to sort all rows in a certain column of the table based on the problem. If necessary, the sorting task is performed and the sorted table is returned.

[0040] Preferably, the table expansion operation includes:

[0041] According to the needs of answering user questions, specific columns are added to the table to present relevant information in a more structured form, thereby improving LLM search efficiency.

[0042] Preferably, the table structure enhancement prompt operation includes a table size detection task, a merged cell detection task, a table boundary detection task, and a column data type annotation task.

[0043] Preferably, the confidence score of the LLM confidence self-assessment is limited to the interval 0-1, and a confidence threshold of 0.9 is set.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention simultaneously balances globality, reliability, and flexibility. It combines overall generation planning with dynamic generation planning to achieve the fusion of global and local information. Overall planning provides the global optimal path, while dynamic planning adjusts local paths based on real-time feedback. Overall generation planning reduces unnecessary calculations, while dynamic generation planning avoids repeated calculations. In a dynamic environment, dynamic planning can quickly respond to environmental changes and adjust the planned path, improving the efficiency of LLMs in table understanding.

[0046] 2. The present invention takes into account both efficiency and accuracy. Among the six alternative operations designed, the table transposition operation analyzes the table header position to determine whether to convert the list into a more readable row table, which can enhance the accuracy of LLM answers. The question rewriting operation uses RAG technology and combines external knowledge bases to rewrite the question into a format that is easier to analyze and reason through semantic analysis of the question, and extracts context information that is highly relevant to the question from the external knowledge base. The table compression operation narrows the search scope, removes redundant information that is irrelevant to the question, and focuses more on content related to the question, thereby improving retrieval efficiency. The table conversion operation uses the reasoning and analysis capabilities that LLM excels at to better understand the hierarchical structure of data points and enhance the accuracy of tasks related to numerical calculation and analysis. The table expansion operation presents free-form contextual information in a structured form, thereby improving LLM retrieval efficiency. The four tasks designed for the table structure enhancement operation help LLM enhance its understanding of table structure information and improve the accuracy of answering questions. For complex table comprehension questions, any combination of these alternative operations can enhance the efficiency and accuracy of LLM in table comprehension questions and answers to a certain extent.

[0047] 3. The LLM confidence self-assessment method designed in this invention makes LLM planning analysis more reliable. The confidence level of each analysis result is evaluated. In certain scenarios, low-reliability results can be responded to with responses such as "I can't answer" or "I'm not sure" based on the confidence threshold, avoiding deviations from the correct planning path and the generation of incorrect responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for the specific embodiments or the description of the prior art. Similar elements or parts are generally identified by similar reference numerals throughout the drawings. Elements or parts in the drawings are not necessarily drawn to scale.

[0049] Figure 1 Flow chart of the method of the present invention.

[0050] Figure 2 This is an example diagram of the combination of the overall generation planning and dynamic generation planning based on LLM of the present invention.

[0051] Figure 3 This is a block diagram of the alternative operation pool of the present invention.

[0052] Figure 4 An example diagram of operations rewritten for the problem of the present invention.

[0053] Figure 5 This is an example diagram of the table compression operation of the present invention.

[0054] Figure 6 This is an example diagram of the table conversion operation of the present invention.

[0055] Figure 7 This is an example diagram of the table expansion operation of the present invention.

[0056] Figure 8 This is an example diagram of prompt words for the table structure enhanced prompt operation of the present invention.

[0057] Figure 9 This is an example of the table structure enhancement prompt operation of the present invention. DETAILED DESCRIPTION

[0058] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0059] It will be understood that when used in this specification and the appended claims, the terms “comprises” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0060] It should also be understood that the terms used in the present specification are only for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0061] It should be further understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0062] Example

[0063] See also Figure 1 , a complex table understanding method based on a large language model, comprising: inputting user questions, original tables and alternative operation pools into the LLM; selecting a series of operations required to generate answers from the alternative operation pool for the user questions, and generating an overall planning path based on the LLM at one time; the LLM executes each step of the overall planning path one by one, generates an execution result, and performs an LLM confidence self-assessment on the generated results; if the LLM confidence self-assessment score is greater than the set threshold, it means that the result of this step is reliable and effective for subsequent planning, and judges whether all steps of the planning have been completed. If completed, the answer is output and returned to the user. If not all steps of the planning have been completed, the next operation in the planning path is continued; if the LLM confidence self-assessment score is less than the set threshold, the current operation and execution result are added to the existing context prompt, and the previous step is returned to re-planning.

[0064] In the above scheme, if Figure 1 The technical solution process shown in the figure is that the user inputs a natural language question, and LLM is based on the input user question Q, the original table T and the candidate operation pool F, which contains n candidate operations, represented as, F = {f1, f2, ..., f n}Select a series of operations required to generate the answer from the alternative operation pool F, and generate an overall planning path R at one time based on LLM, such as R{f1,f3,f4,f n-1 LLM executes each step of the overall planning path one by one, generates an execution result, and performs LLM confidence self-evaluation on the generated result. If the LLM confidence self-evaluation score is greater than the set threshold, it means that the result of this step is reliable and valid for subsequent planning, and determines whether all steps of the plan have been executed. If they have been executed, the answer is output and returned to the user. If all steps of the plan have not been executed, the next operation in the planning path is continued. If the LLM confidence self-evaluation score is less than the set threshold, the current operation and execution result are added to the existing context prompt, and the previous operation is returned to re-plan. LLM is required to reflect on the cause of the incorrect planning and correct the planning path to avoid the repetition of the path re-planned by LLM with the path that previously produced an incorrect result, until all operations are executed and the answer is output. This process of self-evaluation and re-planning of the execution results of each operation step in the overall planning generates a dynamic planning path, ensuring that each step of LLM's reasoning before answering the question is rigorous and reliable, constraining the uncertainty of LLM in the intermediate reasoning steps, avoiding the generation of unreliable results, and avoiding falling into local optimal solutions.

[0065] As attached Figure 2 The example shown is a combination of overall generation planning and dynamic generation planning based on LLM. The input user question Q is "How many senior reviewers are currently valid in Guangxi Power Grid Company?" The original table T contains 6 columns: "name", "age", "professional title", "company", "level", and "validity period". The alternative operation pool F contains 6 operations: table transposition, question rewriting, table compression, table conversion, table expansion, and table structure enhancement prompts. The threshold of the confidence self-assessment score is set to 0.9. According to the input {Q, T, F}, LLM generates an overall planning path R = {table compression, table expansion, table transposition, table structure enhancement}. LLM executes each operation of the overall planning path R, executes the table compression operation to generate the result and the confidence self-assessment score of 0.6. The LLM confidence self-assessment score is less than the set threshold of 0.9, and it is necessary to return to the previous operation for re-planning. However, the table compression operation is the first operation in the overall planning path. Therefore, returning to the previous operation for re-planning is equivalent to re-planning the operation path from the beginning. The corrected planning path by LLM is R1 = {question rewrite, table compression, table expansion, table structure enhancement prompt}. LLM executes the corrected planning path R1, performing question rewrite and table compression operations. The confidence self-assessment scores are both greater than the set threshold of 0.9. It then continues with table expansion to generate an intermediate result and a confidence self-assessment score of 0.85. The confidence self-assessment score is less than the set threshold, and the path is replanned by returning to the previous step of table compression. The already executed planning path is {question rewrite, table compression}. The corrected planning path by LLM is R2 = {table conversion, table structure enhancement prompt}. LLM executes the table conversion and table structure enhancement prompt operations in R2. The confidence self-assessment scores of the generated results are both greater than the set threshold, and no further replanning is required. At this point, all operations in the planning path have been completed. Therefore, before retrieving the answer and outputting the final result, the effective planning path executed by LLM is {question rewrite, table compression, table conversion, table structure enhancement prompt}.

[0066] As a further example, Figure 3 As shown in Figure 2, the alternative operation pool is a set of flexible basic tools or operations that can be combined to form more complex solutions. Alternative operation pool F includes six types of operations, each focusing on a single function: table transposition, question rewriting, table compression, table conversion, table expansion, and table structure enhancement prompts.

[0067] As a further embodiment, the table transposition operation includes: inputting the contents of the first row and first column of the original table into the LLM; prompting the LLM to determine whether the header of the original table is in the first row or the first column to determine the position of the header and constrain the output of the LLM; if the header is in the first column, the LLM outputs "the table needs to be transposed", and if the header is in the first row, the LLM outputs "the table does not need to be transposed".

[0068] As a further embodiment, question rewriting includes: converting user questions into semantic vectors, performing semantic similarity search on fragments in the vectorized external knowledge base, extracting fragments with high similarity and inputting them into LLM together with the original user questions; prompting LLM to rewrite the questions based on the fragments, and outputting the original user questions and the rewritten questions; RAG retrieval includes: vectorizing and encoding the rewritten questions, calculating cosine similarity with content fragments in the vectorized external knowledge base, sorting all similarity results obtained, and extracting the text fragment corresponding to the first-ranked value as context information for answering the question.

[0069] As you can understand, the question rewriting operation uses an external knowledge base to refine and rewrite the user's question, retrieve relevant context, and help the LLM better understand the question and infer answers. If the user asking the question is unfamiliar with LLM input techniques, the question may be vague and uncontrollable, making it impossible for the LLM to capture the user's true intent. Or the question may contain technical terms that are difficult for the LLM to understand, resulting in an inability to match the user's question with the table content. Furthermore, answering questions requires the use of a real-time external knowledge base. Directly extracting relevant context from this vast external knowledge base may introduce redundant context, interfering with the LLM's retrieval and judgment, leading to retrieval failures or the return of results irrelevant to the question. Furthermore, excessively long context not only increases inference time but also the cost of using the LLM. Therefore, to make it easier for the LLM to understand questions and retrieve answers faster, a question rewriting operation is designed to improve the relevance and accuracy of LLM retrieval results. The question rewriting operation includes two subtasks: question rewriting and RAG retrieval.

[0070] The question rewriting task converts user questions into semantic vectors, performs a semantic similarity search on the vectorized fragments in the external knowledge base, extracts highly similar fragments, and feeds them into the LLM together with the original user question. The LLM is prompted to rewrite the question based on the fragments and outputs both the original user question and the rewritten question. This semantic reconstruction allows the LLM to understand domain-specific knowledge, making the question more detailed and relevant to the table content, thereby increasing the likelihood of retrieving relevant information.

[0071] As attached Figure 4In the example of the question rewriting operation shown, the original user question was "What is the diamond level of senior reviewers at the Guangxi Power Grid Company?" The LLM did not retrieve information related to "diamond level" in the provided table, resulting in failure to answer the question. However, when provided with an external knowledge base, the LLM learned that "diamond level" is associated with "number of senior reviewers" and could rewrite the user question to "How many senior reviewers currently have valid terms at the Guangxi Power Grid Company? What is the corresponding diamond level?" It is important to note that not all questions require rewriting. Rewriting questions that can be directly answered from the table will complicate the reasoning process and waste resources. Therefore, before the question rewriting task, the entire contents of the first five and last five rows of the table, the table column names, and the user question are input to the LLM. This guides the LLM to consider whether it can directly answer the question. If not, the question needs to be rewritten.

[0072] The RAG retrieval task vectorizes the rewritten question, calculates cosine similarity with content snippets in the vectorized external knowledge base, ranks all similarity results, and extracts the text snippets corresponding to the top 1 values ​​as context for answering the question. For specific scenarios in the power grid sector, RAG retrieval is performed on background knowledge related to safety production risk management system reviews, effectively filtering out content with low relevance to the question while retaining the truly relevant content as valid context for subsequent tasks. For example, when counting the number of senior reviewers at the Guangxi Power Grid Company, the RAG retrieval task reveals that senior reviewers include senior A-level reviewers, senior B-level reviewers, and senior C-level reviewers.

[0073] As a further embodiment, the table compression operation includes a task of retrieving relevant columns and a task of retrieving relevant rows;

[0074] The task of retrieving relevant columns includes: inputting all column names of the table into the LLM, treating the relevance of each column to the question as a binary classification task, asking the LLM whether the column is relevant to the question, deleting irrelevant columns, outputting the column names relevant to the question, the LLM outputting all retrieved column names, calling an external Python executor to perform operations on the entire table, and updating the intermediate table after column reduction to the next task;

[0075] The task of retrieving relevant rows includes: inputting all column names of the table into the LLM, treating the relevance of each column to the question as a binary classification task, asking the LLM whether the column is relevant to the question, deleting irrelevant columns, outputting the column names that are relevant to the question, the LLM outputting all retrieved column names, calling an external Python executor to perform operations on the entire table, and updating the intermediate table after column reduction to the next task;

[0076] The task of retrieving related rows includes: inputting the intermediate table obtained after retrieving related columns and the question into the LLM, and prompting the LLM to retrieve the row index output related to the question.

[0077] As can be understood, table compression helps the LLM focus on table content that is more relevant to the user's question, improving its accuracy and efficiency in complex table reasoning tasks. On the one hand, due to the LLM's context length limit, it cannot input all the content of large tables, which may result in the loss of important information relevant to answering the question. On the other hand, using the entire table as input does not necessarily require accessing the entire table content to answer every question. This makes it inefficient for the LLM to retrieve answers from such a large amount of information, affecting the output results. Therefore, it is necessary to effectively compress the table content before answering the question, using table compression to narrow the search scope. Table compression involves two tasks: retrieving relevant columns and retrieving relevant rows. Since the LLM cannot guarantee high accuracy in numerical calculations, after executing the corresponding operations and outputting key result information, it calls an external executor to perform operations on the table to generate an intermediate result table.

[0078] The task of retrieving relevant columns inputs all column names of the table into LLM, treats the relevance of each column to the question as a binary classification task, asks LLM whether the column is relevant to the question, deletes irrelevant columns, outputs the column names relevant to the question, LLM outputs all retrieved column names, calls an external Python executor to perform operations on the entire table, and updates the intermediate table after column reduction to the next task. Figure 5 The table compression operation shown in the figure is an example of the user's question: "How many senior reviewers are currently valid in Guangxi Power Grid Company? What is the corresponding diamond level?" After searching for relevant columns, the intermediate result table T1 deletes the "name", "age", and "professional title" columns that are not relevant to the question in the original table T0, and retains the "company", "level", and "validity period" columns that are relevant to the question.

[0079] The task of retrieving related rows inputs the table and question obtained after retrieving related columns into LLM, prompting LLM to retrieve the row index output related to the question. Figure 5 In the example of the table compression operation shown, the user question contains the condition "senior reviewers who are currently within the validity period". Among all the rows in the validity period column of the intermediate result table T1 output by the task of retrieving related columns, some rows are not within the validity period. Therefore, the intermediate result table T2 deletes all the contents of the row index that are not within the validity period through the task of retrieving related rows.

[0080] As a further embodiment, the table conversion operation includes a table grouping task and a table row sorting task:

[0081] Table grouping tasks include:

[0082] Group data according to the value of a column to perform aggregate statistics;

[0083] Table row sorting tasks include:

[0084] For problems that require numerical calculations and statistical analysis, the LLM is prompted to determine whether it is necessary to sort all rows in a certain column of the table based on the problem. If necessary, the sorting task is performed and the sorted table is returned.

[0085] As you can understand, table transformation operations include table grouping and table row sorting. LLM is not good at numerical calculations and often outputs incorrect results, which can easily affect the answering of questions. Designing table transformation operations allows LLM to use its inherent reasoning and analytical capabilities to analyze whether the operation is necessary, output the required parameters, extract the parameters from the LLM output, and call an external executor to perform the actual processing on the table.

[0086] The table grouping task helps LLM better understand the hierarchical structure of data. Group data based on the value of a column to perform aggregate statistics. Figure 6 The example of the table conversion operation shown is that the question asks which company has the largest number of senior reviewers. The context information provided is "senior reviewers include senior A-level reviewers, senior B-level reviewers, and senior C-level reviewers." The input table includes two columns: "Company" and "Level." To find the company with the largest number of senior reviewers, it is necessary to first count the number of senior reviewers in each company. The LLM is prompted to perform a grouping task on the table based on the question and context information, and group senior reviewers belonging to the same company through the "Company" column. A new intermediate result table is obtained, which contains information on the number of companies and senior reviewers, further simplifying the difficulty of reasoning.

[0087] The table row sorting task is aimed at some problems that require numerical calculations and statistical analysis. It prompts LLM to determine whether it is necessary to sort all the rows of a certain column in the table according to the problem. If necessary, it executes the sorting task and returns the sorted table. Figure 6 In the example of the table conversion operation shown, a table containing "Company" and "Number of Senior Reviewers" has been obtained through the previous table grouping task. To find the company with the largest number of senior reviewers, it is necessary to perform a row sorting task on all values ​​in the "Number of Senior Reviewers" column and update the table. At this time, the answer to the question can be quickly retrieved.

[0088] As a further embodiment, the table expansion operation includes: adding specific columns to the table according to the needs of answering user questions, reflecting relevant information in a more structured form, and improving the LLM retrieval efficiency. It can be understood that the table expansion operation adds specific columns to the table according to the needs of answering user questions, reflecting relevant information in a more structured form, and improving the LLM retrieval efficiency. Although contextual information highly relevant to the question has been extracted from a huge external knowledge base and the context for assisting in answering the question has been simplified, expanding the inference context information into a table in a structured form is clearer than free-form text, making the retrieval more accurate and making it easier for LLM to quickly retrieve the correct answer. The table expansion operation includes a task of adding columns to the table. The task of adding columns to the table prompts LLM to add specific columns based on the contextual information to store intermediate inference results.

[0089] As attached Figure 7 The following table expansion example uses the question "What is the diamond level of senior reviewers at each company?" The table contains two columns: "Company" and "Number of Senior Reviewers." The context provides criteria for assessing the "Diamond Level of Senior Reviewers at a Company" based on the number of senior reviewers. After executing the "Add Column to Table" task based on the question, table, and context, the "Diamond Level of Senior Reviewers" column and the cell contents in that column are added to the table, allowing LLM to easily retrieve the answer.

[0090] As a further embodiment, the table structure enhancement prompt operation includes a table size detection task, a merged cell detection task, a table boundary detection task, and a column data type annotation task. It can be understood that the table structure enhancement prompt operation is designed to enhance the LLM's table structure understanding ability. When the table structure is more complex, for example, there is a merged cell structure in the table, which interferes with the LLM reasoning, it is necessary to enhance the LLM's table structure understanding ability. Combining the ideas of the table understanding method GPT4Table, the table structure enhancement prompt operation is designed to have 4 tasks to help understand the table structure: table size detection, merged cell detection, table boundary detection, and column data type annotation.

[0091] Attachment Figure 8 Examples of hints for enhancing table structure prompts. To ensure more accurate output, constraints are added to the output format. For example, the table size detection task includes the instruction "Please answer the question using the following format," and the merged cell detection task includes the instruction "Please use | to split the answer (e.g., (2, 3) | (3, 3)). Column indexes start at 0. If there is no answer, return "No merged cells in the table." The output format is specified in the table boundary detection task, and examples of output format are provided in the column data type annotation task.

[0092] Attachment Figure 9 The example table structure enhancement operation shown here adds the table structure information derived from each task as supplementary contextual cues before the table content. This structural information constraint prevents the LLM from incorrectly analyzing the table structure and causing inference errors. The table size detection task prompts the LLM to identify the number of rows and columns in the table, outputting "the table has r rows and c columns," where r represents the number of rows and c represents the number of columns. The table merge cell detection task prompts the LLM to check whether there are merged cells in the table. If so, it outputs the merged cell's position index "(r1,c1)|(r2,c2)" and the merged cell's content. (r1,c1) indicates that the top left corner of the merged cell is in row r1, column c1, and (r2,c2) indicates that the bottom right corner of the merged cell is in row r2, column c2. The table boundary detection task identifies the table boundaries by adding labels, normalizing the table format, and adding structural descriptions at the beginning and end of the table. The "[Table Start]" label is inserted before the table and the "[Table End]" label is inserted after the table. Row separators are inserted between rows to indicate row boundaries, and the labeled table is output. The column data type annotation task prompts the LLM to analyze the data type of each column and add a data type annotation. For example, the data type of the "Number of Senior Reviewers" column is numeric (unit: ). Relevant internal structural information is extracted from the original table input and incorporated into the LLM input as a hint, providing additional structural knowledge constraints and improving the LLM's reasoning ability in subsequent table reasoning tasks.

[0093] As a further embodiment, the LLM confidence self-assessment method can effectively enhance the reliability of LLM's planning analysis. When executing each operation step of the overall planning path, the LLM is required to perform a self-confidence assessment on the result of that step through a prompt. The output confidence score is required to be limited to the interval 0-1, and a confidence threshold of 0.9 is set. Based on the confidence result, it is determined whether the planning scheme needs to be dynamically re-adjusted. When the confidence of the LLM self-assessment exceeds the set threshold, that is, the LLM's evaluation of the operation of this step has met the specified requirements, no further planning is required, and the next step of the overall planning can be entered for reasoning iteration. Otherwise, it is determined that the reasoning result of this time is incorrect, and the user question, the original planning path, and the operation results of all completed steps (including the incorrect results and confidence of this round of reasoning) are added to the prompt, prompting the LLM to re-plan the scheme from the previous step and not to generate the same planning scheme as the current one.

[0094] like Figure 2In the example shown in the combination of LLM-based overall generation planning and dynamic generation planning, the planning path R1 {question rewriting, table compression, table expansion, table structure enhancement prompt}, the original user question is "How many senior reviewers are currently valid in Guangxi Power Grid Company?", after the question rewriting operation, LLM retrieved the relevant context information to answer the question "Senior reviewers include senior A-level reviewers, senior B-level reviewers, and senior C-level reviewers", and the original table T deleted the "name", "age", and "professional title" columns that are not related to the question after the table compression operation, and the remaining "company", "level", and "validity period" columns were deleted. The relevant rows that are not within the validity period were deleted, and an intermediate result table was generated. The LLM confidence self-assessment scores of the question rewriting operation and the table compression operation are both greater than the set threshold of 0.9. Based on the question, the retrieved relevant context information and the intermediate result table, LLM performs a table expansion operation and outputs "This step does not require adding new columns, only the number of rows that meet the conditions needs to be counted, and the confidence self-assessment score is 0.85". The confidence self-assessment score is less than the set threshold of 0.9, so this plan is unreasonable and it is necessary to return to the previous step of table compression operation for re-planning. The current user question "How many senior reviewers are currently valid in Guangxi Power Grid Company?", relevant context information, the original planning path R1 {question rewriting, table compression, table expansion, table structure enhancement prompt}, the completed planning path {question rewriting operation, table compression operation} and the answer result of this time are added to the prompt, prompting LLM to reflect on the reasons for the incorrect planning and re-plan, and use "Please re-plan the operation plan, do not generate a planning scheme that is the same as the original planning scheme" to limit the output plan.

[0095] In addition, in order to verify the effectiveness of the solution proposed in this invention, baseline method performance comparison tests were conducted on the test sets of two common complex table comprehension benchmark datasets, WikiTQ and TabFact. WikiTQ is a table question-answering dataset that contains a large number of "question-answer" pairs. Most of the "question-answer" pairs are answers to questions with shorter text spans. This dataset is used to fully verify the model's table question-answering capabilities. TabFact is a table fact verification dataset that asks the LLM to give a correct or incorrect judgment result for a statement about the content of a table. This dataset is used to verify the model's ability in table fact verification tasks. The chain-of-table, ReAcTable, and GPT4Table methods were selected as the main baseline methods. In addition, the Binder method integrates language models with external tools such as Python, allowing the language model to generate executable programs, and external tools call the executable programs. The Dater method improves task accuracy by decomposing tables and questions and breaking down the task into multiple fixed steps. The ideas of the Binder and Dater methods are more advanced in complex table comprehension tasks, so Binder and Dater are also used as baseline methods for comparison in this experiment. To align with other baseline methods, we used the GPT-3.5 series model as the inference model. We used execution accuracy as the performance metric to compare the difference between the LLM's predictions of table content and the actual results.

[0096] According to the comparative test results of table understanding methods shown in Table 1, on the WikiTQ benchmark test set, before the method of the present invention, the execution accuracy of the chain-of-table method reached 59.94%, which is the best among all methods. The execution accuracy of the table question and answer task using the method of the present invention is 62.0%, which is 2.06% higher than that of the chain-of-table method, proving that the method combining LLM overall generation planning and dynamic generation planning is feasible in complex table understanding tasks. On the TabFact benchmark test set, the execution accuracy of the method of the present invention reached 79.6%, surpassing a series of methods such as ReacTable and second only to the effect of the Chain-of-table method, reflecting the versatility of the method of the present invention in complex table understanding related tasks, that is, whether it is a task about table question and answer or a task about table fact verification, the method proposed by the present invention has advantages.

[0097] Table 1 Comparative test results of table comprehension methods

[0098]

[0099]

[0100] Those skilled in the art will appreciate that the units of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0101] In the embodiments provided by the present invention, it should be understood that the division of units is merely a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0102] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nly Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc., various media that can store program code.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

Claims

1. A complex table understanding method based on a large language model, characterized in that: include: Input user questions, original forms, and alternative operation pools into LLM; Select a series of operations required to generate an answer to the user question from the candidate operation pool, and generate an overall planning path at one time based on the LLM; LLM executes each step of the overall planning path one by one, generates execution results, and performs LLM confidence self-assessment on the generated results; If the LLM confidence self-assessment score is greater than the set threshold, it means that the result of this step is reliable and effective for subsequent planning, and it is determined whether all steps of the planning have been executed. If so, the answer is output and returned to the user. If not, the next step in the planning path is continued; If the LLM confidence self-assessment score is less than the set threshold, the current operation and execution result are added to the existing context prompt, and the previous operation is returned to re-plan.

2. A complex table understanding method based on a large language model according to claim 1, characterized in that: The operation types in the candidate operation pool include table transposition, question rewriting, table compression, table conversion, table expansion and table structure enhancement prompts.

3. A complex table understanding method based on a large language model according to claim 2, characterized in that: The table transposition operation includes: Enter the first row and first column of the original table into LLM; Prompt LLM to determine whether the header of the original table is in the first row or the first column to determine the location of the header and constrain the output of LLM; If the table header is in the first column, LLM outputs "Table needs to be transposed". If the table header is in the first row, LLM outputs "Table does not need to be transposed".

4. A complex table understanding method based on a large language model according to claim 2, characterized in that: The question rewriting operation includes question rewriting and RAG retrieval; The question is rephrased to include: Convert user questions into semantic vectors, perform semantic similarity search on the fragments in the vectorized external knowledge base, extract the fragments with high similarity and input them into the LLM together with the original user questions; Prompt LLM to rewrite the question based on the fragment and output the original user question and the rewritten question; The RAG search includes: The rewritten question is vectorized and encoded, and the cosine similarity is calculated with the content fragments in the vectorized external knowledge base. All the similarity results are ranked, and the text fragment corresponding to the first-ranked value is extracted as the context information for answering the question.

5. A complex table understanding method based on a large language model according to claim 2, characterized in that: The table compression operation includes a task of retrieving relevant columns and a task of retrieving relevant rows; The task of retrieving related columns includes: Input all the column names of the table into the LLM, treat the relevance of each column to the problem as a binary classification task, ask the LLM whether the column is relevant to the problem, delete irrelevant columns, output the column names relevant to the problem, the LLM outputs all the retrieved column names, calls the external Python executor to perform operations on the entire table, and updates the intermediate table after column reduction to the next task; The task of retrieving related rows includes: Input all the column names of the table into the LLM, treat the relevance of each column to the problem as a binary classification task, ask the LLM whether the column is relevant to the problem, delete irrelevant columns, output the column names relevant to the problem, the LLM outputs all the retrieved column names, calls the external Python executor to perform operations on the entire table, and updates the intermediate table after column reduction to the next task; The task of retrieving related rows includes: The intermediate table obtained after retrieving the relevant columns and the question are input into LLM, prompting LLM to retrieve the row index output related to the question.

6. A complex table understanding method based on a large language model according to claim 2, characterized in that: The table conversion operation includes table grouping tasks and table row sorting tasks: The table grouping task includes: Group data according to the value of a column to perform aggregate statistics The table row sorting task includes: For problems that require numerical calculations and statistical analysis, the LLM is prompted to determine whether it is necessary to sort all rows in a certain column of the table based on the problem. If necessary, the sorting task is performed and the sorted table is returned.

7. The complex table understanding method based on a large language model according to claim 2, characterized in that: The table expansion operation includes: According to the needs of answering user questions, specific columns are added to the table to present relevant information in a more structured form, thereby improving LLM search efficiency.

8. The complex table understanding method based on a large language model according to claim 2, characterized in that: The table structure enhancement prompt operation includes a table size detection task, a merged cell detection task, a table boundary detection task, and a column data type annotation task.

9. The method for complex table comprehension based on a large language model according to claim 1, wherein the confidence score of the LLM confidence self-assessment is limited to the interval 0-1, and a confidence threshold of 0.9 is set.

Images