Table question and answer method and system based on large language model

Abstract

The invention discloses a table question and answer method and system based on a large language model. The table question-answering system is a system capable of understanding table data (table-form data such as excel files, csv files and database db files) and carrying out question-answering, data query statistics and analysis in combination with tables. The invention provides a table question-answering method based on a large language model, which comprises a table information extraction module, a question detailed rewriting module, a solution and code generation module, a Rewad model scoring evaluation module, a code security detection module, a code executor module, a summary answer generation module and the like. Safety and stability are achieved, and the ability of a large language model in table data analysis question and answer is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a table question-answering method and system based on a large language model. Background Art

[0002] With the rapid development of large language models (LLM), the launch of advanced models such as ChatGPT, Claude, LLaMA and Qwen has brought broad industry opportunities for artificial intelligence (AI) applications. These models have promoted automation and intelligence in many fields through natural language processing and understanding.

[0003] Tabular question answering systems have a wide range of applications in many fields, especially in industries such as finance, healthcare, and supply chain management that require extremely high data analysis accuracy. Advanced tabular question answering systems will become a key tool for improving efficiency and reducing decision-making risks. Although large language models have made significant breakthroughs in natural language understanding and generation, they still face many challenges when processing structured data.

[0004] Tabular data often contains complex multi-dimensional information, involving cross-row and cross-column relationship analysis, cross-table query, and fuzzy field understanding. These challenges lead to poor performance of large language models in the following aspects: Table semantic understanding: When interpreting complex table structures and their contextual relationships, large language models often find it difficult to accurately capture nested relationships and dependencies between fields; Multi-table linkage analysis: When users' questions involve related queries of multiple tabular data, it is difficult for the model to accurately derive and output fusion information; Advanced data operations: Existing tabular question-and-answer systems mainly focus on information extraction, but lack support for operations such as statistical analysis, visual presentation, and complex calculations. This greatly limits its scope of application when it comes to high-level data analysis scenarios. Therefore, how to enhance the deep understanding ability of large language models in tabular data processing and improve their output accuracy in complex queries, data analysis, and reasoning tasks has become the key to promoting the further development of AI in tabular data.

[0005] The Chinese patent (CN117972070A) discloses a method for large-model table question-answering. It proposes an automatic corpus generation scheme based on template design and rule formulation for large models to correctly understand the semantic problems of tables. The obtained corpus can be used to build a table question-answering knowledge base and fine-tune large models. It is based on cutting-edge technologies such as prompt learning and Lora fine-tuning to achieve data-driven large-model specific tasks with fewer resources. It improves the table question-answering capabilities of large models based on table serialization semantic analysis and question classification combined with rows and columns. This invention only involves simple table questions and answers. If the user's questions involve table calculations, statistical analysis, visualization, etc., the problem requirements cannot be handled, and it is difficult to handle complex multi-table data, so its use is very limited.

[0006] Chinese Patent (CN118245485A) A method, device and medium for processing tabular data with a large model, including: converting the user's natural language into SQL query to perform a tabular data query request; parsing the tabular task in the SQL query into corresponding operators to generate a coarse-grained calculation graph; using operator decomposition, operator combination, operator rearrangement, and combining the cost function to optimize the coarse-grained calculation graph to generate a fine-grained calculation graph; compiling into code according to the fine-grained calculation graph; executing the code to obtain a user response. The present invention can realize natural language interaction with the table, can realize functions such as extracting information, calculation, reasoning, etc., and has a stronger ability to understand and execute tabular tasks. This patent converts the user's natural language into SQL query statements and further improves the accuracy of the code through methods such as operator decomposition. It innovates in the generation of content and methods, but is limited to generating SQL queries and can meet users' needs for complex data analysis problems. However, this patent only generates code based on the problem, cannot output the solution process, and does not summarize the final answer based on the code execution results. It also has deficiencies in aspects such as code execution security verification, and is prone to generate dangerous deletions or huge computing power calculation operations, causing system crashes. Summary of the invention

[0007] The purpose of the present invention is to address the shortcomings of the prior art: the current table large model question and answer system has a low accuracy rate and cannot meet the needs of business intelligence and other fields for statistical analysis of table data and precise operations, and proposes a table question and answer method system based on a large model.

[0008] In a first aspect, the present invention provides a table question answering method based on a large language model, comprising the following steps:

[0009] Extraction of table data information:

[0010] Receive the table data uploaded by the user, parse the table data, extract the table name, column name and data type, and vectorize the column name and content value;

[0011] By calculating the similarity between the user question vector and the table data vector, the table columns and content values ​​most relevant to the user question are screened out;

[0012] User question adjustment:

[0013] Parse the questions input by users, extract keywords through natural language processing technology, and use the preset prompt to guide the large language model to rewrite the questions in detail to ensure that the questions are clear and have sufficient context information;

[0014] Solution and code generation:

[0015] Input the extracted table data and adjusted questions into the large language model to generate problem-solving ideas and corresponding Python or SQL codes;

[0016] Generate multiple candidate solutions through multiple samplings, score the candidate solutions using the Reward model, and select the solution with the highest score as the output;

[0017] Code execution and result generation:

[0018] Send the code that passes the security check to the code executor module, execute the code and capture the output results to provide data support for the generation of the final answer;

[0019] Combine the execution results to generate a summary answer:

[0020] The code execution results, problem-solving ideas, and the user's initial question are input into the large language model to generate the final summary answer and present it to the user, ensuring that the user obtains accurate and easy-to-understand query results.

[0021] In a second aspect, the present invention provides a table question answering system based on a large language model, comprising:

[0022] Data extraction module: used to receive and parse table data and vectorize the column names and content values;

[0023] Question adjustment module: used to rewrite the questions input by users in detail;

[0024] Code generation module: used to generate problem-solving ideas and corresponding Python or SQL codes;

[0025] Code execution module: used to execute code that passes security checks and capture output results;

[0026] Answer generation module: used to generate the final summary answer based on the code execution results.

[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the table question answering method based on a large language model is implemented.

[0028] Beneficial effects of the present invention: The present invention extracts tabular information related to user questions through a special data information extraction method, thereby avoiding too much tabular data to be input into the large model. Through task decomposition, the process of each step is optimized to improve the accuracy of tabular questions and answers, including detailed rewriting of questions, solution and code generation, and summary answers combined with code execution results. The innovative method of using a large language model and a Rewad model for interactive judgment improves the quality of the output of the large language model through a multiple sampling and scoring mechanism. By adding a code security detection module, the generated unsafe code is prevented from affecting the source data or database system. The tabular question and answer system based on a large language model of the present invention has the advantages of high accuracy, stable system, clear and specific output solutions, and traceable solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The present invention provides a process of a large-model-based table data question-answering system. DETAILED DESCRIPTION

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0031] The present application embodiment provides a table question-answering method based on a large model, comprising the following steps:

[0032] Step 1: Extracting table data information.

[0033] When faced with a large amount of structured tabular data, such as Excel files, CSV files, and database files (such as MySQL and SQLite), directly inputting the tabular data into the large language model will result in a long processing time, and may even cause the system to crash due to the large amount of input data. It may also exceed the input limit of the model, resulting in low processing efficiency. Therefore, an efficient data extraction module is needed to filter out the information most relevant to the user's question, thereby reducing the amount of input data and ensuring that the large language model can more accurately understand the user's query intent.

[0034] In some embodiments: First, the system receives the table file uploaded by the user, and performs a preliminary analysis on it to extract basic information, such as table name, column name, data type, etc. At the same time, the system detects whether the content in the column contains missing values ​​and determines the uniqueness of the data. All preprocessing information will be used in subsequent steps to improve data extraction efficiency. Then, vector encoding of column names and content values ​​is performed. This step not only vectorizes the column names of the table, but also embeds (embedding) vector encoding of the content values ​​of each column. This process is to more accurately match user questions with the fields and content in the table data in subsequent similarity calculations.

[0035] further:

[0036] Column name vectorization: Use pre-trained embedding models (such as BGE-M3, E5-embedding) to encode all column names of the table. For example, the table contains columns such as "customer name", "product category", and "sales". These column names will be encoded into vectors for similarity matching in the vector space.

[0037] Content value vectorization: Not only the column names are processed, but also the content values ​​in each column are vectorized. For example, if the "Product Category" column contains categories such as "Electronics", "Household Goods", etc., these content values ​​will also be converted into embedding vectors. This allows the system to have a deeper understanding of the semantic information of the data in the column, rather than relying solely on the column name for matching.

[0038] Assume the table is as follows:

[0039] |Customer Name|Product Category|Sales|Date|

[0040] |----------|------------|---------|------------|

[0041] |Zhang San|Electronic products|5000|2024-01-15|

[0042] |Li Si|Household Goods|12000|2024-02-12|

[0043] |Wang Wu|Electronic products|15000|2024-03-10|

[0044] This example vectorizes column names such as "Customer Name", "Product Category", and "Sales Volume", and also vectorizes content values ​​such as "Zhang San", "Electronic Products", and "5000" to better match user questions.

[0045] Next, question vectorization and similarity matching are performed. When a user asks a question, such as "What is the highest sales volume of electronic products in the past three months?", the system vectorizes and encodes the question. Subsequently, the system uses cosine similarity to calculate the similarity between the question vector and the vectors of column names and their content values ​​in the table, thereby filtering out the most relevant fields and data. For example, question vectorization: The user question "What is the highest sales volume of electronic products?" will be vectorized into a semantic vector. The system will not only match the question with the column name vector, but also compare it with the content value vector in the column. For example, by calculating the similarity with the content value of "electronic products", the system can quickly identify that "product category" and "sales volume" are the columns most relevant to the user's question.

[0046] Based on the similarity scores, the system selects the most relevant top-k column names and top-q content values ​​from the table.

[0047] At the same time, the following information is further extracted:

[0048] Data type: Identifies whether the column is character, numeric, or date data.

[0049] Content uniqueness: Determine whether the value in the column is unique (such as customer ID).

[0050] Missing value detection: Count the number and proportion of missing values ​​to assess the completeness of the data.

[0051] In some embodiments: In order to better adapt to different scales of table data, the system dynamically adjusts the extraction strategy of top-k column names and top-q content values ​​according to the size of the data set. This mechanism ensures that the system does not suffer performance degradation due to data overload when processing large tables, while ensuring the comprehensiveness of information extraction when processing small data sets. Further, the adjustment strategy is as follows:

[0052] Small column size (less than 10 columns): The system only selects the 3-5 most relevant columns.

[0053] Medium column size (10-50 columns): The system extracts the 5-10 most relevant columns.

[0054] Large column size (more than 50 columns): The system will extract 10-15 most relevant columns based on actual needs to ensure information coverage.

[0055] Content value adjustment: If the content of a column exceeds 100 records, the system will extract the top 10 most relevant content values ​​based on similarity instead of processing the entire column. If the content of a column is less than 100 records, the system will extract less than or equal to 10 content values.

[0056] The table data information extraction module can dynamically adjust the depth and breadth of data extraction according to the table size and user questions, ensuring efficient processing of various data sets. Through the dual vectorization of column names and content values, the system can understand complex problems more deeply and avoid relying solely on column names for superficial matching. In a large data set environment with multiple tables and multiple fields, the system can efficiently extract key information and ensure the simplicity and efficiency of input data.

[0057] Step 2: User problem adjustment.

[0058] When users interact with table question answering systems, they often encounter a common problem: the questions they ask are usually short or unclear, which makes it difficult to directly input them into a large language model (LLM) for understanding and processing. Especially in scenarios involving complex table data queries, statistical analysis, or cross-table associations, short questions often lack context and sufficient information, resulting in biased answers generated by the large language model, which may even fail to accurately meet user needs.

[0059] In order to effectively solve this problem, this step introduces the user question adjustment step. The core goal of this step is to use Prompt to guide the large language model to optimize and refine the user's initial question, making it more suitable for the specific scenario requirements of form question answering, while ensuring that the core semantics and intent of the question remain unchanged.

[0060] In some embodiments: First, the natural language question input by the user is preliminarily parsed, and keywords and key phrases are extracted through natural language processing (NLP) technology. Guided by keywords and prompts, the large language model rewrites the initial question in detail. This rewriting not only adds contextual information to the question, but also combines the characteristics of the table data to make the question clearer.

[0061] For example, "How are the sales this quarter?" can be rewritten as "Please count the sales of each product based on the data from the first quarter of 2024, and compare the sales changes from the same period last year." Furthermore, if the system detects that the user input is ambiguous during the question rewriting process and it is still not clear after the rewriting, the system will automatically trigger the follow-up mechanism. Through further interaction with the user, the system can collect more contextual information to ensure that the question is clearly stated. For example, when a user asks about "inventory status", the system will ask: "Do you want to view the current inventory of all products, or count them by product category?" The rewritten question will be presented to the user for confirmation to ensure that the rewritten version accurately reflects the user's intention. If the user has any objection to the rewriting result, the system will allow the user to fine-tune or redefine the question to ensure that the question finally input into the large language model has complete and clear semantic information.

[0062] Through this step, the table question-answering system of this embodiment can effectively improve the depth of understanding and processing efficiency of user questions, especially in complex data query and analysis scenarios, ensuring that the answers generated by the large language model are more accurate and meet actual needs. This not only improves the user experience, but also significantly improves the response accuracy and stability of the system.

[0063] Step 3: Solution and code generation

[0064] The core goal of this step is to generate an efficient and accurate solution based on tabular data. In this step, not only Python or SQL code is generated, but also detailed analysis of ideas is provided, so that users can clearly understand the steps to solve the problem, thereby better verifying the correctness and reliability of the answer.

[0065] Based on the table information extraction (step one) and question adjustment (step two), the system has obtained the table column data that is highly relevant to the user's question and the precise description of the user's needs. The system uses a carefully designed prompt template to input the above data and problem description into the large language model to guide it to generate solution ideas and code. When the model generates the answer, it will output the step-by-step solution ideas and corresponding code snippets at the same time. The purpose of this design is to help users understand the logic behind the generated code, rather than just seeing the code itself. For example, the model may first explain the solution ideas:

[0066] Question: Count the quarterly sales for the past year and find the highest quarter.

[0067] Thinking analysis: First, you need to extract the year and quarter from the "Date" field, then group and sum by quarter based on the "Sales" column, and finally find the quarter with the highest sales.

[0068] Generated Python code:

[0069] import pandas as pd

[0070] df=pd.read_csv('sales_data.csv')

[0071] df['date'] = pd.to_datetime(df['date'])

[0072] df['quarter'] = df['date'].dt.to_period('Q')

[0073] quarterly_sales = df.groupby('quarter')['sales'].sum()

[0074] highest_quarter=quarterly_sales.idxmax()

[0075] print(f"The quarter with the highest sales is: {highest_quarter}")

[0076] Furthermore, in order to improve the accuracy and stability of the output, the system will generate candidate solutions multiple times based on different sampling parameters (such as temperature, topK, topP, etc.). Specifically, the system will loop N samplings to generate different versions of solutions, and the answers generated each time may differ in terms of logical details, code implementation, etc.

[0077] Assuming N=5, the system will generate 5 different sets of solutions, each of which contains idea analysis and Python code. By adjusting the sampling parameters, it is possible to ensure that the generated solutions are diversified within a certain range, thereby improving the robustness of the system in dealing with complex problems. After generating N sets of solutions, the system will score each set of solutions. The Reward model is a scoring model built based on training data. It can score the matching degree of input and output, the accuracy and completeness of the solution, and give a score. After being scored by the Reward model, the system will select the solution with the highest score as the final output and return it to the user. This not only improves the accuracy of the answer, but also ensures that the user gets the highest quality answer.

[0078] The Reward model in this embodiment is a model for evaluating the quality of generative AI output. The output of the Reward model is usually a score that reflects the quality of the generated answer in multiple dimensions. The score is usually within a specific range, such as between 0 and 1 or between 0 and 100.

[0079] In generative AI models (such as GPT, LLaMA, Qwen, etc.), inference parameters are used to control the quality and diversity of generated answers. The main inference parameters include temperature, Top-p, and Top-k, which have an important impact on the output of the model.

[0080] Further, temperature: Temperature is a parameter that controls the randomness of the model when generating answers. Its value is usually between 0 and 1. A lower temperature value (such as 0.2) will make the model more inclined to choose the vocabulary with the highest probability, thereby generating more certain and predictable answers. This is usually suitable for scenarios that require strict accuracy (such as technical answers, programming). Higher temperature values ​​(such as 0.8 or 1.0) increase the randomness of the generation, giving the model more freedom to choose different vocabulary combinations. This is suitable for scenarios that require creativity or more open questions, such as writing and content creation.

[0081] Further, Top-p (Nucleus Sampling): Top-p (also known as nuclear sampling) controls the model to select only from words whose cumulative probability reaches a certain threshold (p value, such as 0.9), while ignoring words with lower probabilities. When Top-p = 0.9, the model will select a set of words with a total probability of 90% for sampling, which means that some low-probability words will be excluded, thereby reducing the randomness of the generated content. If Top-p is close to 1.0, the model will consider more possible words, and the generated content will be richer and more diverse. When used in conjunction with the temperature parameter, Top-p can better balance the diversity and accuracy of the content. When generating longer texts, Top-p sampling is usually more effective than pure temperature control in preventing the generation of irrelevant content.

[0082] Further, Top-k (Top-k Sampling): Top-k controls the model to select only from the top k words with the highest probability, while ignoring other words with lower probability. The value of k is usually between 10 and 50. When Top-k = 10, the model only samples from the 10 words with the highest probability, thereby reducing the randomness of the generation and making the answer more stable. When the k value is larger (such as 50 or higher), the model will have more choices, and the generated content will be more diverse and creative.

[0083] Step 4: Code security testing

[0084] The generated Python or SQL code needs to undergo a strict security review before actual execution to ensure that it does not pose a potential threat to system data, database performance, or user privacy. To this end, a special code security detection step is designed to automatically review the security of the code and identify and block operations that may cause security issues. This step mainly uses rule matching methods and a multi-layer detection mechanism based on security policies to perform a comprehensive security check on the generated code.

[0085] In some embodiments, illegal code injection is detected: illegal code injection refers to the attempt to perform unauthorized operations on the system by entering malicious code, such as accessing sensitive data, executing system commands, or manipulating databases. This module parses the generated code to detect whether it contains potential malicious code fragments, such as executing system commands (os.system, subprocess, etc.) or calling dangerous built-in functions (such as eval(), exec()). If suspicious code injection behavior is found, the system will immediately terminate the code execution and feedback to the user "illegal code injection detected, the operation has been blocked."

[0086] In some embodiments, detection of deletion operations: Data deletion operations (such as DROP TABLE, DELETE FROM, etc.) usually have irreversible effects on the database and may cause data loss. Therefore, this step will scan the generated SQL code to check whether it contains deletion operation instructions. If a deletion command is detected, the system will immediately interrupt the execution process and prompt the user: "A dangerous deletion operation has been detected and the operation has been blocked." This detection ensures that the system does not accidentally delete key data tables or records when processing user requests, thereby protecting data integrity.

[0087] In some embodiments, review of data insertion and modification operations: Insert and update operations may change existing data in the database, and there is a risk of overwriting or tampering with data. This step will identify all INSERT INTO and UPDATE statements and further analyze whether there is unauthorized data writing behavior. For example, if the generated code attempts to modify user information or financial data in the database without permission, the system will immediately issue an alarm and prevent execution.

[0088] In certain embodiments, control of large-scale data queries and calculations: In order to prevent the generated code from causing excessive consumption of database and server resources, this step will detect large-scale data queries and complex calculation operations that may cause system performance bottlenecks. For example, if a large number of data aggregation operations (such as GROUP BY, JOIN, etc.) or unoptimized nested queries are detected, the system will analyze the complexity and data volume of the query. If the estimated query time is too long or the data set is too large, the system will suggest that the user optimize the query or directly limit its execution to avoid system crashes due to excessive resource consumption.

[0089] In some embodiments, infinite loop detection: Infinite loop means that the program enters an infinite loop state during operation, resulting in infinite occupation of system resources. To avoid such problems in the generated Python code, this step will review all loop structures (such as while and for statements) and evaluate the rationality of the loop termination conditions. For example, if a while True statement is detected and there is no clear termination condition, the system will immediately block execution.

[0090] In some embodiments, monitoring of external data requests: To prevent unauthorized data leakage and privacy risks, this step will review whether the generated code contains external data request operations, such as accessing external APIs or downloading files through the requests module. Especially when processing sensitive data, unauthorized external data requests may lead to data leakage. If such operations are detected, the system will immediately interrupt execution and alert the user.

[0091] Furthermore, the code security detection in this step not only relies on predefined rules, but also supports a context-based dynamic detection mechanism. By comprehensively analyzing the contextual logic of the code, the system can identify potential risk behaviors, even if these behaviors do not appear to directly violate security rules. For example, the system can detect that even if it is a legitimate SQL query, an alarm will be triggered if its execution logic causes unexpected data modification or resource occupation. Only after all tests pass will the generated code be submitted to the code executor module for actual operation. If any test item fails, the system will stop the operation immediately and feedback the test results to the user so that the user can adjust the input questions or regenerate the code. Through this multi-layer protection mechanism, the system ensures data security and system stability while improving the efficiency of automated form question answering.

[0092] Step 5: Code execution and result generation

[0093] After the security check in step 4 to ensure that the generated code does not contain any potential risks (such as illegal data operations, large-scale computing requests, or external data leakage), the system will enter the code execution phase. In this phase, this step introduces a dedicated code executor module to ensure the efficiency, stability, and scalability of code execution.

[0094] In some embodiments, the code executor module is intended to provide a safe and efficient execution environment for the generated Python code. In order to reduce the user's operating burden and ensure the smooth execution of the code, the system will automatically configure the required execution environment for the user. First, the code executor will pre-load the necessary libraries related to data analysis (such as Pandas, Numpy, Matplotlib, etc.) to ensure that the code can successfully call these tools for data processing and visualization operations. This automated preprocessing step can greatly improve the execution efficiency of the code while avoiding execution errors caused by improper environment configuration.

[0095] The code executor captures all output during execution (including print information, data analysis results, and visualization charts). During execution, if the system detects a code execution error (such as data format mismatch, missing key libraries, etc.), the code executor will automatically diagnose and try to fix it. For example, if a specific Python library is missing, the system will prompt the user to install the corresponding dependencies. If the problem is beyond the scope of automatic repair, the system will provide a detailed error log to help the user understand the cause of the problem and guide them to make further adjustments.

[0096] Step 6: Generate summary answers based on execution results

[0097] After the code executor module successfully completes the data processing and analysis tasks, the system needs to further generate the final natural language answer so that the user can clearly understand the query results and analysis process. To this end, the system combines the output results of the code execution, the user's initial question, and the previously generated solution ideas to generate the final summary answer through the large language model (LLM). This step is designed to ensure that the answer is not only accurate and easy to understand, but also highly explanatory and complete.

[0098] In the previous stage, the system obtained the final analysis results (such as tabular data, statistical calculation results, or visual charts) through the code executor module. At the same time, in the early stage of problem solving, the large language model has generated detailed solution ideas and execution steps. To generate the final summary answer, this step integrates this information into a complete set of inputs, including the user's original question, the generated code, the execution results, and the logical steps of the solution. This input combination ensures that the large language model can fully understand the background and process of the entire problem solving when generating the summary.

[0099] The system uses carefully designed prompts to guide the large language model to analyze the integrated data and output a summary answer. The large language model will generate multiple possible summary answers based on the prompts. Since the output of the generative model may be affected by random factors, the system will sample multiple times (using different inference parameters such as temperature, topK, topP, etc.) to obtain multiple versions of answers.

[0100] For example, the system might generate the following versions of the answer:

[0101] Version 1: "According to the analysis, sales in the third quarter of 2023 were the highest, reaching 120,000 yuan, mainly driven by the launch of new products and marketing activities."

[0102] Version 2: “The data shows that the third quarter had the best sales performance, with total sales of 120,000 yuan, a year-on-year increase of 15%. This is because an effective promotion strategy was implemented in that quarter.”

[0103] Version 3: “Analysis shows that sales will be highest in the third quarter of 2023, accounting for 35% of total annual sales, thanks to increased seasonal demand.”

[0104] To ensure the quality of the output answers, the system will use the Reward model to score the multiple answer versions generated. This includes whether the answer accurately solves the user's question, whether it explains the logical process of data analysis, and whether it is clear and easy to understand. After the Reward model scores, the system selects the answer with the highest score as the final output and presents it to the user.

[0105] Based on the same technical concept, the present application also provides a table question answering system based on a large language model, including:

[0106] Data extraction module: used to receive and parse table data and vectorize the column names and content values;

[0107] Question adjustment module: used to rewrite the questions input by users in detail;

[0108] Code generation module: used to generate problem-solving ideas and corresponding Python or SQL codes;

[0109] Code security detection module: used to perform security review on the generated code;

[0110] Code execution module: used to execute code that passes security checks and capture output results;

[0111] Answer generation module: used to generate the final summary answer based on the code execution results.

[0112] Based on the same technical concept, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, a table question answering method based on a large language model is implemented.

[0113] In summary, this application proposes a table data question-answering method and system based on a large language model (LLM), which effectively solves many challenges of current table question-answering systems in processing complex table data through innovative architecture design and modular processing flow. It includes the following aspects:

[0114] 1. Efficient data extraction and screening;

[0115] 2. Task decomposition and process optimization;

[0116] 3. Intelligent solutions based on code execution results;

[0117] 4. Innovative multiple sampling and reward model scoring mechanism;

[0118] 5. Comprehensive code security testing to ensure system stability;

[0119] 6. The output plan is clear and the results can be explained and traced;

[0120] 7. Efficient and stable system architecture.

[0121] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0122] like Figure 1 As shown, the implementation process of a tabular data question answering system based on a large language model proposed in this embodiment includes the following steps:

[0123] The embodiment process includes the following steps, which belong to the field of big data analysis and artificial intelligence. The embodiment takes a fund intelligent query project of a securities company as an example. The system construction process for data query analysis based on natural language is based on the following parts of work:

[0124] 1) Data input: Provide the data source of the table data type required for the table data problem. It can be structured table data such as csv, excel, json uploaded by the user, or table data in the form of databases such as sqlite and mysql. Clearly specify the table name and column name information of the input data.

[0125] 2) Accept questions from users and rewrite them in detail. If the user's question is unclear or vague, the user will be prompted to clarify or redefine the question by asking follow-up questions to ensure that the question is clearly stated.

[0126] 3) Combine the user's question and the provided table data, and use embedding models such as BGE-M3, E5embedding and other models to extract the table column data information most relevant to the question. In the implementation process, all table data sources need to be vectorized using the embedding model and stored using redis. When the user enters a question, the embedding model is used to vectorize the question, and the cosine similarity is used to calculate the similarity between the input question vector and each column name and column content value. The k column names and column values ​​with the highest similarity are sorted to form the final table data information.

[0127] 4) Send the table data and the user's detailed rewritten questions into the large language model, and use the Prompt prompt word to output the problem-solving ideas and the corresponding python or sql code. The Prompt prompt word needs to be debugged and modified in conjunction with the large language model used. The output code can be Python or SQL, which depends on the user's business needs. If there is a data visualization requirement, Python code is generally used. If the data source provided is an SQL database, it will be more convenient to generate SQL code. The output problem-solving ideas refer to the step-by-step thinking process for the problem-solving ideas, which correspond to the code. In this process, the Rewad scoring model is used to score the output answers and select the answers with the highest output quality.

[0128] The process of a large language model outputting output based on input is affected by various output configuration parameters. Commonly used inference configuration parameters include temperature, topK, and topP parameters. Different configuration parameters affect different output contents. By modifying the configuration parameters, you can control the large language model to output different answers under the same input. The Reward model is a scoring model. It inputs input and output sample pairs, and outputs a score for the quality of the output's answer to the input in the sample. The more accurate and complete the output answer is, the higher the score. Currently, commonly used Reward models include Skywork-Reward-Model, Nemotron-Reward-Model, and llama-Reward-Model, and the open source market is constantly updating. By configuring sampling with the Reward model and inference parameters, you can get a relatively good answer under an output.

[0129] 5) The code output by the large language model also needs to go through a code security check process, which uses various rule detection methods to detect whether there are illegal code injections, deletion operations, data insertion or modification operations, large-scale data query calculations, dead loop operations, external data requests, etc. in the generated code. The above are just a series of general filtering rules. Users can also add business-related data query filtering rules according to actual business needs. The rule base also needs to be continuously updated according to usage. When non-standard or unsafe code is detected, the process is directly interrupted and feedback is given to the user in the form of a message.

[0130] 6) After the code is tested and found to be problem-free, it is sent to the code executor for execution. The code executor is connected to the data source, and some necessary pre-package imports, data loading and other pre-processing are loaded in advance, and the code can be executed to output the execution results.

[0131] 7) After receiving the code execution results and the above-mentioned problem-solving process content, they are sent to the summary answer module to generate the final answer and present it to the user. In this project, the idea of ​​generating multiple answers and using the reward model to score and select the answer with the best quality is also used to improve the output quality.

[0132] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A table question answering method based on a large language model, characterized in that: The following steps are involved: Extraction of table data information: Receive the table data uploaded by the user, parse the table data, extract the table name, column name and data type, and vectorize the column name and content value; By calculating the similarity between the user question vector and the table data vector, the table columns and content values ​​most relevant to the user question are screened out; User question adjustment: Parse the questions input by users, extract keywords through natural language processing technology, and use the preset prompt to guide the large language model to rewrite the questions in detail to ensure that the questions are clear and have sufficient context information; Solution and code generation: Input the extracted table data and adjusted questions into the large language model to generate problem-solving ideas and corresponding Python or SQL codes; Generate multiple candidate solutions through multiple samplings, score the candidate solutions using the Reward model, and select the solution with the highest score as the output; Code execution and result generation: Send the code that passes the security check to the code executor module, execute the code and capture the output results to provide data support for the generation of the final answer; Combine the execution results to generate a summary answer: The code execution results, problem-solving ideas, and the user's initial question are input into the large language model to generate the final summary answer and present it to the user, ensuring that the user obtains accurate and easy-to-understand query results.

2. A table question answering method based on a large language model according to claim 1, characterized in that: In the step of extracting table data information: Use a pre-trained embedding model to vectorize the table column names and content values; Dynamically adjust the extraction strategy of relevant columns and content values ​​according to the scale of table data to adapt to data sets of different sizes.

3. A table question answering method based on a large language model according to claim 2, characterized in that: In the user question adjustment step: If it is detected that the user's question is ambiguous or unclear, the follow-up mechanism is automatically triggered to clarify the question through further interaction with the user; The rewritten question is presented to the user for confirmation to ensure that the rewritten version accurately reflects the user's intent.

4. The table question answering method based on a large language model according to claim 1, characterized in that: The solution and code generation steps are: The large language model outputs step-by-step solutions and corresponding Python or SQL codes based on the input table data and adjusted questions; By adjusting the sampling parameters, candidate solutions are generated multiple times, and the reward model is used to score the candidate solutions, and the solution with the highest score is selected as the final output.

5. The table question answering method based on a large language model according to claim 1, characterized in that: The code security check is as follows: Conduct security review on the generated code to detect whether there is illegal code injection, deletion operation, data insertion or modification operation, large-scale data query calculation, dead loop operation and external data request; If the test passes, proceed to the next step to ensure that the executed code will not cause damage to the system or data.

6. A table question answering method based on a large language model according to claim 5, characterized in that: A rule-matching and security policy-based multi-layer detection mechanism is used to perform comprehensive security testing on the generated code.

7. The table question answering method based on a large language model according to claim 1, characterized in that: In the code execution and result generation steps: The code executor module automatically configures the required execution environment for the user and pre-loads the necessary libraries related to data analysis; All output is captured during execution, including print information, data analysis results, and visualization charts, and execution errors are automatically diagnosed and repaired when they are detected.

8. A table question answering method based on a large language model according to claim 1 or 7, characterized in that: In the step of generating a summary answer by combining the execution results: Integrate the code execution results, problem-solving ideas, and the user's initial question into a complete set of inputs to guide the large language model to generate the final summary answer; Use the Reward model to score the multiple answer versions generated and select the answer with the highest score as the final output.

9. A table question answering system based on a large language model, characterized in that: include: Data extraction module: used to receive and parse table data and vectorize the column names and content values; Question adjustment module: used to rewrite the questions input by users in detail; Code generation module: used to generate problem-solving ideas and corresponding Python or SQL codes; Code execution module: used to execute code that passes security checks and capture output results; Answer generation module: used to generate the final summary answer based on the code execution results.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the table question answering method based on a large language model as described in any one of claims 1 to 8 is implemented.

Images