Method for training and evaluating large model in field of table data analysis

Abstract

The invention discloses a method for training and evaluating a large model in the field of table data analysis. Screening and scoring the obtained table and text data; compiling a plurality of sets of different prompt words, and making'question 'data and'question-answer' data pairs under a table data scene as basic training samples; performing data enhancement on the generated'question 'and'question-answer' sample pairs; collecting and screening common text questions and answers, logic mathematics and code generation task data, selecting a proper data ratio, and performing all-parameter instruction alignment training on the base model; collecting and making an evaluation data set, compiling an evaluation script, and running to obtain a corresponding evaluation index; and iteratively optimizing the training data set according to the evaluation index and the feedback of the evaluation set until the model reaches an expected index. According to the invention, the large model in the field of table data analysis obtained by training not only has strong table question and answer ability, but also has strong general problem solving ability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for training and evaluating a large model in the field of table data analysis. Background Art

[0002] Two-dimensional structured table data is the most common form of data in daily production and life, and there is a general demand for query, calculation, visualization, and predictive analysis of table data. With the development of typical auto-encoding generative large model technologies such as GPT / LLAMA, research on using this technology to solve the task of automated analysis of two-dimensional table data has gradually deepened.

[0003] TableLlama (2023) uses several common table task data such as column type labeling, relationship extraction, entity, row clustering, cell question and answer, special-shaped table question and answer, table fact verification, etc., and conducts supervised instruction fine-tuning training (hereinafter referred to as SFT) on the basis of Llama2-7B. On the corresponding task test data set, it obtains evaluation indicators that exceed GPT4; TableLLM (2024) uses Llama2-7B and Llama2-13B as basic models, studies the reasoning generation ability of large language models (hereinafter referred to as LLM) on general two-dimensional tables in actual office scenarios, and open-sources the model's benchmark evaluation script code; TableBench (2024) deeply studies the application of tabular data in industrial scenarios, and designs and proposes an evaluation benchmark for table question and answer (TableQA) including four major categories and a total of 18 fields. At the same time, based on self-made training data, it trains and open-sources large models related to the table field on 6 models such as DeepSeekCoder-7B and Qwen2-7B.

[0004] In addition to the above-mentioned large-scale model research in the table domain based on the supervised fine-tuning school, there are also agent-workflows (Agent-WorkFlow) built based on common Coder expert models such as DeepSeekCoder, Qwen2.5-Coder, Yi-Coder, etc., which enhance the ability of LLM to analyze table data in actual scenarios through prompt engineering (Prompt-Engineering) and good workflow design. Mainstream BI manufacturers such as NetEase Cloud Computing ChatBI use the above technical solutions.

[0005] However, unlike general sequence-to-sequence generation tasks, the input of data analysis in tabular scenarios is generally composed of two elements: tables and human queries. LLM needs to have a strong understanding of both tables and human queries, as well as strong scenario code generation capabilities (such as Python or SQL). Currently, both the above technical solutions based on tabular data SFT and intelligent agent workflows have problems such as poor cross-scenario and domain generalization and difficulty in migration due to excessive fine-tuning or workflow solidification, which makes it difficult to meet the needs of more universal tabular data analysis. According to the conclusions of LIMA (Zhou, 2023), in order to enhance the robustness and generalization of the model after supervised fine-tuning and reduce the cost and difficulty of cross-scenario migration of the model, a set of process methods is needed to enhance the diversity and quality of the training corpus at the data level, and to select appropriate supervised training fine-tuning strategies and parameters to solve the bottlenecks in the application of large models in the current tabular field. Summary of the invention

[0006] The purpose of the present invention is to design a process method for training and evaluating large models in the field of tabular data analysis in view of the above-mentioned shortcomings of the current large model application in the field of tabular data analysis, covering a complete set of technical processes and methods from data acquisition, data screening, data generation, data enhancement, model training and model evaluation. Through reasonable data screening and matching, high-quality data generation control, various data enhancement methods, reasonable fine-tuning algorithms, parameter combinations and comprehensive model evaluation settings, a large model in the field of two-dimensional tabular data analysis with strong robustness and good generalization is finally obtained.

[0007] In a first aspect, the present invention provides a method for training and evaluating a large model in the field of tabular data analysis, the method comprising the following steps:

[0008] Step 1: Data Collection

[0009] The collected data include text question answering, mathematical logic reasoning, code generation, and table question answering.

[0010] Step 2: Basic table screening

[0011] Classify and filter basic tables based on fixed table processing and filtering rules;

[0012] Select a set of tables suitable for making table question sets and table question-answer pairs from the collected basic tables;

[0013] Step 3: Create tabular question sets and question-answer pairs

[0014] Combined with the basic tables, use role-playing, industry vocabulary guidance, and task differentiation instructions to input the strong generative model to generate a set of table questions;

[0015] Use the tabular question set and the corresponding table, combined with multiple sets of prompt words, to generate single-round and multi-round tabular question-answering data, i.e., QA pairs;

[0016] Step 4: Data Enhancement

[0017] Use rule matching and model generation to enhance the existing data to expand the amount and diversity of data.

[0018] Step 5: Data screening

[0019] Eliminate the following three samples:

[0020] a. The output content cannot fully respond to the input content;

[0021] b. The output code contains obvious errors in fields or values, which makes the code unexecutable;

[0022] c. The output code is executable, but has obvious logic and step defects;

[0023] Step 6: Model training and fine-tuning

[0024] The first part uses general data and mixes some table QA pair data, and adopts pure supervised fine-tuning loss function to train the model;

[0025] The second part uses pure table QA sample data and ORPO fine-tuning algorithm to strengthen the model's table QA question-answering ability while also enabling it to refuse to answer ambiguous questions.

[0026] Step 7: Model Evaluation

[0027] Model evaluation includes two parts: form-related task evaluation and general ability evaluation;

[0028] After each training session, run the evaluation script on the evaluation set corresponding to the task to obtain the corresponding evaluation indicators, and compare them with the evaluation indicators of the baseline model;

[0029] Step 8: Repeat steps 1 to 7 until all form-related tasks have achieved the expected evaluation objectives.

[0030] In a second aspect, the present invention provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0031] The memory stores computer-executable instructions;

[0032] The processor executes the computer-executable instructions stored in the memory to implement the above method.

[0033] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the above method when executed by a processor.

[0034] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which implements the above method when executed by a processor.

[0035] The beneficial effects of the present invention are:

[0036] The present invention covers the entire process of data collection, data screening, data enhancement, model fine-tuning training, and model evaluation. By reasonably allocating general text question and answer, data logical reasoning, general code generation, and table-related task training data, the trained large model in the field of table data analysis has not only stronger table data analysis capabilities, but also strong general task capabilities.

[0037] For tasks such as data analysis in the tabular field, which require strong parsing capabilities for both human language and data tables, this effectively increases the generalization of the model across different industry fields and can effectively reduce the alignment training cost for adapting the model to downstream application tasks.

[0038] Different from the training process of large models in most fields, the present invention uses a multi-stage fine-tuning technology in the model fine-tuning process, that is, the alignment fine-tuning process is divided into two stages. The first stage uses a supervised fine-tuning algorithm (SFT cross entropy loss function), and mixes general data and tabular analysis data during the fine-tuning process. The main purpose is to enable the model to maintain its general capabilities while obtaining the ability to perform tabular analysis tasks; the second stage uses the ORPO algorithm, and uses tabular analysis sample pairs (good and bad answer sample pairs) data during the fine-tuning process. The purpose is to enhance the model's ability to perform tabular data analysis tasks, while enabling the model to obtain the ability to reject (or ask questions) about ambiguous tabular questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0040] Figure 1 The present invention provides a process for building a data analysis query framework using natural language. DETAILED DESCRIPTION

[0041] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] The present application embodiment discloses a method for training and evaluating a large model in the field of tabular data analysis, the method comprising the following steps:

[0043] Step 1: Data collection.

[0044] Table analysis LLM requires strong general text comprehension, table serialization information comprehension, mathematical logic, and analytical code generation. Based on this, the collected data should include general text question and answer, mathematical logic reasoning, code generation, table question and answer, etc.

[0045] In addition, because the tabular question and answer data are relatively scarce and too homogeneous, a large number of basic tables must be collected separately, and then question and answer data are produced based on the tables;

[0046] In this embodiment, the table serialization information refers to converting a structured two-dimensional data table into text information that is convenient for input into LLM. Common serialization methods include markdown format, string format, json format, SQLSchema and some custom mixed formats.

[0047] Step 2: Basic table screening.

[0048] Based on a set of fixed table processing and screening rules, the basic tables are classified and screened, and table sets suitable for making table question sets and table question and answer pairs are selected from the collected basic tables.

[0049] The table processing and screening rules include the following:

[0050] 1. The number of rows in the table is greater than 5, and the number of columns is greater than 3;

[0051] 2. The number of null values ​​in any column does not exceed 30%;

[0052] 3. Delete the horizontal table;

[0053] 4. Delete the table where the column name and the first value in the column can be directly matched;

[0054] 5. Delete the Unnamed columns in the table, the columns whose names are single numbers or letters, and the columns whose cells store more than 100 bits of content.

[0055] Step 3: Preparation of tabular question sets and question-answer pair data.

[0056] Combined with the basic table, multiple methods such as role-playing, industry vocabulary guidance, and task differentiation instructions are used to input into strong generative models such as GPT-4 to generate a table question set. Using the question set and the corresponding table, combined with multiple sets of prompt words, single-round and multi-round table question and answer data (QA pairs) are generated.

[0057] Furthermore, in order to cooperate with the use of subsequent training and fine-tuning algorithms, each time a QA sample is paired, GPT-4 is used to generate [two sets of high-quality and low-quality answers] for the same set of questions. All table QA pair training samples are made into the following form:

[0058] {

[0059] “query”:<content of human query> ,

[0060] “table”:<content of table information> ,

[0061] “chat_history”:<content of chat history>

[0062] “good_response”:<content of good response> ,

[0063] “bad_response”:<content of bad response>

[0064] }

[0065] For all common text data, they are made into the following form:

[0066] {

[0067] “instruction”:<content of input> ,

[0068] “response”:<content of llm output>

[0069] }.

[0070] In this embodiment, the role-playing refers to injecting industry role information into the prompt words for the generation model, such as financial data analyst, data operation and maintenance manager, manufacturing supply chain expert, technical engineer, etc. The use of role-playing can effectively enhance the depth and diversity of problems when LLMs such as GPT-4 generate problem sets.

[0071] In this embodiment, the two groups of answers, high-quality and low-quality, are as follows: the high-quality answers can fully respond to user questions, and the data, fields, and code logic are complete and error-free; the low-quality answers have at least one error in the above points.

[0072] Step 4: Data enhancement.

[0073] Use rule matching and model generation to enhance the existing data to expand the amount and diversity of data.

[0074] Furthermore, the rule matching refers to the use of regular expression matching to synchronously modify the context (input) input into the model and the output content (output) in the sample, for example: modifying the table fields and cell contents in the input table serialization information, and then synchronously modifying the corresponding extraction fields and single-value content in the output code.

[0075] Furthermore, the model generation enhancement refers to the use of a strong generation model such as GPT-4, combined with a set of text enhancement generation prompt words, to synchronously modify and re-output the input and output parts of the sample.

[0076] Step 5: Data screening.

[0077] The following rules and models are used to filter out the following three types of samples:

[0078] 1. The output content cannot fully respond to the input content (problem);

[0079] 2. The output code contains obvious errors in fields and values, which makes the code unable to execute;

[0080] 3. The output code is executable, but has obvious logic and step defects.

[0081] At the same time, you can also refine the scoring rules for evaluation samples in the model screening prompts, make specific requirements on the output form and output complexity of specific types of samples (such as visualization, modeling prediction), and then select samples with scores exceeding a certain higher threshold.

[0082] The model screening prompt words need to clarify the scoring criteria and output format, and an example for different sample types is as follows:

[0083] Evaluate the quality of the AI ​​assistant's response to the user's question based on the following criteria, score it between 1 and 10, and provide a brief explanation of the reason for the score.

[0084] Scoring criteria:

[0085] 1-2 points: Very inadequate answer

[0086] 1 point: The response has little relevance or information to the user's inquiry and may be completely misleading or misleading.

[0087] 2 points: The response contains some words or information that is relevant to the user's question, but is still mostly irrelevant or useless.

[0088] 3-4 points: Partially relevant but insufficient information

[0089] 3 points: The response provides some information relevant to the user's inquiry, but the information is very limited or not specific enough and may contain a lot of irrelevant content.

[0090] 4 points: The reply contains more information related to the user's question, but still fails to address the main part of the question and may have some misunderstanding or be off-topic.

[0091] 5-6 points: basically relevant answer

[0092] 5 points: The response directly answers the user's question and, although the content may be incomplete or vague, provides useful basic information.

[0093] 6 points: The response addresses the main part of the question, but may miss some key points or contain some unnecessary content.

[0094] 7-8 points: effective answer

[0095] 7 points: The response fully answers the user's question, the information is useful and basically complete, and the structure is reasonable, although there is still room for improvement.

[0096] 8 points: The response is very clear and directly addresses the user's issue, with little superfluous information, and shows good organization and relevance.

[0097] 9-10 points: Excellent answer

[0098] 9 points: The response not only fully solves the user's problem, but also demonstrates high professional knowledge, is concise and contains almost no superfluous information, and is overall very useful.

[0099] 10 points: The response perfectly matches the user's needs, demonstrates a high level of professionalism and insight, does not contain any superfluous or irrelevant information, is extremely clearly organized, and is a highly customized and precise answer.

[0100] ## Provided command data

[0101] [Chat History]

[0102] {history}

[0103] [Question]

[0104] {input}

[0105] [The Start of Assistant's Answer]

[0106] {output}

[0107] [The End of Assistant's Answer]

[0108] ## Output:

[0109] Strictly output in the following json format

[0110] ```json

[0111] {{

[0112] "score": "evaluation score of type int",

[0113] "explanation": "A brief explanation of the reason for the rating"

[0114] }}

[0115] ```]

[0116] Step 6: Model training and fine-tuning.

[0117] Considering the complexity of two-dimensional table tasks, the input includes both tables and human query questions. Among these two parts of input, there may be two problems: table ambiguity (the meaning of fields and cell values ​​is unclear) and human question ambiguity. For such tasks with potential input and output mismatches, the model itself should have a strong ability to reject (or ask back) answers.

[0118] At the same time, considering the need to cover the overall task alignment, the traditional training method is to combine supervised finetuning (SFT) with RLHF (Reinforcement learning with Human Feedback) algorithms such as PPO (Proximal Policy Optimization) or DPO (Direct Preference Optimization) for multi-stage fine-tuning training.

[0119] This embodiment uses the ORPO (Odds Ratio Preference Optimization) fine-tuning algorithm, which can merge the two-stage training process of SFT and RLHF, and at the same time achieve the effect of omitting the loading of the preference model and saving video memory. The entire training and fine-tuning process is divided into two parts. The first part uses general data and mixed with some table QA pair data (the label part uses good response), and uses a pure supervised fine-tuning loss function to train the model. The purpose is to make the model have strong general capabilities; the second part uses pure table QA sample pair data and uses the ORPO fine-tuning algorithm to strengthen the model's table QA question-answering ability while also enabling it to refuse (reverse) answer ambiguous questions.

[0120] The ORPO algorithm is a fine-tuning technique that simplifies the process of adjusting LLM to a specific task. It solves the limitations of the traditional two-stage method. The core idea is to add a penalty term for poor-quality answers on the basis of the SFT loss function to meet the needs of preference alignment training.

[0121] The objective function of the ORPO algorithm is as follows:

[0122]

[0123] in, Indicates that the input is , high-quality output is , the inferior output is expectations of the time. represents the supervised fine-tuning loss (cross entropy loss function), as follows:

[0124]

[0125] In the above formula, m represents the number of tokens of generated content, |V| represents the vocabulary set, Is the actual encoding of each token position in the generated content And its corresponding model output log probability The product of .

[0126] represents the odds penalty loss term, and its specific formula is as follows:

[0127]

[0128] The function definition is as follows:

[0129]

[0130] in It represents the input of x, token y k The corresponding probability.

[0131] at last, represents the odds penalty loss function The corresponding weight is a hyperparameter.

[0132] Step 7: Model evaluation.

[0133] Model evaluation includes two parts: table-related task evaluation and general capability evaluation. After each training, the evaluation script is run on the evaluation set corresponding to the task to obtain the corresponding evaluation index, which is then compared with the evaluation index of the baseline model.

[0134] It should be made clear that the trained model should have a significant improvement on the benchmark of table data analysis related tasks without reducing its general capabilities. The error cases should be carefully checked. For batch errors such as table field extraction, erroneous data generation, and single-line code errors in multi-condition extraction, the generated sample data should be checked back to ensure that no similar errors occur in the sample data, and then data supplementation should be performed for specific errors.

[0135] Furthermore, the table-related evaluation tasks include verification of a total of 17 task benchmarks in six categories, including table understanding, question answering, fact verification, table to text generation, natural language to SQL, and real table verification (RealTabBench).

[0136] Furthermore, the general task evaluation mainly includes two types of tasks, namely general semantic understanding and code generation, and a total of four task benchmark verification evaluations, specifically the four benchmark test sets of MMLU, CMMLU, HumanEval and MBPP.

[0137] Step 8: Repeat steps 1 to 7 until all form-related tasks have achieved the expected evaluation objectives.

[0138] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0139] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0140] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0141] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0142] Example:

[0143] like Figure 1 As shown, this embodiment takes the training and evaluation process of a large model for analyzing a banking business data table as an example. The entire training and evaluation process is based on the following parts:

[0144] 1) Look for high-quality general text generation, code generation, mathematical logic question-answering data, and basic table data from official websites such as huggingface, github, government, and research institutions.

[0145] In the implementation example, it should be noted that if the model needs to connect to specific industry scenarios such as banking and financial credit business and specific business table data, it is also necessary to collect and build an industry knowledge base, build text question and answer data based on the knowledge base, and build table question and answer data based on the business table data.

[0146] 2) Use rule matching to filter out low-quality tables that do not meet the requirements. To ensure the diversity of tables in the final table question and answer data, try to keep as many different basic tables as possible while the quality allows, and use model classification or keyword matching to classify the tables by industry (finance, manufacturing, education, etc.).

[0147] In this embodiment, except for the tables related to the banking and financial industry, which maintain the highest proportion, the proportion of tables in other industries should be kept average as much as possible to maintain the diversity of table field information input into the model during training.

[0148] 3) Based on the selected tables, use role-playing, task description, industry-specific term embedding and other methods to generate table question sets. In order to maintain data diversity, the number of questions generated by a single table should not exceed 5.

[0149] 4) Generate answers to tabular data analysis questions by combining several different sets of code generation prompt word templates, several different sets of table information serialization methods, and input table questions.

[0150] It should be noted that for tabular question-answering data, in this embodiment, two different LLMs, strong and weak, can be used to generate high-quality and low-quality answers for the same question. In this embodiment, the strong generative model can use GPT-4-turbo, and the weak generative model can use Llama3-72B-instruct.

[0151] The code generates prompt word templates, including Plan-and-Solve or React templates according to the type of generated samples (multi-round or single-round), and the React template is taken as an example. Its typical input form is as follows: [ You have access to several pandas dataframes. Given a user questionabout the dataframes, write the Python code to answer it. ## Use the following format: Question: The user question you need to answer. Thought: Think about what you should do based on the providedDataFrames, the question, etc. Python code: Generate python code, wrap it up with ```python ... ```.If no code is required, print the string "no code needed". Observation: the result of the execution of your Python code. No need to generate results provided by the Python execution tool. ... (this Thought / Python code / Observation can be repeated N times) Final answer: Conclude with a clear and concise answer based on the observation and the initial question. ]

[0153] In the REACT template, there is an observation part (Observation), which is the output of the code executor. LLM will automatically make a summary statement (Final Answer) based on the results of code execution, input questions and table information. Since the code generated by the React template always has multiple interactions between LLM and external input each time, it is considered that the React template generates multiple rounds of data. These multiple rounds of data can significantly improve the multi-round question-answering ability and long-context information extraction ability of the trained LLM.

[0154] 5) Use rule matching and model generation to enhance tabular question-answering (QA) pairs and general text generation data.

[0155] In this embodiment, in addition to enhancing the code details of the table data question and answer, the answer part generated by the LLM can also be refined and polished in detail in combination with the banking and financial industry knowledge base to enhance the professionalism of the final trained LLM in answering industry questions.

[0156] 6) Use rule filtering and model scoring to filter general data and tabular question and answer data.

[0157] In this embodiment, when using the model scoring to filter and select samples, it is necessary to match the scoring prompt words for the general data and the table question and answer data respectively.

[0158] The general data scoring prompts can refer to the part described in the invention content. The scoring of table question and answer data needs to be classified according to the task of table question and answer data, such as numerical query, numerical calculation, statistical modeling, visual analysis, etc., and then be refined to give more comprehensive and specific requirements from various aspects such as the integrity, logic, complexity, and robustness of the generated code. The scoring threshold is uniformly set to 9 points (the score is 1-10 points), and finally the samples with scores exceeding the threshold are screened.

[0159] 7) Use the SFT+ORPO two-stage training method to align and fine-tune the model.

[0160] In this embodiment (corresponding to the final weight TableGPT2-7B), an 8-card H800 server is used, the learning rate of the first stage is 1e-5, the number of single-card batches is 3, the number of cumulative gradient steps is 32, and 1.5 epochs are trained using general data plus table QA data; in the second stage, other parameters are kept unchanged, the learning rate is adjusted to 1e-6, and pure table QA data is used for another epoch.

[0161] 8) The model effect is evaluated using a series of public test datasets collected as shown in Tables 1 and 2, as well as a test set separately separated from the generated tabular QA dataset.

[0162] Table 1 shows the values ​​of the evaluation indicators of the final training model (TableGPT2-7B and TableGPT2-72B) of the embodiment of the present invention on the table-related tasks and the comparison with other large competitive models such as Deepseekcoder, TableLLM, Qwen2.5-Coder, etc.; Table 2 shows the values ​​of the evaluation indicators of the final training model (TableGPT2-7B) of the embodiment of the present invention on the general task and the comparison with the large models of competitors such as Deepseekcoder, TableLLM, and Qwen2.5-Coder.

[0163] In this embodiment, the table task evaluation is first performed using a separately separated table QA test set, and compared with the effect of the original base model. Ensure that the execution pass rate (execution-pass-rate) of the selected training weight on the test set exceeds 0.9. Then run the test on each collected task data set, and compare the indicators with the training base model, competitor model, etc. According to the increase and decrease of the values ​​of each indicator and the summary and analysis of the wrong use cases, reasonably adjust the data ratio or add new training samples, and repeat steps 1) to 8) until all indicators reach the specified target value.

[0164] Table 1

[0165] Table comprehension

[0166] Questions answered

[0167] Fact Verification

[0168] Convert table to text

[0169] Natural Language to SQL

[0170] Real table task evaluation

[0171] Table 2

[0172] Coding ability

[0173] Languages

[0174] The results of table-related tasks (such as table comprehension, table question and answer, fact verification, etc.) and general ability assessment (such as coding ability and language comprehension ability) shown in Tables 1 and 2 show that the table domain large model (TableGPT2-7B) finally obtained by using a set of process methods including data screening and production, data enhancement, model training, model evaluation, etc. described in this embodiment has excellent performance in various table-related tasks; at the same time, it also maintains good indicator performance in general tasks, which is of great significance for reducing the cost of scenario domain migration.

[0175] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for training and evaluating a large model in the field of tabular data analysis, characterized in that The method comprises the following steps: Step 1: Data Collection The data collected include text question answering, mathematical logic reasoning, code generation, and table question answering; Step 2: Basic table screening Classify and filter basic tables based on fixed table processing and filtering rules; Select a set of tables suitable for making table question sets and table question-answer pairs from the collected basic tables; Step 3: Create tabular question sets and question-answer pairs Combined with the basic tables, use role-playing, industry vocabulary guidance, and task differentiation instructions to input the strong generative model to generate a set of table questions; Use the tabular question set and the corresponding table, combined with multiple sets of prompt words, to generate single-round and multi-round tabular question-answering data, i.e., QA pairs; Step 4: Data Enhancement Use rule matching and model generation to enhance the existing data to expand the amount and diversity of data; Step 5: Data screening Eliminate the following three samples: a. The output content cannot fully respond to the input content; b. The output code contains obvious errors in fields or values, which makes the code unexecutable; c. The output code is executable, but has obvious logic and step defects; Step 6: Model training and fine-tuning The first part uses general data and mixes some table QA pair data, and adopts pure supervised fine-tuning loss function to train the model; The second part uses pure table QA sample data and ORPO fine-tuning algorithm to strengthen the model's table QA question-answering ability while also enabling it to refuse to answer ambiguous questions. Step 7: Model Evaluation Model evaluation includes two parts: form-related task evaluation and general ability evaluation; After each training session, run the evaluation script on the evaluation set corresponding to the task to obtain the corresponding evaluation indicators, and compare them with the evaluation indicators of the baseline model; Step 8: Repeat steps 1 to 7 until all form-related tasks have achieved the expected evaluation objectives.

2. A method for training and evaluating a large model in the field of tabular data analysis according to claim 1, characterized in that: The table processing and screening rules described in step 2 include the following: A. The number of rows in the table is greater than 5, and the number of columns is greater than 3; B. The number of null values ​​in any column does not exceed 30%; C. Delete the horizontal table; D. Delete the table where the column name and the first value in the column can be directly matched; E. Delete the Unnamed columns in the table, the columns whose names are single numbers or letters, and the columns whose cells store more than 100 bits of content.

3. The method for training and evaluating a large model in the field of tabular data analysis according to claim 1, characterized in that: The role-playing described in step three refers to injecting industry role information into the generated model in the prompt words.

4. The method for training and evaluating a large model in the field of tabular data analysis according to claim 3, characterized in that: Each time a QA sample is paired, two sets of answers, high-quality and low-quality, are generated for the same set of questions.

5. The method for training and evaluating a large model in the field of tabular data analysis according to claim 1, characterized in that: The rule matching described in step 4 refers to the use of regular expression matching to simultaneously modify the context of the sample input into the model and the output content.

6. The method for training and evaluating a large model in the field of tabular data analysis according to claim 1, characterized in that: The table-related evaluation tasks described in step 7 include table understanding, question answering, fact verification, table text generation, natural language to SQL, and actual table verification.

7. The method for training and evaluating a large model in the field of tabular data analysis according to claim 1, characterized in that: The general task evaluation includes two types of tasks: semantic understanding and code generation.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 7 when being executed by a processor.

Images