A table data analysis large model training and application method based on agent interaction reinforcement learning

Abstract

The application discloses a table data analysis large model training and application method based on agent interaction reinforcement learning. After collecting multi-source data and generating reference reply text, multi-source word vector data is obtained by conversion; the multi-source word vector data is filtered through multiple strong models and multiple repeated inference processing; the extracted data is input into a cold start model with preset weight parameters for full parameter supervised fine-tuning training processing; the multi-source word vector data is input into the trained cold start model for training by using a reinforcement learning method; and the cold start model trained by using the reinforcement learning method is used in the reply processing of the text questions in the actual scene. The application improves the actual application effect of the large language model in the two-dimensional table data analysis field through reasonable multi-source data setting and proportioning, training process and algorithm design.

Description

Technical Field

[0001] This invention relates to the fields of big data and artificial intelligence, specifically to a method for training and applying a large-scale tabular data analysis model based on agent-interactive reinforcement learning. Background Technology

[0002] Tabular data (such as fact / dimension tables, financial statement tables, and industry statistical tables in enterprise data warehouses) dominates in business intelligence, risk control and compliance, and scientific research statistics. Traditional tabular analysis relies on a human-computer division of labor model that combines SQL / ETL with visualization tools. This model has a high barrier to entry and weak feedback mechanisms, making it difficult to meet the end-to-end experience needs of non-technical users for "natural language questioning → calculation → visualization / conclusion".

[0003] Early academic research focused on table understanding representation learning and weakly supervised table question answering. For example, TaBERT (2024) proposed joint pre-training of "text + table" to promote the alignment of columns / cells with natural language, targeting tasks such as NL2SQL / table QA. HiTab (2022) proposed a hierarchical table dataset and evaluation, emphasizing the challenges posed by hierarchical indexes and implicit summary relationships to numerical reasoning.

[0004] In recent years, Large Language Models (LLMs) have rapidly penetrated data analysis scenarios, leading to a trend of "using natural language to drive SQL / Pandas / visual pipelines." With the continuous advancement of LLM technology and its application in code generation and tabular data analysis, data analysis products and related model research based on LLM intelligent agent technology have also emerged. Google Gemini in BigQuery supports generating SQL / Python from natural language, interpreting SQL, and automatically generating questions and analysis suggestions in "DataCanvas / Insights," covering data exploration to engineering stages; Microsoft Copilot for Power BI supports natural language analysis of reports and models, generating DAX and summaries; addressing the issues of insufficient understanding capabilities of large models in specific tabular domains and poor cross-domain generalization ability, the Zhejiang University team's TableGPT2.0 (2024), trained based on Qwen2.5, has achieved good feedback in domain applications.

[0005] However, due to the scarcity of data in two-dimensional table data analysis scenarios, the special nature of the application environment, and the accuracy of calling tools such as code executors and file data reading, writing, and modification tools, most open-source large model or data analysis products based on closed-source large model APIs still cannot meet the end-to-end application requirements mentioned above. They have shortcomings such as insufficient code capabilities in table scenarios, poor ability to call proprietary toolsets, and insufficient semantic understanding of data analysis scenarios. Summary of the Invention

[0006] The purpose of this invention is to address the aforementioned shortcomings in the application of large-scale models in the field of tabular data analysis. This invention designs a method for training and applying large-scale tabular data analysis models based on agent-interactive reinforcement learning.

[0007] This invention covers the collection and matching of multi-source training data, the construction of environmental feedback systems, the establishment of external execution toolsets, the optimization of reinforcement learning training algorithms, the creation of model evaluation datasets, and the design of multi-round iterative process mechanisms.

[0008] To address the shortcomings of the prior art, the present invention provides the following technical solution:

[0009] Step 1: Data Collection

[0010] Collect multi-source data, including text question-and-answer data, mathematical logic reasoning data, code generation data, and table question-and-answer data. After generating a reference response text for each data item, use the corresponding model vocabulary dictionary to convert and process it into a set of word vector data that is subdivided into word vector tokens, thereby obtaining multi-source word vector data in the form of word vectors.

[0011] This invention trains a large-scale model for tabular data analysis by incorporating various multi-source data, thereby maintaining general-purpose capabilities while maintaining tabular processing capabilities.

[0012] Step Two:

[0013] The multi-source word vector data is filtered through repeated inference processing by multiple strong models;

[0014] Step 3: Supervised Fine-tuning Training

[0015] A certain proportion of data is extracted from the multi-source word vector data obtained in step two and input into the cold start model with preset weight parameters for fully supervised fine-tuning training to obtain the trained cold start model.

[0016] Step 4: Optimize the training algorithm (objective function)

[0017] The multi-source word vector data obtained in step two is input into the trained cold-start model and trained using reinforcement learning methods.

[0018] Step 5: Apply the cold-start model trained using reinforcement learning to answer text questions in real-world scenarios.

[0019] The text-based question-and-answer data includes text data based on descriptive questions, their constraints, and answers;

[0020] The mathematical logic reasoning data includes text data concerning mathematical operation problems, their constraints, and the operation results;

[0021] The code generation data includes text data of modeling code for specific application problems and their constraints;

[0022] The tabular question-and-answer data refers to text data of questions, their constraints, and answers based on known tabular data;

[0023] The table-based question-and-answer data is divided into self-built table-based QA datasets, SQL-QA datasets, and numerical analysis Table-QA datasets with standard answers.

[0024] The self-built table QA dataset specifically refers to Query-PythonCodeAnswer data pairs created based on a filtered two-dimensional table.

[0025] The SQL-QA dataset specifically refers to Query-SQLCodeAnswer data pairs created based on a filtered two-dimensional table.

[0026] The Table-QA dataset with standard answers specifically refers to open-source table (irregular table) datasets with accurate labels, such as Hitab-Benchmark.

[0027] Each piece of data in the text-based question-and-answer data, mathematical logic reasoning data, code generation data, and tabular question-and-answer data is used to generate a reference response text through a text generation reasoning model and added to that piece of data.

[0028] Step two specifically involves:

[0029] Multiple different evaluation and scoring models are used as strong models. Each data point of multi-source word vector data is input into different evaluation and scoring models to obtain its own data inference score. Then, a comprehensive evaluation is carried out based on the data inference score of each data point. Data points with all data inference scores not lower than the preset reliable answer score threshold are retained, while other data points are discarded.

[0030] The data answer verification refers to determining whether the generated data answer is the true solution to the given question, given a table file or database and a user's query.

[0031] Due to the large volume of data, multiple reasoning verifications using a voting method similar to that described above, along with sampling based on actual data samples, can effectively improve the accuracy of sample answers.

[0032] The cold start model specifically adopts the qwen3-8b-instruct model.

[0033] In step three, the supervised fine-tuning of all parameters involves sequentially inputting the reference response text and prompt words for each data point, predicting the word vector corresponding to the reference response text for the next data point, and then setting the following loss function with all parameters:

[0034] L SFT (θ)=-Σ T t=1 logp θ (y t |y <t ,x)

[0035] In the formula:

[0036] L SFT () represents the loss function for supervised fine-tuning training;

[0037] x represents the word vector corresponding to the prompt word, which is composed of the question and condition constraints in the data;

[0038] y t The word vector represents the reference reply text corresponding to the t-th data item, where t represents the data item number and T represents the total number of data items.

[0039] y <t This represents the word vectors corresponding to all reference response texts for data up to the t-th entry;

[0040] p θ This represents the probability corresponding to the output word vector;

[0041] θ represents the weight vector consisting of all the weights in the cold start model;

[0042] p θ (y t |y <t (x) represents the current parameters θ of the cold start model given the prompt word x and all currently generated word vectors y. <t In the case of the next word vector y t The predicted probability.

[0043] The reinforcement learning methods in step four include:

[0044] S1. Sampling Inference: Each data point in the input multi-source word vector data is sampled and grouped for multiple inference predictions. By setting the generative model temperature parameter, the multiple inference predictions are different. According to the different data types, the multiple different inference results corresponding to each data point are input into the corresponding scoring feedback acquisition method and the corresponding sampling evaluation parameters are obtained. The evaluation parameters are added to each data point.

[0045] S2. Input the multi-source word vector data with evaluation parameters obtained in step S1 into the cold start model after training in step three for reinforcement learning training.

[0046] S3. For the reinforcement learning training in step S2, specifically establish an objective function that can suppress entropy decay and reduce memory overhead, and train with the goal of maximizing the expected value of the objective function.

[0047] In step S1, the multi-source word vector data is divided into six types: general question and answer, mathematical reasoning, code generation, table question and answer solved by Python, table question and answer solved by SQL, and table question and answer with accurate answers. According to different types, after each inference prediction, the data is processed according to four different scoring feedback acquisition methods. The four different scoring feedback acquisition methods are LLM-EVAL, pre-guided evaluation method, post-run guided evaluation method, and post-run model evaluation method.

[0048] General question and answer refers to data that has no standard answer or whose results have multiple variable forms, making extraction difficult.

[0049] Mathematical reasoning refers to the data consisting of the answer generated by the model and the given standard answer;

[0050] Code generation refers to the executable code data generated by the model;

[0051] Table-based question-and-answer data that can only be solved using Python refers to table-based question-and-answer data that can only be solved using Python.

[0052] SQL-resolved tabular question-and-answer data refers to tabular question-and-answer data that can only be resolved using SQL.

[0053] A table with accurate answers refers to table question and answer data with accurate answers.

[0054] The LLM-Eval feedback refers to an evaluation method that uses generative large models to evaluate the answers or responses obtained during the sampling process.

[0055] The aforementioned pre-guided evaluation method refers to a method that uses regular expression matching combined with rule-based extraction techniques such as specific format parsing to compare the generated answer with the given standard answer.

[0056] The post-run guided evaluation method refers to the method of taking the executable code generated by the training model, feeding it into an executor (Python or SQL) to obtain the corresponding answer, and comparing the obtained execution result with the reference answer through rules to obtain the corresponding score feedback.

[0057] The post-run model evaluation method refers to the method of taking the executable code generated by the training model, feeding it into an executor (Python or SQL) to obtain the corresponding answer, and then feeding the execution result and the reference answer into a large model for evaluation to obtain the corresponding score feedback.

[0058] In step S3, the objective function is specifically set using the following formula for reinforcement learning training:

[0059] L GRPO++ (θ)=max E[(1 / G)×Σ G i=0 min((s i (θ)A i , clip(s i (θ),1-ε low ,1+ε high A i )]

[0060] +C H E k [H(π θ )]-ηE k [max(0, H(π θ )-H(π θold ))]

[0061] s i (θ)=exp((1 / |y t |)×Σ T t=0 log(π θ / π θold ))

[0062] H(π θ )=-Σ T t=0 (π θ logπ θ )

[0063] In the formula:

[0064] L GRPO++ (θ) represents the objective function during the reinforcement learning training phase, and θ represents the weight vector consisting of all the weights of the cold-start model.

[0065] E[] represents the expectation, and max E[] represents the expectation function given the sample prompt word input;

[0066] i represents the ordinal number of the sampling result within the sampling group, and G represents the total number of sampling results within the sampling group;

[0067] π θ This represents the policy distribution (probability distribution of model weight output) for the current training epoch, π. θold This represents the policy distribution of the previous round of reinforcement learning (the probability distribution of the model weights output in the previous round).

[0068] y t This represents the t-th word vector within the i-th sampled sentence in the sampling group; t represents the index of the word vector in the current i-th sampled sentence.

[0069] s i (θ) represents the policy score difference at the output sentence level for the i-th sampled result within the sampling group output by the model weights θ, A i The dominance function represents the i-th sampling result within the sampling group;

[0070] clip(s i (θ),1-ε low ,1+ε high ) represents the clipping function; ε low ε represents the lower threshold for clipping. high C represents the upper limit threshold for clipping. H η represents the standard entropy coefficient, and η represents the entropy reduction inhibition coefficient.

[0071] E k [] represents the expected cross-entropy of the policy distribution after the k-th training round and the policy distribution before training, H() represents the policy model entropy function, min() represents the minimum value function, max() represents the maximum value function, and exp() represents the exponential function.

[0072] In this invention, all log values ​​are base 10.

[0073] This invention possesses strong general text comprehension capabilities, strong mathematical logic capabilities, and strong analytical code generation capabilities, enabling it to effectively perform table analysis LLM.

[0074] This invention addresses the collection of data analysis scenarios, including establishing a sampling evaluation feedback system, obtaining reasonable large-scale model output score feedback through matching, large-scale model calibration, and post-tool execution set comparison. It also considers potential external interaction dependencies in data analysis scenarios, designs a reasonable execution toolset, and obtains corresponding tool execution results (observation results). Based on feedback curves and sample analysis and evaluation during training, it optimizes algorithm loss components and sampling strategies to accelerate training efficiency and enhance training effects. Furthermore, it leverages capabilities in tabular data analysis and corresponding tool calls to create and collect evaluation sets, conducting multiple rounds of evaluation and iteration on the trained breakpoint weights until the model performance reaches expectations.

[0075] The beneficial effects of this invention are:

[0076] This invention, through the above research content and the design and construction of subsystems, fully simulates the interaction process of a large model in a real table-based data analysis scenario, forming a "simulation environment" for training large model intelligent agents, thereby enhancing the accuracy and application effect of large model-based data analysis intelligent agent interaction.

[0077] This invention establishes a reasonable multi-source data allocation and a multi-source data feedback evaluation system to achieve accurate evaluation and feedback for various types of data, enabling precise model training and updates. It also establishes a reinforcement learning objective function based on an improved GRPO algorithm. By reducing the KL loss term and adding an entropy decay suppression term, it saves memory on the reference model (REF-MODEL), significantly increasing the batch size for single-sample inference and thus improving training speed. Furthermore, it reduces the entropy decay rate of the training model's output, maintaining the diversity of sampled outputs to ensure continuous improvement in model accuracy during training.

[0078] In summary, this invention improves the practical application effect of large language models in the field of two-dimensional tabular data analysis through innovations in multi-source data setting and matching, training process, and computer operation design. Attached Figure Description

[0079] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0080] Figure 1 A detailed flowchart of the method provided in the embodiments of the present invention. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0082] like Figure 1 As shown, the specific process of the embodiment of the present invention is as follows:

[0083] Step 1: Data Collection

[0084] Collect multi-source data, including text question-and-answer data, mathematical logic reasoning data, code generation data, and table question-and-answer data. After generating a reference response text for each data item, use the corresponding model vocabulary dictionary to convert and process it into a set of word vector data that is subdivided into word vector tokens, thereby obtaining multi-source word vector data in the form of word vectors.

[0085] In practice, question-and-answer pairs are further created based on the question-and-answer data in the table.

[0086] The data volume distribution among text-based question-and-answer data, mathematical logic reasoning data, code generation data, and tabular question-and-answer data is as follows: text-based question-and-answer data 6.1%; mathematical logic reasoning data 41.7%; code generation data 2.57%; and tabular question-and-answer data (Python + SQL + accurate answers) totaling 213,554 entries, accounting for 49.6%.

[0087] The table-based question-and-answer data is divided into three types: self-built table-based QA dataset, SQL-QA dataset, and data analysis table-QA dataset with standard answers. The data volume ratio of the three datasets is as follows: self-built table-based QA dataset accounts for 87.8%; SQL-QA dataset accounts for 6.12%; and data analysis table-QA dataset with standard answers accounts for 6.12%.

[0088] The overall configuration of the multi-source word vector data is shown in Table 1 below:

[0089] Table 1

[0090] Data types Data Entries Remark General Questions and Answers 26261 Semantic understanding and general text question answering Mathematical reasoning 179564 Mathematical and logical reasoning Code generation 11072 Code dataset, including Python and other common languages Python solution for table-based question answering 187434 Self-built table QA dataset SQL-based solutions to table-related questions 13060 SQL-QA Dataset Question and answer table with accurate answers 13060 Table-QA dataset for mathematical analysis with standard answers

[0091] Each data entry in the text-based question-and-answer data, mathematical logic reasoning data, code generation data, and tabular question-and-answer data is used by a text generation and reasoning model to generate a reference response text, which is then added to that data entry. The text data of the answer is used as the label. A specific text generation and reasoning model implemented is, for example, the Qwen3-8B-instruct model.

[0092] Step Two: Data Answer Verification

[0093] The multi-source word vector data is filtered through repeated inference processing by multiple strong models;

[0094] Multiple different evaluation and scoring models are used as strong models. Each data point of multi-source word vector data is input into different evaluation and scoring models to obtain its own data inference score. Each evaluation and scoring model corresponds to a data inference score. Then, a comprehensive evaluation is performed based on the data inference score of each data point. Data points with all data inference scores not lower than the preset reliable answer score threshold are retained, i.e., the union is calculated and the remaining data points are discarded. In other words, if the data inference score obtained by any evaluation and scoring model is lower than the preset reliable answer score threshold, it is discarded.

[0095] Different evaluation and scoring models include DeepSeek-v3, Qwen-max, and gpt-4o.

[0096] The table-based question-and-answer pairs generated by a large model are used. Multiple strong models, such as DeepSeek-v3 and Qwen-max, are employed for repeated inference. For example, the inference temperature parameter (the API parameter called by the model, Temperature) can be set to 1.0, and the number of repeated inferences can be set to 3, performing three inferences on the same data point. Finally, only data where the answers from multiple inferences across multiple strong models are completely consistent are retained.

[0097] Step 3: Supervised Fine-tuning Training

[0098] A certain proportion of data is extracted from the multi-source word vector data obtained in step two, and then fed into the cold start model with preset initial weight parameters for training using a fully parameter-supervised fine-tuning method to obtain the trained cold start model.

[0099] The specific cold start model adopted is the qwen3-8b-instruct model, and its preset initial weight parameters are preset based on experience.

[0100] In practice, step three involves extracting approximately 3-5% of multi-source word vector data.

[0101] The purpose of this supervised fine-tuning training step is not to improve the model's ability in any domain (general or tabular), but to enhance the model's output response to specific output formats, with the aim of improving the efficiency of sampling effective samples during the agent interaction reinforcement learning phase.

[0102] In step three, the supervised fine-tuning of all parameters involves sequentially inputting the reference response text and prompt words for each data point, predicting the word vector token corresponding to the reference response text of the next data point, and then setting the following loss function with all parameters:

[0103] L SFT (θ)=-Σ T t=1 logp θ (y t |y <t ,x)

[0104] In the formula:

[0105] L SFT () represents the loss function for supervised fine-tuning training;

[0106] x represents the word vector token corresponding to the prompt word. The prompt word x is composed of the text of the question and condition constraints in each data item.

[0107] y t The token represents the word vector corresponding to the reference response text of the t-th data item, where t represents the data item number in the extracted multi-source word vector data, and T represents the total number of data items.

[0108] y <t This represents the word vector token corresponding to all reference response texts before the t-th data item;

[0109] p θ This represents the probability corresponding to the output word vector token;

[0110] θ represents the weight vector consisting of all the weights in the cold start model;

[0111] p θ (y t |y <t (x) represents the current parameters θ of the cold start model given the prompt word x and all currently generated word vectors y. <t In the case of the next word vector y t The predicted probability.

[0112] Step 4: Optimize the training algorithm (objective function)

[0113] The multi-source word vector data obtained in step two is input into the trained cold-start model and trained using reinforcement learning. Specifically:

[0114] S1. Sampling Inference: Each data point in the input multi-source word vector data is sampled and grouped in parallel, and then multiple inference predictions are performed. By setting a high generative model temperature parameter (specifically set to 1.0), the multiple inference predictions are different. According to the different data types, the multiple different inference results corresponding to each data point are input into the corresponding scoring feedback acquisition method and the corresponding sampling evaluation parameters are obtained. The evaluation parameters are added to each data point.

[0115] In the current step S1, based on the data type of the collected multi-source word vectors, and considering whether the multi-source word vector data contains standard answer labels (i.e., the standard answer to the question or task) and whether an additional executor is needed to generate the execution answer, the multi-source word vector data is divided into six types: general question and answer, mathematical reasoning, code generation, table question and answer solved by Python, table question and answer solved by SQL, and table question and answer with accurate answers. According to different types, after each inference prediction, it is processed according to four different scoring feedback acquisition methods. The four different scoring feedback acquisition methods are LLM-EVAL, pre-guided evaluation method, post-run guided evaluation method, and post-run model evaluation method.

[0116] In practice, the data is categorized into several types according to the multi-source data allocation table above, including LLM-EVAL, pre-launch evaluation, post-run evaluation, and post-run model evaluation, as shown in Table 2 below:

[0117] Table 2

[0118] Data types How to get rating feedback Remark General Questions and Answers Pre-model evaluation method Evaluation of Results Using Large Models Mathematical reasoning Pre-guidance evaluation method Rule Extraction Code generation Post-run guidance and evaluation methods Rule extraction after running Python solution for table-based question answering Post-run model evaluation method Large model evaluation after running SQL-based solutions to table-related questions Post-run guidance and evaluation methods Rule extraction after running Question and answer table with accurate answers Post-run guidance and evaluation methods Rule extraction after running

[0119] S2. Input the multi-source word vector data with evaluation parameters obtained in step S1 into the cold start model after training in step three for reinforcement learning training.

[0120] S3. For the reinforcement learning training in step S2, specifically establish an objective function that can effectively suppress entropy decay and significantly reduce memory overhead, and train with the goal of maximizing the expected value of the objective function.

[0121] In step S3, the objective function is specifically set using the following formula for reinforcement learning training:

[0122] L GRPO++ (θ)=max E[(1 / G)×Σ G i=0 min((s i (θ)A i , clip(s i (θ),1-ε low ,1+ε high A i)]

[0123] +C H E k [H(π θ )]-ηE k [max(0, H(π θ )-H(π θold ))]

[0124] s i (θ)=exp((1 / |y t |)×Σ T t=0 log(π θ / π θold ))

[0125] H(π θ )=-Σ T t=0 (π θ logπ θ )

[0126] In the formula:

[0127] L GRPO++ (θ) represents the objective function during the reinforcement learning training phase, and θ represents the weight vector consisting of all the weights of the cold-start model.

[0128] E represents the expectation, and max E[] represents the expectation function given the sample prompt words input;

[0129] i represents the ordinal number of the sampling result within a single sampling group, and G represents the total number of sampling results within a single sampling group. During the sampling process of a data point, G different sampling results are generated. These G different sampling results are grouped together to form a sampling group. The sampling results are all inferred and predicted sequentially, and different sampling results are inferred and predicted differently.

[0130] π θ This represents the policy distribution (probability distribution of model weight output) for the current training epoch, π. θold This represents the policy distribution of the previous round of reinforcement learning (the probability distribution of the model weights output in the previous round).

[0131] t represents the index of the word vector in the current i-th sampled sentence;

[0132] y t This represents the t-th word vector within the i-th sampled sentence in the sampling group;

[0133] s i (θ) represents the policy score difference at the output sentence level for the i-th sampled result within the sampling group output by the model weights θ, Ai The dominance function represents the i-th sampling result within the sampling group;

[0134] clip(s i (θ),1-ε low ,1+ε high () represents the clipping function, which modifies the policy score difference parameter s. i (θ) is constrained in 1-ε low ,1+ε high Between, that is, when the strategy score difference parameter s i When (θ) is less than or greater than the boundary value, the policy score difference parameter s will be adjusted. i (θ) is set to the corresponding boundary value; otherwise, it remains unchanged.

[0135] ε low ε represents the lower threshold for clipping. high C represents the upper limit threshold for clipping. H η represents the standard entropy coefficient, and η represents the entropy reduction inhibition coefficient.

[0136] E k [] represents the expected cross-entropy of the policy distribution after the k-th training round and the policy distribution before training, H() represents the policy model entropy function, min() represents the minimum value function, max() represents the maximum value function, and exp() represents the exponential function.

[0137] In the last two terms of the objective function above, the first term encourages the model to maintain a high entropy value during exploration, while the second term suppresses the rate at which entropy decreases during each update.

[0138] Existing target algorithms still suffer from the following problems: 1) Insufficient sampling efficiency; 2) The KL divergence term constrains the efficiency of model updates and the corresponding effect; 3) The target function at the output character level focuses too much on generation details, which makes the model overly sensitive to the generation of a few abnormal characters, thereby reducing the robustness of the model and affecting the convergence and performance improvement speed; 4) During the training process, existing algorithms are prone to the "entropy collapse" phenomenon, where the sampling consistency of each sample converges rapidly, the model's exploratory nature deteriorates, and the upscaling space decreases sharply.

[0139] To address the above issues, this invention mainly improves the objective function of existing algorithms in the following ways:

[0140] 1) Remove the KL divergence term from the existing algorithm. Removing this term can save the loading of the reference model (Refine-model) and further save training memory space.

[0141] 2) Modify the ε sub-item in the clip function to divide it into ε sub-items. low and ε highBased on the successful research of the DAPO algorithm, ε low and ε high Setting them to (0.2, 0.28) can effectively increase sampling efficiency and better crop the output score distribution.

[0142] 3) Modify the objective function from word level to sentence level.

[0143] 4) Incorporate entropy reward and entropy reduction rate suppression terms to continuously encourage the model to maintain high entropy exploration during training and avoid rapid entropy collapse.

[0144] After determining the objective function, the reinforcement learning phase of training can begin. One round of reinforcement learning training can consist of the following steps:

[0145] 1) Load the model to be trained and perform sampling. Each sampled data point is evaluated with n inferences, and the sampling results corresponding to every n identical samples are aggregated into a group G;

[0146] 2) Calculate the objective function L GRPO++ The values ​​of each sub-item in (θ) (i.e., the calculated values ​​of each sub-item in the objective function formula) are stored in the cache;

[0147] 3) Load the values ​​of each sub-item in the cache and calculate the objective function L. GRPO++ The function value of (θ) is used to update the weights of the model to be trained using the gradient descent method;

[0148] 4) Repeat process 1)-3) continuously until the maximum number of iterations is reached.

[0149] After completing the methodology, we considered four key capabilities that need to be emphasized in practical tabular data analysis scenarios: semantic understanding, code generation, long thought processes for complex problems, and QA for complex tables. The selected validation set (Benchmark) is shown in Table 3 below:

[0150] Table 3

[0151] Benchmark Classification effect HumanEval Coding Evaluate the model's code generation capabilities MBPP Coding Evaluate the model's code generation capabilities GSM8K Math-logic Assess the model's logical reasoning and mathematical abilities MATH Math-logic Assess the model's logical reasoning and mathematical abilities GPQA-diamond LongCOT-Logic The assessment model's ability to perform long-term logical reasoning and thought processes. Spider 1.0 Database-QA Database table QA capabilities Bird Database-QA Database table QA capabilities TableBench Table-QA Single-round form QA capability TableGPT-Agent Table-QA Self-built dataset, multi-round Agentic-QA capability

[0152] Regarding the reinforcement learning weight optimization process, this invention does not train all samples at once and then evaluate the weights and iterate the data during the interactive reinforcement learning training of the agent, as is the case with traditional supervised fine-tuning training.

[0153] Step 5: Applying the cold-start model trained using reinforcement learning to answer text questions in real-world scenarios can improve the accuracy of the model's responses.

[0154] The example uses the design and training process of a virtual environment for a large-scale intelligent model of a factory's supply chain database analysis agent as an example. Its entire workflow is based on the following parts.

[0155] 1) Obtain general, table-related datasets, including shareGPT and TableBench, from open-source communities such as Huggingface and GitHub. Simultaneously, based on the factory's own table data, construct and filter table QA data pairs using a combination of manual and generative models.

[0156] 2) The same method of manually verifying answers using data from both human and large-scale models is employed. The method for verifying answers from the large-scale models is described in step 2) of the instruction manual. Human verification in this step primarily uses a sampling method, such as selecting 1 out of 10. The accuracy rate of this manual sampling needs to be above 0.95; otherwise, it will severely impact the subsequent model training effect.

[0157] 3) The model is trained using the llamafactory training framework and supervised fine-tuning. On a single H800 server, the core training parameters are as follows:

[0158] Learning rate: 2e -6 ;

[0159] Maximum sample length cutoff-len: 16384;

[0160] single-device training batch size: 32;

[0161] Gradient accumulation steps: 32.

[0162] 4) Develop and test the toolset needed for the intelligent agent in interactive sampling and actual deployment, specifically including Python & SQL executors, web search tools, RAG tools, etc.

[0163] 5) Develop and construct an evaluation feedback system. For rule-matching feedback, construct specific regular expression extraction and structured extraction scripts, ensuring stable script operation. For the Generative Large Language Model Evaluation (LLM-EVAL) method, the core is to construct appropriate scoring prompts. Specifically, a model evaluation prompt for reference is shown below:

[0164] 6) Based on the training framework OpenRLHF or VERL, develop and implement the training algorithm described in step 6 of the manual, and complete the corresponding training script. On a single H100 machine, the core parameter configurations during training are as follows:

[0165] Learning rate: 2e -6 ;

[0166] Number of samples per step: rollout_batch_size: 512;

[0167] Number of times a single sample is sampled, n_sample_prompt: 16;

[0168] Total number of training cycles per GPU: micro_train_batch_size: 16;

[0169] KL estimator types: kl_estimator: k1;

[0170] Initial kl-loss weights init_kl_coef: 0.0;

[0171] PPO update count (ppo-epoch): 2;

[0172] Ppo-eps clipping parameters eps_clip_low_high: (0.2, 0.28)

[0173] 7) Based on the above validation set, weight training and optimization iterations are performed using the following screening and iteration steps:

[0174] 1. Use the original weights to perform sampling inference (three inferences and obtain sample score feedback), remove samples that are all correct or all wrong in three inferences, and keep the remaining samples for reinforcement learning training;

[0175] 2. Set the Step-batch-size to 512, meaning that 512 different samples are sampled and trained each time. Training typically takes 60-90 steps, corresponding to 30,000-46,000 samples. Use the validation set to evaluate the model weights corresponding to steps 60 and 90, selecting the best weights and discarding those corresponding to already trained samples.

[0176] 3. Using the weights of the best selection, continue the sampling reasoning of step 1) and eliminate samples that are all correct or all wrong.

[0177] 4. Perform multiple rounds of steps 1) to 3) until the remaining samples are 0 or the validation set score has reached the expected value.

[0178] The above specific embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

[0179] The above description is only a preferred embodiment of the present invention. Therefore, all equivalent changes or modifications made to the structure, features and principles described in the claims of this patent application are included in the scope of this patent application.

Claims

1. A method for training and applying a large-scale model for tabular data analysis based on agent-based interactive reinforcement learning, characterized in that: The method is specifically as follows: Step 1: Collect multi-source data, including text question-and-answer data, mathematical logic reasoning data, code generation data, and table question-and-answer data. After generating a reference response text for each data item, use the corresponding model vocabulary dictionary to convert it into a set of word vectors that are further subdivided into word vector tokens, thereby obtaining multi-source word vector data in the form of word vectors. Step 2: Filter the multi-source word vector data through repeated inference processing using multiple strong models; Step 3: Extract a certain proportion of data from the multi-source word vector data obtained in Step 2, and input it into the cold start model with preset weight parameters for fully supervised fine-tuning training to obtain the trained cold start model. Step 4: Input the multi-source word vector data obtained in Step 2 into the trained cold-start model and train it using reinforcement learning methods; Step 5: Apply the cold-start model trained using reinforcement learning to answer text questions in real-world scenarios; The reinforcement learning methods in step four include: S1. Sampling Inference: Each data point in the input multi-source word vector data is sampled and grouped, and then multiple inference predictions are performed. By setting the temperature parameter of the generative model, the multiple inference predictions are different. According to the different data types, the multiple different inference results corresponding to each data point are input into the corresponding scoring feedback acquisition method and the corresponding sampling evaluation parameters are obtained. The evaluation parameters are added to each data point. S2. Input the multi-source word vector data with evaluation parameters obtained in step S1 into the cold start model after training in step three for reinforcement learning training. S3. Specifically, for the reinforcement learning training in step S2, establish an objective function that can suppress entropy decay and reduce memory overhead, and train with the goal of maximizing the expected value of the objective function. In step S3, the objective function is specifically set using the following formula for reinforcement learning training: L GRPO++ (θ)=max E[(1 / G)×Σ G i=0 min((s i (i)A i , clip(s) i (i),1-e low ,1+e high )A i )] +C H E k [H(π θ )]-ηE k [max(0, H(π θ )-H(π θold ))] s i (θ)=exp((1 / |y t |)×S T t=0 log (π θ / p θold )) H(p θ )=-S T t=0 (p θ log p θ ) In the formula: L GRPO++ (θ) represents the objective function during the reinforcement learning training phase, and θ represents the weight vector consisting of all the weights of the cold-start model. E represents the expectation, and max E[] represents the expectation function given the sample prompt words input; i represents the ordinal number of the sampling result within the sampling group, and G represents the total number of sampling results within the sampling group; π θ π represents the distribution of reinforcement learning policies in the current training round. θold This represents the policy distribution of the previous round of reinforcement learning; y t This represents the t-th word vector within the i-th sampled sentence in the sampling group; t represents the index of the word vector in the current i-th sampled sentence. s i (θ) represents the policy score difference at the output sentence level for the i-th sampled result within the sampling group output by the model weights θ, A i The dominance function represents the i-th sampling result within the sampling group; clip(s i (θ),1-ε low ,1+ε high ) represents the clipping function; ε low ε represents the lower threshold for clipping. high C represents the upper limit threshold for clipping. H η represents the standard entropy coefficient, and η represents the entropy reduction inhibition coefficient. E k [] represents the expected cross-entropy of the policy distribution after the k-th training round and the policy distribution before training, H() represents the policy model entropy function, min() represents the minimum value function, max() represents the maximum value function, and exp() represents the exponential function.

2. The method for training and applying a large-scale model for tabular data analysis based on agent-based reinforcement learning according to claim 1, characterized in that: The text-based question-and-answer data includes text data based on descriptive questions, their constraints, and answers; the mathematical logic reasoning data includes text data on mathematical operation questions, their constraints, and operation results; the code generation data includes text data on specific application questions, their constraints, and modeling code; the table-based question-and-answer data refers to text data based on known table data, its constraints, and answers; the table-based question-and-answer data is divided into self-built table QA datasets, SQL-QA datasets, and mathematical analysis Table-QA datasets with standard answers.

3. The method for training and applying a large-scale model for tabular data analysis based on agent-based interactive reinforcement learning according to claim 1, characterized in that: Each piece of data in the text-based question-and-answer data, mathematical logic reasoning data, code generation data, and tabular question-and-answer data is used to generate a reference response text through a text generation reasoning model and added to that piece of data.

4. The method for training and applying a large-scale model for tabular data analysis based on agent-based interactive reinforcement learning according to claim 1, characterized in that: Step two specifically involves: Multiple different evaluation and scoring models are used as strong models. Each data point of multi-source word vector data is input into different evaluation and scoring models to obtain its own data inference score. Then, a comprehensive evaluation is carried out based on the data inference score of each data point. Data points with all data inference scores not lower than the preset reliable answer score threshold are retained, while other data points are discarded.

5. The method for training and applying a large-scale model for tabular data analysis based on agent-based interactive reinforcement learning according to claim 1, characterized in that: The cold start model specifically adopts the qwen3-8b-instruct model.

6. The method for training and applying a large-scale model for tabular data analysis based on agent-based interactive reinforcement learning according to claim 1, characterized in that: In step three, the supervised fine-tuning of all parameters involves sequentially inputting the reference response text and prompt words for each data point, predicting the word vector corresponding to the reference response text for the next data point, and then setting the following loss function with all parameters: L SFT (θ)=-Σ T t=1 log p θ (y t |y <t ,x) In the formula: L SFT () represents the loss function for supervised fine-tuning training; x represents the word vector corresponding to the prompt word, which is composed of the question and condition constraints in the data; y t The word vector represents the reference reply text corresponding to the t-th data item, where t represents the data item number and T represents the total number of data items. y <t This represents the word vectors corresponding to all reference response texts for data up to the t-th entry; p θ This represents the probability corresponding to the output word vector; θ represents the weight vector consisting of all the weights in the cold start model; p θ (y t |y <t (x) represents the current parameters θ of the cold start model given the prompt word x and all currently generated word vectors y. <t In the case of the next word vector y t The predicted probability.

7. The method for training and applying a large-scale model for tabular data analysis based on agent-based interactive reinforcement learning according to claim 1, characterized in that: In step S1, the multi-source word vector data is divided into six types: general question and answer, mathematical reasoning, code generation, table question and answer solved by Python, table question and answer solved by SQL, and table question and answer with accurate answers. According to different types, after each inference prediction, the data is processed according to four different scoring feedback acquisition methods. The four different scoring feedback acquisition methods are LLM-EVAL, pre-guided evaluation method, post-run guided evaluation method, and post-run model evaluation method.

Images