Table data prediction optimization method based on table questions and answers
By automatically optimizing the feature values based on table questions and answers, the resource waste and time consumption problems caused by manual adjustment of feature values in the prior art are solved, and efficient and accurate data prediction is achieved.
Patent Information
- Application Number
- CN202411792731.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-07
- Publication Date
- 2025-03-14
AI Technical Summary
When performing data prediction in machine learning, prior art requires manual adjustment of feature values, resulting in waste of resources and excessive time consumption.
The table-based Q&A method is adopted, and the table data is converted into eigenvalues through natural language processing, and the xgboost model is used to predict and tune the eigenvalues, and the model is optimized through continuous online learning.
The optimization of automated feature values is achieved, reducing manual intervention, improving the efficiency and accuracy of data prediction, and saving resources and time.
Smart Images
Figure CN119623645A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of eigenvalue tuning in machine learning, and in particular to a table data prediction optimization method based on table question and answer and a decision method thereof. Background Art
[0002] Xgboost and other tools are data modeling and prediction modeling tools in machine learning. In the traditional machine learning data prediction tuning scheme, it is necessary to manually adjust and select feature values, build a prediction model and make predictions, and then modify the feature value selection according to the data fit of the prediction set and the test set. After repeated selection, the optimal feature value is selected, which is the feature value engineering. This process consumes a lot of manpower and time resources.
[0003] Therefore, it is necessary to provide a table data prediction optimization method based on table question and answer to overcome the shortcomings of the existing technology. Summary of the invention
[0004] The purpose of the present invention is to avoid the shortcomings of the prior art and provide a table data prediction optimization method based on table question and answer, which can obtain the optimal feature value required for prediction.
[0005] The purpose of the present invention is achieved through the following technical measures.
[0006] A table data prediction optimization method based on table question answering is provided, which is carried out through the following steps:
[0007] Step 1: The table question answering model outputs the predicted dimensions, feature values, and data sets in the table according to the user's input description, and extracts the data sets and feature sets;
[0008] Step 2: Combine the generated feature set and data set into training set D-Train and test set D-Test according to the prediction dimension; build a model and calculate the fit;
[0009] Step 3: Assemble the fine-tuning dataset and perform model fine-tuning. Specifically, use the prediction dimension and the optimal feature set as the fine-tuning dataset, and in turn fine-tune the table question answering model to generate a new feature value extraction model M-sn.
[0010] Table question answering is a function in deep learning that can query, extract, and integrate the content in a given table through natural language.
[0011] Preferably, the above-mentioned table data prediction optimization method based on table question and answer further includes step 4, calling the fine-tuning model, specifically:
[0012] When the prediction task of the same prediction dimension appears again, the corresponding model M-sn is found in the model library through the prediction dimension;
[0013] At the same time, the maximum fit degree and the corresponding feature set generated again are assembled into a new fine-tuning data set, and the model is further fine-tuned. The fine-tuned model is updated to the model library as a new feature value extraction model M-sn.
[0014] Preferably, in the above-mentioned table data prediction optimization method based on table question and answer, step 1 specifically includes:
[0015] Step 1.1, data preprocessing, that is, preprocessing the table data in the input table file through a conversion program, and converting the table data into JSON data;
[0016] The specific conversion logic is to mark the table in order, generate a mark code, and record it in the table_id item; record the table name in the table_name item; record the column names in the table in the header_name item in column order; take the content of each row in the table as an array, and record multiple rows into a two-dimensional array in the row item;
[0017] Step 1.2: Use natural language to describe the required feature value name, data item, and prediction dimension, and input the description language and JSON data into the table question answering model at the same time. The table question answering model uses the natural language processing capability Text-to-SQL to create a SQL query statement corresponding to the question, and uses the SQL query statement to query the input data table or database;
[0018] Step 1.3: The json data result obtained by model query is converted into a feature set T containing feature data and a data set D that needs to be predicted.
[0019] Preferably, in the above-mentioned table data prediction optimization method based on table question and answer, the specific conversion logic in step 1.3 is: extract header_name as the column name, and then map the array in row to each row of the table to generate a new result table.
[0020] Preferably, in the above-mentioned table data prediction optimization method based on table question and answer, step 2 specifically includes:
[0021] Step 2.1, sort the data set according to the prediction dimension, and arrange it in the positive order of the prediction direction. Select the front data block as the data set D-sub and the back data block as the test set D-Test, and cut them according to the cutting ratio.
[0022] Step 2.2, the dataset D-sub is merged with the feature set T column. The merging is based on the prediction dimension item of the dataset D-sub, and the feature values with the same prediction dimension are selected to form the training set D-Train.
[0023] The training set is used as the original data for model training, and the test set is used as the test data for performance comparison after model training and prediction to evaluate the performance of the model.
[0024] Step 2.3, select some feature values of the training set D-Train, in addition to retaining the prediction dimension TT and data DD, select one or more columns in the column order, and combine them to generate a sub-training set D-Train-sub; generate a decision tree-based model through the XGBRegressor function of the xgboost library, and then use the fit function to import the sub-training set, train the model, and generate a prediction model M;
[0025] Step 2.4, based on the feature items selected in the sub-training set D-Train-sub of this cycle, and based on the prediction dimension of the test set D-Test, query the values of the feature items of the same prediction dimension in the feature set T to construct the sub-test set D-Test-Sub;
[0026] Use the predict function of the xgboost library to calculate the prediction data of the prediction model M on the sub-test set D-Test-Sub to generate the prediction set D-Fore;
[0027] Step 2.5, calculate the goodness of fit for the test set D-Test and the prediction set D-Fore, calculate the root mean square error RMSE of the predicted values DF and DD in the two data sets, and obtain the goodness of fit De;
[0028] Step 2.6, cyclically select different eigenvalue combinations, repeat steps 2.3 to 2.5, train, calculate and fit different eigenvalues, and compare to obtain the maximum fitting degree De-max and its corresponding feature set T-max.
[0029] Preferably, in the above-mentioned table data prediction optimization method based on table question and answer, step 3 specifically includes:
[0030] Step 3.1, assemble the predicted dimension, the maximum fit De-max and the corresponding feature set T-max into the fine-tuning dataset D-stf;
[0031] In step 3.2, use the fine-tuning dataset D-stf to fine-tune the tabular question answering model M. The fine-tuning behavior is completed using the Trainer trainer of the Transformers framework. Load the tabular question answering model M in the trainer, and load the fine-tuning dataset D-stf in a loop. After fine-tuning, a new model file Ms is generated.
[0032] Step 3.3, merge the new model file with the prediction dimension record.
[0033] Preferably, in the above-mentioned table data prediction optimization method based on table question and answer, the table question and answer model in step 1 is SPACE-T or Berta.
[0034] Preferably, the above-mentioned table data prediction optimization method based on table question and answer, data prediction in machine learning can also use the LightGBM library.
[0035] The present invention is based on a table data prediction optimization method for table question answering, which uses table question answering capabilities to perform feature value engineering and optimize the system of the xgboost data prediction model. It combines the natural language processing advantages in deep learning and the data prediction capabilities of machine learning. Through the table question answering capabilities in deep learning, the data in the table is extracted and integrated according to the user's natural language, and converted into feature values in the machine learning xgboost model, and data modeling and data prediction are performed based on the feature values.
[0036] The technical solution of the present invention has the following innovative features:
[0037] Innovation 1: Using natural language and deep learning models, the data in the original table is integrated into feature values, and the fit calculation of the prediction model is performed repeatedly to obtain the optimal feature values required for prediction.
[0038] Innovation 2: Through continuous online learning, the optimal feature values and prediction dimensions are assembled into a fine-tuning data set, and the fine-tuning data set is used to fine-tune the deep learning model, and the fine-tuning version is quickly iterated. When making predictions of similar dimensions again, the direction and priority of integrating feature values can be adjusted through the tuned model, and the optimal prediction model and prediction value that conforms to the natural language input rules can be output more quickly. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.
[0040] Figure 1 It is a flow chart of a table data prediction optimization method based on table question and answer of the present invention.
[0041] Figure 2It is a data extraction flow chart of step 1 in the method of the present invention.
[0042] Figure 3 It is a data preprocessing conversion relationship diagram in the method of the present invention.
[0043] Figure 4 It is a JSON data result conversion relationship diagram in the method of the present invention.
[0044] Figure 5 It is a flow chart of data set cutting and merging in the method of the present invention.
[0045] Figure 6 It is a flow chart of the cyclic training in the method of the present invention.
[0046] Figure 7 It is a flow chart of model fine-tuning in the method of the present invention.
[0047] Figure 8 It is a model calling flow chart in the method of the present invention. DETAILED DESCRIPTION
[0048] The present invention will be further described with reference to the following examples.
[0049] Example 1.
[0050] A table data prediction optimization method based on table question answering, such as Figure 1 As shown, proceed as follows:
[0051] Step 1: The table question answering model outputs the predicted dimensions, feature values, and data sets in the table according to the user's input description, and extracts the data sets and feature sets;
[0052] Step 2: Combine the generated feature set and data set into training set D-Train and test set D-Test according to the prediction dimension; build a model and calculate the fit;
[0053] Step 3: Assemble the fine-tuning dataset and fine-tune the model. Specifically, the prediction dimension and the optimal feature set are used as the fine-tuning dataset, and the table question answering model is fine-tuned in turn to generate a new feature value extraction model M-sn.
[0054] It also includes step 4, calling the fine-tuning model, specifically:
[0055] When the prediction task of the same prediction dimension appears again, the corresponding model M-sn is found in the model library through the prediction dimension;
[0056] At the same time, the maximum fit degree and the corresponding feature set generated again are assembled into a new fine-tuning data set, and the model is further fine-tuned. The fine-tuned model is updated to the model library as a new feature value extraction model M-sn.
[0057] Table question answering is a function in deep learning that can query, extract, and integrate the content in a given table through natural language. In this example, the table question answering model in step 1 is SPACE-T. It should be noted that the table question answering model in step 1 can also be selected from Berta, etc.
[0058] The present invention is based on a table data prediction optimization method for table question answering, which uses table question answering capabilities to perform feature value engineering and optimize the system of the xgboost data prediction model. It combines the natural language processing advantages in deep learning and the data prediction capabilities of machine learning. Through the table question answering capabilities in deep learning, the data in the table is extracted and integrated according to the user's natural language, and converted into feature values in the machine learning xgboost model, and data modeling and data prediction are performed based on the feature values.
[0059] The table data prediction optimization method based on table question answering, the data extraction process of step 1 is as follows Figure 2 Step 1 specifically includes:
[0060] Step 1.1, data preprocessing, that is, preprocessing the table data in the input table file through a conversion program and converting the table data into json data.
[0061] The specific conversion logic is to mark the table in order, generate a tag code, such as biao-1, biao-2..., and record it in the table_id item; record the table name in the table_name item; record the column names in the table in the order of the columns in the header_name item; take the content of each row in the table as an array, and record multiple rows into a two-dimensional array in the row item. The data preprocessing conversion relationship is as follows Figure 3 shown.
[0062] In step 1.2, describe the required feature value names, data items, and prediction dimensions (such as time) in natural language, and input the description language and JSON data into the tabular question answering model at the same time. The tabular question answering model uses the natural language processing capability Text-to-SQL to create the SQL query statement corresponding to the question, and uses the SQL query statement to query the input data table or database.
[0063] Determine the prediction dimension. Before prediction, you need to clarify the prediction dimension of the entire modeling prediction. For example, if you are predicting a certain data in a certain time period in the future, the prediction dimension is the time item; if you are predicting a certain indicator of a government region, the prediction dimension is the government region ID.
[0064] Formulate query statements. In the natural language used for query, use the prediction dimension as the query condition and the content to be output as the query result. For example, if you need to use TT as the query dimension and the three values of AA, BB, and AA+BB as output items, you need to use the statement "output the data of AA, BB, and AA+BB items in the range of t1 to t4 in the TT item as the feature set" to input it into the table question answering model.
[0065] Clarify the output items. During the query process, at least two natural language interactions are performed to query the data items that need to be predicted and multiple related items that may be related to the data items through the same prediction dimension. The former is used as a data set, and the latter can expand the mathematical results in addition to the existing data items. For example, in addition to outputting AA and BB, it can also output SUM(AA,BB), mean AVG(AA,BB), standard deviation STDEV(AA,BB) and other data as feature data sets. Both data sets contain the same prediction dimension.
[0066] Step 1.3: The json data result obtained by model query is converted into a feature set T containing feature data and a data set D that needs to be predicted.
[0067] The specific conversion logic in step 1.3 is: extract header_name as the column name, then map the array in row to each row of the table to generate a new result table. The json data result conversion relationship is as follows Figure 4 shown.
[0068] Step 2: Select the eigenvalue modeling and calculate the fit, that is, combine the generated feature set and data set D with the feature set T into the training set D-Train and the test set D-Test according to the prediction dimension. The data set cutting and merging flow chart is as follows: Figure 5 shown.
[0069] The table data prediction optimization method based on table question answering, step 2 specifically includes:
[0070] Step 2.1, sort the data set according to the prediction dimension, and sort it in the positive order of the prediction direction. For example, if you need to predict data at a certain time in the future, sort it in positive order according to the time dimension, select the front data block as the data set D-sub, and the back data block as the test set D-Test, and cut it according to the cutting ratio. The cutting ratio can be selected from 5:5 to 7:3. It should be noted that the specific cutting ratio can be flexibly set according to actual needs.
[0071] Step 2.2, the dataset D-sub is merged with the feature set T column. The merge is based on the prediction dimension items of the dataset D-sub, and the same prediction dimension and feature values are selected to form the training set D-Train. For example, if the prediction dimension TT of the dataset is t1, t2, and t3, then the feature set data with TT of t1, t2, and t3 is selected and merged with the dataset.
[0072] The training set serves as the original data for model training, and the test set serves as the test data for performance comparison after model training and prediction to evaluate the performance of the model.
[0073] Step 2.3, select some feature values of the training set D-Train, in addition to retaining the prediction dimension TT and data DD, select one or more columns in column order, and combine them to generate a sub-training set D-Train-sub; use the XGBRegressor function of the xgboost library to generate a decision tree-based model, and then use the fit function to import the sub-training set, train the model, and generate a prediction model M.
[0074] Step 2.4: Based on the feature items selected in the sub-training set D-Train-sub of this cycle, and then based on the prediction dimensions of the test set D-Test, query the values of the feature items of the same prediction dimensions in the feature set T to construct the sub-test set D-Test-Sub.
[0075] For example, in this cycle, the feature item selected by the sub-training set D-Train-sub is AA, and the predicted dimension TT value of the test set D-Test is t4, t5, t6. Therefore, the values a4, a5, a6 of the AA column with TT t4, t5, t6 are obtained from the feature set T, and the sub-test set D-Test-Sub is constructed by combining the obtained TT column values and AA column values.
[0076] Use the predict function of the xgboost library to calculate the predicted data of the prediction model M on the sub-test set D-Test-Sub to generate the prediction set D-Fore.
[0077] Step 2.5, calculate the goodness of fit for the test set D-Test and the prediction set D-Fore, calculate the root mean square error RMSE of the predicted values DF and DD in the two data sets, and obtain the goodness of fit De;
[0078] Step 2.6, select different eigenvalue combinations in a loop, repeat steps 2.3 to 2.5, train, calculate and fit different eigenvalues, and compare to obtain the maximum fitting degree De-max and its corresponding feature set T-max. The loop training flow chart of repeating steps 2.3 to 2.5 is as follows: Figure 6 shown.
[0079] Step 3: Assemble the fine-tuning dataset and perform model fine-tuning. The flowchart of model fine-tuning is as follows: Figure 7 shown.
[0080] Step 3 specifically includes:
[0081] In step 3.1, the prediction dimension, the maximum fitting degree De-max and the corresponding feature set T-max are assembled into the fine-tuning dataset D-stf.
[0082] In step 3.2, use the fine-tuning dataset D-stf to fine-tune the tabular question answering model M. The fine-tuning behavior is completed using the Trainer trainer of the Transformers framework. Load the tabular question answering model M in the trainer, and load the fine-tuning dataset D-stf in a loop. After fine-tuning, a new model file Ms is generated.
[0083] Step 3.3, merge the new model file with the prediction dimension record.
[0084] Step 4: Call the fine-tuning model. The model calling flow chart is as follows: Figure 8 shown.
[0085] When the prediction task of the same prediction dimension appears again, the corresponding model M-sn can be found in the model library through the prediction dimension, which improves the efficiency of querying and integrating feature value data. At the same time, the maximum fit and the corresponding feature set generated again can be assembled into a new fine-tuning data set, and the model can be further fine-tuned. The fine-tuned model is updated to the model library to achieve the ability of continuous online learning optimization.
[0086] It should be noted that other libraries can be used to replace xgboost for data prediction in machine learning, such as LightGBM.
[0087] The technical solution of the present invention uses a natural language method to integrate the data in the original table into feature values through a deep learning model, and cyclically calculates the degree of fit of the prediction model to obtain the optimal feature values required for prediction. Through continuous online learning, the optimal feature values and prediction dimensions are assembled into a fine-tuning data set, and the deep learning model is fine-tuned using the fine-tuning data set, and the fine-tuning version is quickly iterated. When a prediction of similar dimensions is made again, the direction and priority of the integrated feature values can be adjusted through the tuned model, and the optimal prediction model and prediction value that conforms to the natural language input rules can be output more quickly.
[0088] Example 2.
[0089] A table data prediction optimization method based on table question and answer of the present invention is described through a specific example.
[0090] This example uses astronomical tide table data to predict tide levels.
[0091] The original input data is the historical data table of astronomical tides, the predicted data is the tide level, the prediction dimension is the timestamp, and the year, month, day, week, hour, and minute are all set as feature values.
[0092] Table 1 Excerpt from the historical data table of astronomical tides
[0093] Timestamp Year moon day week Hour point Sensor pressure tide level 2023 / 1 / 20:33 2023 1 2 1 0 33 39.31274584 -0.39 2023 / 1 / 26:33 2023 1 2 1 6 33 63.20567369 -0.2 2023 / 1 / 210:31 2023 1 2 1 10 31 31.95917135 -0.34 2023 / 1 / 218:28 2023 1 2 1 18 28 3.487637142 0.48 2023 / 1 / 31:45 2023 1 3 1 1 45 6.265383652 -0.57 2023 / 1 / 37:59 2023 1 3 1 7 59 80.234737 -0.24 2023 / 1 / 310:53 2023 1 3 1 10 53 12.26577703 -0.31 2023 / 1 / 319:03 2023 1 3 1 19 3 51.30981871 0.58 2023 / 1 / 42:28 2023 1 4 1 2 28 34.61607382 -0.73 2023 / 1 / 48:51 2023 1 4 1 8 51 87.67646383 -0.25 2023 / 1 / 411:14 2023 1 4 1 11 14 65.0271353 -0.31 2023 / 1 / 419:39 2023 1 4 1 19 39 33.11018714 0.65 2023 / 1 / 53:06 2023 1 5 1 3 6 73.24674727 -0.85 2023 / 1 / 59:28 2023 1 5 1 9 28 82.84365408 -0.25 2023 / 1 / 511:47 2023 1 5 1 11 47 68.78572431 -0.33 2023 / 1 / 520:16 2023 1 5 1 20 16 73.3499918 0.7 2023 / 1 / 63:42 2023 1 6 1 3 42 22.08796862 -0.92 2023 / 1 / 610:02 2023 1 6 1 10 2 20.72849513 -0.25 2023 / 1 / 612:27 2023 1 6 1 12 27 26.09647647 -0.36 2023 / 1 / 620:56 2023 1 6 1 20 56 28.0904236 0.72 2023 / 1 / 74:15 2023 1 7 1 4 15 17.76823636 -0.94
[0094] Convert the data table to json through the conversion program. Query the data set and feature set from the table learning model.
[0095] Query statement: Query the time and tide data from 2023 to the present.
[0096] The output data set is shown in Table 2 (excerpt).
[0097] Table 2
[0098]
[0099]
[0100] Query statement: Query the astronomical tide data, and output the timestamp, year, month, day, week, hour, and minute data from 2023 to the present as the feature set.
[0101] The output feature set (excerpt) is shown in Table 3.
[0102] Table 3
[0103]
[0104]
[0105] The data set is split into 7:3 parts and merged according to the timestamp and feature set to obtain the training set and the test set.
[0106] The training set (excerpt) is shown in Table 4.
[0107] Table 4
[0108] Timestamp Year moon day week Hour point tide level 2023 / 1 / 20:33 2023 1 2 1 0 33 -0.39 2023 / 1 / 26:33 2023 1 2 1 6 33 -0.2 2023 / 1 / 210:31 2023 1 2 1 10 31 -0.34 2023 / 1 / 218:28 2023 1 2 1 18 28 0.48 2023 / 1 / 31:45 2023 1 3 1 1 45 -0.57 2023 / 1 / 37:59 2023 1 3 1 7 59 -0.24 2023 / 1 / 310:53 2023 1 3 1 10 53 -0.31 2023 / 1 / 319:03 2023 1 3 1 19 3 0.58 2023 / 1 / 42:28 2023 1 4 1 2 28 -0.73 2023 / 1 / 48:51 2023 1 4 1 8 51 -0.25 2023 / 1 / 411:14 2023 1 4 1 11 14 -0.31 2023 / 1 / 419:39 2023 1 4 1 19 39 0.65 2023 / 1 / 53:06 2023 1 5 1 3 6 -0.85 2023 / 1 / 59:28 2023 1 5 1 9 28 -0.25 2023 / 1 / 511:47 2023 1 5 1 11 47 -0.33 2023 / 1 / 520:16 2023 1 5 1 20 16 0.7 2023 / 1 / 63:42 2023 1 6 1 3 42 -0.92 2023 / 1 / 610:02 2023 1 6 1 10 2 -0.25 2023 / 1 / 612:27 2023 1 6 1 12 27 -0.36 2023 / 1 / 620:56 2023 1 6 1 20 56 0.72
[0109] The test set (excerpt) is shown in Table 5.
[0110] Table 5
[0111]
[0112]
[0113] The cyclic training begins. First, the feature value of this round of training is selected as year according to the order of feature values to generate a sub-training set, as shown in Table 6.
[0114] Table 6
[0115] Timestamp Year tide level 2023 / 1 / 20:33 2023 -0.39 2023 / 1 / 26:33 2023 -0.2 2023 / 1 / 210:31 2023 -0.34 2023 / 1 / 218:28 2023 0.48 2023 / 1 / 31:45 2023 -0.57 2023 / 1 / 37:59 2023 -0.24 2023 / 1 / 310:53 2023 -0.31 2023 / 1 / 319:03 2023 0.58 2023 / 1 / 42:28 2023 -0.73 2023 / 1 / 48:51 2023 -0.25 2023 / 1 / 411:14 2023 -0.31 2023 / 1 / 419:39 2023 0.65 2023 / 1 / 53:06 2023 -0.85 2023 / 1 / 59:28 2023 -0.25 2023 / 1 / 511:47 2023 -0.33 2023 / 1 / 520:16 2023 0.7 2023 / 1 / 63:42 2023 -0.92 2023 / 1 / 610:02 2023 -0.25 2023 / 1 / 612:27 2023 -0.36 2023 / 1 / 620:56 2023 0.72
[0116] Import the sub-training set into the newly created xgboost decision tree model for training and generate a prediction model. According to the selected feature values, construct the sub-test set through the test set and feature set, as shown in Table 7.
[0117] Table 7
[0118]
[0119]
[0120] Use the predict function of the xgboost library to calculate the prediction data of the prediction model on the sub-test set and generate the prediction set, as shown in Table 8.
[0121] Table 8
[0122] Timestamp tide level 2024 / 9 / 103:00 -0.21 2024 / 9 / 104:00 -0.25 2024 / 9 / 105:00 -0.25 2024 / 9 / 106:00 -0.2 2024 / 9 / 107:00 -0.05 2024 / 9 / 107:42 0.16 2024 / 9 / 108:00 0.32 2024 / 9 / 109:00 0.38 2024 / 9 / 1010:00 0.39 2024 / 9 / 1011:00 0.41 2024 / 9 / 1012:00 0.42 2024 / 9 / 1013:00 0.39 2024 / 9 / 1014:00 0.28 2024 / 9 / 1015:00 0.11 2024 / 9 / 1015:06 -0.08 2024 / 9 / 1016:00 -0.25
[0123] For both the test set and the prediction set, the root mean square of the tide levels in the two datasets is calculated.
[0124] This cycle ends and enters the next cycle, selecting new feature values to continue training and obtain the next RMS.
[0125] Finally, after all eigenvalues are calculated once, the set of eigenvalues with the lowest root mean square value is selected as the best feature set. In this example, the best feature set is (month, day, hour).
[0126] The prediction data, astronomical tide data, prediction dimension time and the best feature set (month, day, hour) are combined into a fine-tuning data set, and the tabular question-answering model is fine-tuned to generate a new model.
[0127] When encountering astronomical tide data, the new table question-answering model is called, and the query for the best feature set is added when querying the feature set through natural language. For example, when querying astronomical tide data, the timestamp from 2023 to the present, the year, month, day, week, hour, and minute data are output in order as feature sets. If the best feature set exists, it is output first.
[0128] The more predictions and data volumes, the more accurate the optimal feature set. When the predicted optimal feature values are consistent, it can be considered that the optimal feature values have converged in this type of prediction behavior. Subsequent predictions can be directly trained and predicted using the optimal feature set, reducing the time for cyclic training.
[0129] The technical solution of the present invention uses a natural language method to integrate the data in the original table into feature values through a deep learning model, and cyclically calculates the degree of fit of the prediction model to obtain the optimal feature values required for prediction. Through continuous online learning, the optimal feature values and prediction dimensions are assembled into a fine-tuning data set, and the deep learning model is fine-tuned using the fine-tuning data set, and the fine-tuning version is quickly iterated. When a prediction of similar dimensions is made again, the direction and priority of the integrated feature values can be adjusted through the tuned model, and the optimal prediction model and prediction value that conforms to the natural language input rules can be output more quickly.
[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the essence and scope of the technical solution of the present invention.
Claims
1. A table data prediction optimization method based on table question answering, characterized in that: Proceed as follows: Step 1: The table question answering model outputs the predicted dimensions, feature values, and data sets in the table according to the user's input description, and extracts the data sets and feature sets; Step 2: Combine the generated feature set and data set into training set D-Train and test set D-Test according to the prediction dimension; build a model and calculate the fit; Step 3: Assemble the fine-tuning dataset and perform model fine-tuning. Specifically, use the prediction dimension and the optimal feature set as the fine-tuning dataset, and in turn fine-tune the table question answering model to generate a new feature value extraction model M-sn.
2. The table data prediction optimization method based on table question and answer according to claim 1 is characterized in that: It also includes step 4, calling the fine-tuning model, specifically: When the prediction task of the same prediction dimension appears again, the corresponding model M-sn is found in the model library through the prediction dimension; At the same time, the maximum fit degree and the corresponding feature set generated again are assembled into a new fine-tuning data set, and the model is further fine-tuned. The fine-tuned model is updated to the model library as a new feature value extraction model M-sn.
3. The table data prediction optimization method based on table question and answer according to claim 1 or 2, characterized in that: Step 1 specifically includes: Step 1.1, data preprocessing, that is, preprocessing the table data in the input table file through a conversion program, and converting the table data into JSON data; The specific conversion logic is to mark the table in order, generate a mark code, and record it in the table_id item; record the table name in the table_name item; record the column names in the table in the header_name item in column order; take the content of each row in the table as an array, and record multiple rows into a two-dimensional array in the row item; Step 1.2: Use natural language to describe the required feature value name, data item, and prediction dimension, and input the description language and JSON data into the table question answering model at the same time. The table question answering model uses the natural language processing capability Text-to-SQL to create a SQL query statement corresponding to the question, and uses the SQL query statement to query the input data table or database; Step 1.3: The json data result obtained by model query is converted into a feature set T containing feature data and a data set D that needs to be predicted.
4. The table data prediction optimization method based on table question and answer according to claim 3 is characterized in that: The specific conversion logic in step 1.3 is: extract header_name as the column name, then map the array in row to each row of the table to generate a new result table.
5. The table data prediction optimization method based on table question and answer according to claim 4 is characterized in that: Step 2 specifically includes: Step 2.1, sort the data set according to the prediction dimension, and arrange it in the positive order of the prediction direction. Select the front data block as the data set D-sub and the back data block as the test set D-Test, and cut them according to the cutting ratio. Step 2.2, the dataset D-sub is merged with the feature set T column. The merging is based on the prediction dimension item of the dataset D-sub, and the feature values with the same prediction dimension are selected to form the training set D-Train. The training set is used as the original data for model training, and the test set is used as the test data for performance comparison after model training and prediction to evaluate the performance of the model. Step 2.3, select some feature values of the training set D-Train, in addition to retaining the prediction dimension TT and data DD, select one or more columns in the column order, and combine them to generate a sub-training set D-Train-sub; generate a decision tree-based model through the XGBRegressor function of the xgboost library, and then use the fit function to import the sub-training set, train the model, and generate a prediction model M; Step 2.4, based on the feature items selected in the sub-training set D-Train-sub of this cycle, and based on the prediction dimension of the test set D-Test, query the values of the feature items of the same prediction dimension in the feature set T to construct the sub-test set D-Test-Sub; Use the predict function of the xgboost library to calculate the prediction data of the prediction model M on the sub-test set D-Test-Sub to generate the prediction set D-Fore; Step 2.5, calculate the goodness of fit for the test set D-Test and the prediction set D-Fore, calculate the root mean square error RMSE of the predicted values DF and DD in the two data sets, and obtain the goodness of fit De; Step 2.6, cyclically select different eigenvalue combinations, repeat steps 2.3 to 2.5, train, calculate and fit different eigenvalues, and compare to obtain the maximum fitting degree De-max and its corresponding feature set T-max.
6. The table data prediction optimization method based on table question and answer according to claim 5 is characterized in that: Step 3 specifically includes: Step 3.1, assemble the predicted dimension, the maximum fit De-max and the corresponding feature set T-max into the fine-tuning dataset D-stf; In step 3.2, use the fine-tuning dataset D-stf to fine-tune the tabular question answering model M. The fine-tuning behavior is completed using the Trainer trainer of the Transformers framework. Load the tabular question answering model M in the trainer, and load the fine-tuning dataset D-stf in a loop. After fine-tuning, a new model file Ms is generated. Step 3.3, merge the new model file with the prediction dimension record.
7. The table data prediction optimization method based on table question and answer according to claim 6 is characterized in that: The tabular question answering model in step 1 is SPACE-T or Berta.
8. The table data prediction optimization method based on table question and answer according to claim 4 is characterized in that: Data prediction in machine learning uses the LightGBM library.