Training method for table understanding ability of large language model

Abstract

According to the training method for the table understanding ability of the large language model, effective information of the table can be compressed, meanwhile, row and column information capable of improving the effect is reserved, the effect can be improved at the same time, the effect improvement is embodied in the fact that the same table content is expressed in the form, the number of required tokens is generally small, and the method is easy to implement. The method has the advantages that the number of tokens participating in training and reasoning is smaller, computing power is utilized more efficiently, three major capabilities, namely table reading, numerical reasoning and data analysis, of table understanding are considered, the generation probability of correct answers is further increased, and generation of wrong answers caused by the probability is reduced. According to the method, the accuracy of table understanding is improved, and the information density after token conversion is effectively represented, optimized and improved through the table data, so that the efficiency of table data training and reasoning is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method for training table comprehension ability of a large language model. Background Art

[0002] By pre-training on large amounts of text data, large language models have demonstrated the ability to understand natural language and code, which has potential value for understanding and reasoning about tabular data. In the era of big data, structured data (such as tables and databases) is the main storage form for massive amounts of information. If large language models can effectively process this data, the efficiency of obtaining useful information from this data will be greatly improved. Tabular data is widely used in many fields such as finance, medicine, and business. The application of large language models on tabular data can promote data analysis and decision making in these fields.

[0003] The invention [1] proposes a training method for a table question answering model. By obtaining multiple sample table data sets and training based on a low-rank adaptation method, the low-rank matrix is ​​integrated and the pre-trained large language model is fine-tuned, which effectively improves the model's understanding of table information and the accuracy of answering related questions. This method solves the semantic understanding problem of traditional models when processing table data by integrating low-rank matrices for training, enhances the flexibility and adaptability of the model, and is suitable for a variety of fields that require table question answering.

[0004] Invention [2] proposes a training method for improving the accuracy of large language models in processing tabular data. By acquiring tabular question and answer data, generating extended answers, generating new questions based on the table and conducting intelligent question and answer, cross-validating the accuracy of answers, and training with mixed training sets, the reasoning and processing capabilities of the model are significantly improved. Invention [2] mainly focuses on training data. Tabular question and answer data is acquired through an acquisition module; a data generation module generates new questions based on the table, and two different answers are generated by the intelligent model; the generated training data is verified through a cross-validation module, and a large language model is trained through a training module. Experimental results show that this method effectively improves model performance and enhances its generalization and application capabilities in actual office scenarios.

[0005] Invention [3] proposed a vertical financial big model system, which achieves efficient processing of tabular data by combining table encoders and large language models. The system uses cascaded table encoders and Set Transformer models to extract global table representations, and enhances LLM's understanding of financial data through domain-customized fine-tuning. The innovative command chain structuring method and shielded table modeling technology improve the accuracy of data processing and user experience. Private deployment support ensures data security. At the same time, the generalization ability of the model enables it to adapt to different financial scenarios, effectively solving the challenges of the securities industry in tabular data processing, data security and professionalism.

[0006] The invention [4] proposed a method for improving the reasoning and table data processing capabilities of a large language model. The method converts table data into text and introduces thought chains and reflection prompt text to guide the model to perform two in-depth reasonings to improve the accuracy of the answer. This not only solves the limitations of the model in processing tables and complex problems, but also significantly improves its reasoning ability, bringing a breakthrough to the application of artificial intelligence in the field of natural language processing.

[0007] Invention [5] proposed a table data reconstruction method, which generates a reconstructed table by filtering table headers related to user questions, dividing column types and extracting features. The large language model is used to generate accurate SQL statements, which effectively improves the accuracy of table question answering, making the system smarter and closer to actual application scenarios.

[0008] Inventions [3], [4], and [5] optimize the application of large models to improve the accuracy of table question answering. The above solutions do not involve solving the problem of missing two-dimensional information after the table is converted into text information, nor do they have the ability to correct errors through training. In addition, they do not optimize the table representation to improve the performance of training and reasoning.

[0009] [1] Li Ben. Training method, device, equipment, medium and product of table question answering model: 202410322252[P];

[0010] [2] Zhang Jing, Zhang Bohan, Zhang Xiaokang, et al. Training method, device, medium and equipment for large language model of tabular data processing: 202410340959[P];

[0011] [3] Yu Feng, Zhao Junbo, Huang Wei, et al. A vertical field financial big model system and method for realizing efficient processing of tabular data: 202410412492[P];

[0012] [4] Chi Zheyu. A method and device for generating answers: 202410479602[P];

[0013] [5] Chen Gang, Zhao Junbo, Wu Sai, et al. A table data reconstruction method and device, and question-answering method: 202410490193[P]. Summary of the invention

[0014] The technical problems that need to be solved by the technical solution of the present invention are: 1. Tabular data is two-dimensional, which is different from the one-dimensional text and code that the large language model usually processes, which may cause difficulties for the large language model in reading and understanding the table; 2. In addition, the large language model itself has weak digital understanding and processing capabilities, making it more prone to errors, and these errors are very difficult to correct; 3. As the input of the large language model, the table usually needs to be represented by a markup language such as markdown, XML or Latex. These markup languages ​​represent that the table will use corresponding keywords and format information, which will cause the information density to become low, especially when the large model converts text into tokens.

[0015] In order to solve the above technical problems, the technical solution of the present invention provides a method for training table comprehension ability of a large language model, comprising the following steps:

[0016] Define the table start, table end, table header start, table row start, and table column separation, and preset the number of rows and columns in the table row start and table column separation to define special tokens in the table;

[0017] Add special tokens to the word segmenter vocabulary;

[0018] The word segmenter traverses the document to be processed according to the special token in the word segmenter vocabulary to add the special token at the corresponding position;

[0019] Take any table data with special tokens added as initial data, and randomly generate any table reading type in table reading, numerical reasoning, and data analysis;

[0020] In table reading, the table header is generated according to the random number of rows and columns and the corresponding specific cell values ​​are used to generate the table reading description, and the continued pre-training data for table reading is established;

[0021] In numerical reasoning, the table header and the corresponding specific cell values ​​are generated according to any two random sets of row and column numbers, and the cells are compared according to the row and column numbers or row and column contents. Random variables are used to generate table reading descriptions to establish continuous pre-training data for numerical reasoning.

[0022] In data analysis, a data reasoning-related model is used to analyze the content in the initial data based on the table header to generate a table reading description and establish pre-training data for data analysis.

[0023] The table reading description is used as the output of instruction fine-tuning, the random row and column numbers and table headers in the table are read, the table headers and the corresponding specific cell values ​​are generated according to any two random sets of row and column numbers in numerical reasoning, and the table headers are used as the question basis in the form of questions as input to perform instruction fine-tuning training and obtain the instruction fine-tuning model;

[0024] The predicted answers in the question and answer data generated by the instruction fine-tuning model and the template variables for generating the answers are matched with keywords according to the scoring formula to calculate the scores. The predicted answers with the highest scores that are higher than the preset threshold are replaced with the answers in the instruction fine-tuning training data. The failure rate is calculated as the proportion of the number of answers below the preset threshold to the total data. If the failure rate does not decrease continuously, the training is completed.

[0025] Preferably, the number of table reading descriptions generated is limited according to the token length of a single piece of training data, so as to set a threshold value corresponding to the maximum number of table reading descriptions.

[0026] Preferably, the scoring formula is as follows:

[0027]

[0028] Among them, the number of matching template variables is set to match_keys, the number of all model variables is all_keys, a certain predicted answer is p, the generated answer is r, dist represents the edit distance calculation, len represents the text length, and max represents the maximum value.

[0029] The technical solution of the present invention proposes a method for training the table comprehension ability of a large language model. By compressing the effective information of the table while retaining the row and column information that improves the effect, the efficiency can be improved at the same time. The efficiency improvement is reflected in that the same table content is represented in this form, and the number of tokens required is usually smaller, so the number of tokens involved in training and reasoning will also be smaller, which is a more efficient use of computing power. The three major capabilities of table comprehension are taken into account, namely table reading, numerical reasoning, and data analysis, and the probability of generating correct answers is further improved, that is, the generation of incorrect answers due to probability is reduced. The technical solution of the present invention improves the accuracy of table comprehension, and improves the information density after tokenization through effective representation optimization of table data, thereby improving the efficiency of table data training and reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A flow chart of a method for training table comprehension capability of a large language model provided by an embodiment of the present invention;

[0031] Figure 2 A flowchart of building continuous pre-training data for table reading provided in an embodiment of the present invention;

[0032] Figure 3 A flowchart for constructing continued pre-training data for numerical reasoning provided by an embodiment of the present invention;

[0033] Figure 4 A flowchart of constructing continued pre-training data for data analysis provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0035] The concepts involved in the large language model training in the embodiments of the present invention are described as follows:

[0036] 1. The tokenizer can segment the text according to the vocabulary and convert it into tokens. The token will be mapped to the index ID of the corresponding vocabulary and then mapped to the large model vector representation of the corresponding ID.

[0037] 2. The large language model is based on the transformer model structure. The transformer model has: The large language model in the present invention is a causal decoder structure, that is, the decoder and the unidirectional mask attention mechanism in the transformer are used, so that each input token only pays attention to the token in front of it and itself in the sequence, and then predicts the output autoregressively.

[0038] 3. Training of a large language model: a sequence of texts that are segmented and converted into tokens according to the tokenizer T = {t1, t2, ...t N}, the model is optimized according to the following likelihood function:

[0039] like Figure 1 As shown, an embodiment of the present invention proposes a method for training the table comprehension ability of a large language model, which involves two stages: a continuing pre-training stage and an instruction fine-tuning stage. The training method trains the table comprehension ability based on a pre-trained large language model to enhance the table comprehension ability of the large language model in three aspects: table reading, numerical reasoning, and data analysis.

[0040] Among them, the steps to continue the pre-training stage are as follows:

[0041] 1. Generation of tabular pre-training data. In order to represent tabular data, special tokens are defined:

[0042] Table start <|table_start|>

[0043] Table end <|table_end|>

[0044] Table header starts <|table_header|>

[0045] Table row starts <|table_row_{rownum}|>

[0046] Table column separator <|table_column_{columnnum}|>

[0047] {rownum} and {columnnum} are the specific numbers of rows and columns. Considering the actual application, {rownum} is set to 1 to 500, and {columnnum} is set to 1 to 100. That is, the maximum table supports 500 rows * 100 columns, and the table has a total of 603 special tokens.

[0048] 2. Add the above special tokens to the word segmenter vocabulary. When these strings are encountered in the text representation of the subsequent table, they will be segmented into a token and input into the model. This improves the information density of the model input to a certain extent to improve efficiency. At the same time, the row and column information is also fully reflected in the representation to improve the accuracy of table understanding.

[0049] 3. Define the text representation of the table. Add a table start special token "<|table_start|>" at the beginning of the table. If there is a table header, add a table header start special token "<|table_header|>". Add a corresponding table column separation special token "<|table_column_{columnnum}|>" to the beginning of each column of the table header. Add a table row start special token "<|table_row_{rownum}|>" to the beginning of each row of data. Similarly, add a corresponding table column separation special token "<|table_column_{columnnum}|>" to the beginning of each column of data. Repeat this process to process each row of data, and finally add a table end special token "<|table_end|>" at the end of the table. This can be expressed as shown in Table 1.

[0050] Table 1: Sample table

[0051] Name height weight A 160 50 B 170 60 C 180 70

[0052] The following are examples:

[0053] <|table_start|><|table_header|><|table_column_1}|>Name<|table_column_2}|>Height<|table_column_3}|>Weight<|table_row_1}|><|table_column_1}|>A<|table_column_2}|>160<|table_colum n_3}|>50<|table_row_2}|><|table_column_1}|>B<|table_column_2}|>170<|ta ble_column_3}|>60<|table_row_3}|><|table_column_1}|>C<|table_column_2}|>180<|table_column_3}|>70<|table_end|>

[0054] 4. Generate pre-trained data for tables. Take any table data and construct three types of data: table reading, numerical reasoning, and data analysis.

[0055] like Figure 2 As shown, the steps for constructing the pre-training data for table reading are as follows:

[0056] First, the selected table data is given and expressed in the way in 3, and then the corresponding cell readings are given according to the number of rows and columns or the content of the rows and columns according to the template. For example, in the sample table, "the value of the {first} row and the {third} column is {50}", "the {name} is {A}'s {weight} is {50}". The previous sample "{}" is a template variable, corresponding to the table header, number of rows and columns, and the value of the specific cell, which is generated according to the program random variables. These table reading descriptions are added to the end of the table data. The number of table reading descriptions generated is limited by the token length of a single training data, and the threshold limit of the maximum number of descriptions can be set.

[0057] like Figure 3 As shown, the continued pre-training data construction for numerical reasoning

[0058] First, the selected table data is given and represented in the way in 3. The cells are compared according to the number of rows and columns or the content of rows and columns, and the description is given according to the template and the program comparison results. For example, in the sample table, "{name} is {A}'s {height}{160} is {10} less than {name} is {B}'s {height}{170}". The "{}" in the previous sample is a template variable, corresponding to the table header, number of rows and columns, and the value of a specific cell, which is generated according to the program random variables. These table reading descriptions are added to the end of the table data. The number of table reading descriptions generated is limited by the token length of a single training data, and the threshold limit of the maximum number of descriptions can be set.

[0059] like Figure 4 As shown, the data analysis continues with the construction of pre-training data

[0060] First, the selected table data is given and represented in the way in 3, and continuous reasoning descriptions are generated in the form of data reasoning. For example, in the sample table, "{height}{160}'s {weight}{50} is {10} less than {height}{170}'s {weight}{60} is {10} less than {height}{180}'s {weight}{70}". In the previous example "{}" are template variables, corresponding to the table header, number of rows and columns, and the values ​​of specific cells, which are generated according to the program random variables, and these table reading descriptions are added to the end of the table data. Then these statements are added with "What does the analysis of the above content show" generated by question-answering models such as chatgpt and added to the description. Similarly, the token length of a single training data is limited, and the threshold limit of the maximum number of descriptions can be set.

[0061] 5. The generated pre-training data of the continuation table is mixed with part of the original pre-training data to prevent the decline of the original language ability, and the original pre-training model is used for training optimization according to the likelihood function. Get the continued pre-training model.

[0062] The steps of instruction fine-tuning are as follows:

[0063] After completing the pre-training, the model is fine-tuned to enable it to have question-answering capabilities.

[0064] 1. Generate three types of data: table reading, numerical reasoning, and data analysis according to the data generation format mentioned in pre-training. The description and analysis are used as the answers in instruction fine-tuning, and an additional question is generated on this basis.

[0065] Table reading instructions fine-tune data construction

[0066] Replace the data in the cell of the original description with the expression "what" to generate a question. For example, in the sample table, "The value of the {first} row, {third} column is {50}", and the question is replaced with "What is the value of the {first} row, {third} column". The text representation of the table data plus the question is the question part, and the original description is the answer part.

[0067] Instruction fine-tuning data construction for numerical reasoning

[0068] Remove the cell data and comparison results from the original description, and add a comparison statement "How much more?" at the end to generate a question. For example, in the sample table, "How much more is the {height} of {name} {A} than the {height} of {name} {B}?". The text representation of the table data plus the question is the question part, and the original description is the answer part.

[0069] Data analysis instructions to fine-tune data construction

[0070] Extract the table header name and add the expression "analyze" as the question, for example, "analyze {height} and {weight}" in the sample table. The text representation of the table data plus the question is the question part, and the original description and analysis are the answer part.

[0071] 2. The generated instruction fine-tuning data is mixed with some general question-answering instruction fine-tuning data to prevent the original language ability from decreasing, and training optimization is performed according to the likelihood function on the pre-trained model.

[0072]

[0073] Get instructions to fine-tune the model.

[0074] 3. According to the instructions in step 1, fine-tune the number of generation logic to generate different question and answer data, use the instruction fine-tuned model to infer and generate predicted answers, use different generation parameters to generate multiple sets of different predicted answers, and perform keyword matching with the template variables of the generated answers to calculate the scores. If there is analysis, add the edit distance evaluation. Set the number of matching template variables to match_keys, the number of all model variables to all_keys, a certain predicted answer to p, the generated answer to r, dist to represent the edit distance calculation, len to represent the text length, max to represent the maximum value, and the scoring formula is:

[0075]

[0076] The predicted answers with the highest scores above a certain threshold are selected and replaced with the answers in the instruction fine-tuning training data. In addition, the percentage of the number of answers below this set threshold to the total data is the failure rate.

[0077] 4. Add the data generated in 3 to the instruction fine-tuning data used in 2, and then increase the general question-answering instruction fine-tuning data in equal proportion, and continue training and tuning on the pre-trained model according to the method in 2.

[0078] 5. Repeat 3 and 4 until the failure rate mentioned in 3 continues to increase, and the training is completed.

[0079] The key points and protection points of the embodiments of the present invention are as follows:

[0080] 1. Textual representation of tabular data.

[0081] 2. Continue to pre-train the method for generating tabular data.

[0082] 3. Instructions to fine-tune the method of generating table data.

[0083] 4. A method of iteratively generating new instructions and fine-tuning data for training.

[0084] 5. The text representation of tabular data can compress effective information and improve the accuracy of table comprehension.

[0085] 6. The methods for generating training data and training them are both aimed at improving the large language model’s ability to understand tables.

[0086] 7. Choosing to generate the most accurate answer during iterative training can further improve the probability of generating the correct answer when you have the ability to understand the table. At the same time, since the answer is generated by yourself, it is not easy to affect other general abilities.

[0087] The advantages of the embodiments of the present invention are as follows:

[0088] 1. It can compress the effective information of the table while retaining the row and column information that improves the effect. The efficiency is improved in that the same table content is represented in this way, and the number of tokens required is usually smaller, so the number of tokens involved in training and reasoning will also be smaller, which is a more efficient use of computing power.

[0089] 2. The way of generating training data, the system takes into account the three major capabilities of table understanding, namely table reading, numerical reasoning, and data analysis. The table capabilities of large language models are improved by the system's targetedness.

[0090] 3. Iterative training further improves the probability of generating correct answers, that is, reduces the generation of incorrect answers due to probability.

Claims

1. A method for training table comprehension ability of a large language model, characterized in that: The following steps are involved: Define the table start, table end, table header start, table row start, and table column separation, and preset the number of rows and columns in the table row start and table column separation to define special tokens in the table; Add special tokens to the word segmenter vocabulary; The word segmenter traverses the document to be processed according to the special token in the word segmenter vocabulary to add the special token at the corresponding position; Take any table data with special tokens added as initial data, and randomly generate any table reading type in table reading, numerical reasoning, and data analysis; In table reading, the table header is generated according to the random number of rows and columns and the corresponding specific cell values ​​are used to generate the table reading description, and the continued pre-training data for table reading is established; In numerical reasoning, the table header and the corresponding specific cell values ​​are generated according to any two random sets of row and column numbers, and the cells are compared according to the row and column numbers or row and column contents. Random variables are used to generate table reading descriptions to establish continuous pre-training data for numerical reasoning. In data analysis, a data reasoning-related model is used to analyze the content in the initial data based on the table header to generate a table reading description and establish pre-training data for data analysis. The table reading description is used as the output of instruction fine-tuning, the random row and column numbers and table headers in the table are read, the table headers and the corresponding specific cell values ​​are generated according to any two random sets of row and column numbers in numerical reasoning, and the table headers are used as the question basis in the form of questions as input to perform instruction fine-tuning training and obtain the instruction fine-tuning model; The predicted answers in the question and answer data generated by the instruction fine-tuning model and the template variables for generating the answers are matched with keywords according to the scoring formula to calculate the scores. The predicted answers with the highest scores that are higher than the preset threshold are replaced with the answers in the instruction fine-tuning training data. The failure rate is calculated as the proportion of the number of answers below the preset threshold to the total data. If the failure rate does not decrease continuously, the training is completed.

2. A method for training table comprehension ability of a large language model as claimed in claim 1, characterized in that: The number of table reading descriptions generated is limited according to the token length of a single piece of training data to set a threshold value corresponding to the maximum number of table reading descriptions.

3. A method for training table comprehension ability of a large language model as claimed in claim 1, characterized in that: The scoring formula is as follows: Among them, the number of matching template variables is set to match_keys, the number of all model variables is all_keys, a certain predicted answer is p, the generated answer is r, dist represents the edit distance calculation, len represents the text length, and max represents the maximum value.

Images