Table-text data generation method and device based on large language model
Through the table-text data generation method based on the large language model, combined with tables and descriptive text, the problem of inefficient understanding and generation of table data is solved, and the efficient automation understanding and generation ability of tables by the large language model is realized.
Patent Information
- Application Number
- CN202411040149.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-31
AI Technical Summary
The existing technology is difficult to effectively understand and process table data, which leads to inefficiency in artificial intelligence when utilizing tables. Due to the lack of data sets and poor quality of training samples, it is difficult to generate descriptive language with coherent statements.
Using the table-text data generation method based on the large language model, by constructing a training data set, combining tables and descriptive texts, first conducting basic table understanding training on the large language model, and then performing table processing and fine-tuning of descriptive text generation capabilities , improve the model's understanding and generation ability of tables.
It improves the automation and efficient understanding of tables by large language models, so that tables can express processing results in descriptive text, and improves the table processing and generation effect of the model.
Smart Images

Figure CN118569214B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of natural language processing and the field of large models. The present disclosure specifically relates to a method and device for generating table-text data based on a large language model. Background Art
[0002] As a widely used data carrier, tables are difficult for AI to understand due to their two-dimensional text characteristics during actual use and processing, and therefore their utilization efficiency in the field of artificial intelligence is low. At the same time, data in certain fields are usually highly sensitive and confidential or cover a small scope, making it difficult to obtain open data sets of sufficient scale, limiting the breadth and depth of research. Problems such as lack of data sets and poor quality of training samples always exist and plague researchers.
[0003] Although great progress has been made in table-text data generation methods, traditional naive models cannot accurately extract key information, have insufficient information extraction capabilities, and have difficulty generating coherent descriptive language. Summary of the invention
[0004] The present disclosure provides a method and device for generating table-text data based on a large language model.
[0005] According to one aspect of the present disclosure, a method for generating table-text data based on a large language model is provided, comprising:
[0006] constructing a first training data set based on each first table processing instruction, each first table, and each first table processing result obtained by processing each first table using each first table processing instruction, wherein the first table processing result is any one of a table and a descriptive text;
[0007] Constructing a second training data set based on each second table processing instruction, each second table, and each second table processing result obtained by processing each second table using each second table processing instruction, wherein the second table processing result is a descriptive text;
[0008] Using the first training data set to train the first language model to obtain a second language model;
[0009] Using the second training data set to train the second language model to obtain a tabular text data generation model;
[0010] The target table processing instruction and the target table are input into the table text data generation model to obtain the target table processing result output by the table text data generation model, wherein the target table processing result is a descriptive text.
[0011] According to another aspect of the present disclosure, a table-text data generation device based on a large language model is provided, comprising:
[0012] A first training set determination module, configured to construct a first training data set based on each first table processing instruction, each first table, and each first table processing result obtained by processing each first table using each first table processing instruction, wherein the first table processing result is any one of a table and a descriptive text;
[0013] A second training set determination module, configured to construct a second training data set based on each second table processing instruction, each second table, and each second table processing result obtained by processing each second table using each second table processing instruction, wherein the second table processing result is a descriptive text;
[0014] A first training module, configured to train the first language model using the first training data set to obtain a second language model;
[0015] A second training module, used to train the second language model using the second training data set to obtain a table text data generation model;
[0016] The model application module is used to input the target table processing instruction and the target table into the table text data generation model to obtain the target table processing result output by the table text data generation model, wherein the target table processing result is a descriptive text.
[0017] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0018] at least one central processing unit; and
[0019] A memory in communication with the at least one central processing unit; wherein,
[0020] The memory stores instructions that can be executed by the at least one central processing unit, and the instructions are executed by the at least one central processing unit so that the at least one central processing unit can execute any table-text data generation method based on a large language model in the embodiments of the present disclosure.
[0021] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any table-text data generation method based on a large language model in the embodiments of the present disclosure.
[0022] According to the technology disclosed in the present invention, a first training data set is constructed based on each first table processing instruction, each first table, and each first table processing result obtained by processing each first table using each first table processing instruction, wherein the first table processing result is any one of a table and a descriptive text. A second training data set is constructed based on each second table processing instruction, each second table, and each second table processing result obtained by processing each second table using each second table processing instruction, wherein the second table processing result is a descriptive text. Then, the first training data set is used to train the first large language model to obtain the second large language model; and the second training data set is used to train the second large language model to obtain a table text data generation model. Thus, the target table processing instruction and the target table are input into the table text data generation model to obtain the target table processing result expressed as a descriptive text output by the table text data generation model. Therefore, in the disclosed technology, the model can be first trained for the basic table comprehension ability of the large language model, and then the model can be trained again for the table processing and descriptive text generation capabilities of the large language model. In this way, the basic understanding ability of the large language model for the table can be improved and the processing results of the table can be further generated in the form of descriptive text, so that the table comprehension ability of the large model can achieve an automated and efficient effect.
[0023] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0025] Figure 1 is a flowchart of a method for generating table-text data based on a large language model according to an embodiment of the present disclosure;
[0026] Figure 2 is a schematic diagram of a training sample according to an embodiment of the present disclosure;
[0027] Figure 3 is a schematic diagram of a training sample according to another embodiment of the present disclosure;
[0028] Figure 4 is a schematic diagram of a pseudo code of a model training process according to an embodiment of the present disclosure;
[0029] Figure 5 is a convergence curve diagram of loss values during the model training process of an embodiment of the present disclosure;
[0030] Figure 6is a schematic diagram of a pseudo code of a model merging process according to an embodiment of the present disclosure;
[0031] Figure 7 is a convergence curve diagram of loss values during model training in another embodiment of the present disclosure;
[0032] Figure 8 is a practical example of input data and output data of a model according to an embodiment of the present disclosure;
[0033] Fig. 9 is a practical example of input data and output data of a model of another embodiment of the present disclosure;
[0034] Fig.10 It is a structural block diagram of a table-text data generating device based on a large language model according to an embodiment of the present disclosure;
[0035] Fig.11 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0036] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0037] In order to facilitate understanding of the table-text data generation method based on a large language model provided by the embodiment of the present disclosure, the relevant technologies of the embodiment of the present disclosure are explained below. The following related technologies can be arbitrarily combined with the technical solutions of the embodiment of the present disclosure as optional solutions, and they all belong to the protection scope of the embodiment of the present disclosure.
[0038] Figure 1 It is a flowchart of a table-text data generation method based on a large language model of an embodiment of the present disclosure. The method can be applied to an electronic device. The electronic device is, for example, a terminal, a server or other processing device, wherein the terminal can be a user equipment (UE) such as a desktop computer, a mobile device, a PDA (Personal Digital Assistant), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some implementations, the electronic device can implement the table-text data generation method based on a large language model of an embodiment of the present disclosure by a central processing unit calling computer-readable instructions stored in a memory.
[0039] like Figure 1As shown, the method includes:
[0040] S110, constructing a first training data set based on each first table processing instruction, each first table, and each first table processing result obtained by processing each first table using each first table processing instruction, wherein the first table processing result is any one of a table and a descriptive text;
[0041] S120, constructing a second training data set based on each second table processing instruction, each second table, and each second table processing result obtained by processing each second table using each second table processing instruction, wherein the second table processing result is a descriptive text;
[0042] S130, training the first language model using the first training data set to obtain a second language model;
[0043] S140, training the second largest language model using the second training data set to obtain a table text data generation model;
[0044] S150, inputting the target table processing instruction and the target table into the table text data generation model to obtain the target table processing result output by the table text data generation model, wherein the target table processing result is a descriptive text.
[0045] It can be understood that the above steps S110 and S120 can be executed in parallel or in sequence.
[0046] Exemplarily, in the above step S110, the table processing instruction may also be considered as a table task.
[0047] Exemplarily, a zero-shot learning method is adopted, as shown in Table 1, to establish 13 table tasks, including corresponding cell row IDs or column IDs or exchanging specified rows, etc., to help the large model understand the table relationship from different angles of the relationship between rows and columns.
[0048]
[0049] Exemplarily, the original data set is cleaned and an instruction data set is constructed according to the table task. For the large language model of the table task, the goal is to use the (instruction, table, completion) triple to improve the performance of the large model. The example (t) of the table task is defined as t=(Ins,T,C). Among them, Ins is the natural language instruction that specifies the table task, T is the input table for the table task to be executed, and C is the expected output, that is, the table processing result. The whole process can be divided into two stages, synthesis and enhancement. In the synthesis stage, a data set is constructed, and in the enhancement stage, data enhancement is performed from the table processing instruction level, the input table level, and the table processing result level using rich instructions, diverse tables, and the "reasoning first and then answering" method.
[0050] Exemplarily, any training sample in the first training data set includes a first table processing instruction, a first table (i.e., the input data or output data of the model) and a first table processing result (i.e., the reference output data or reference output result of the model). The instruction can be set according to the above-mentioned table task. Then, according to different task requirements, 5-10 tables are randomly selected from the original input table data set as input data, i.e., the above-mentioned first table. For each task requirement and its corresponding input data, the corresponding output data is determined, i.e., the above-mentioned first table processing result as a reference. Then, the data set was created and annotated using Python.
[0051] For example, in order to facilitate training, any training sample in the data set can be converted into a two-segment format. For example, the following data format is used:
[0052] {"conversations": [{"role": "user", "content": "Instruction:… \nInput:…};
[0053] {"role": "assistant", "content": completion}]}.
[0054] For example, Figure 2 As shown, for the above data format, Figure 2 An example of a training sample is given. The text between "Instruction:" and "\n" is the first table processing instruction of the training sample. The text between "Input:" and "\n" is the input table of the training sample, that is, the first table mentioned above. The text after "content:" is the first table processing result of the training sample, that is, the reference table processing result.
[0055] Exemplarily, the instruction description or task description corresponding to the above 13 table tasks may be as shown in Table 2 below.
[0056] Task Instruction T1 "Task Description: Please sort the table by the [specified] column so that the values in the column are arranged in ascending order. Please return the sorted table." T2 "Task description: Please fill in the missing values in the table. Missing values are represented by '[MISSING]'. Please only return the filled-in values, not the entire table. Return the final result in JSON format {\"value\": \"<missing value in the table>\"}." T3 "Instruction: Please check the input table and let me know which cells have errors. If there are multiple error cells, return them all in a list. If there are no error cells, return "None". Please return only cells with errors or cells with high confidence. Return the final result in JSON format {"Error Cells": "<Error Cell List or None>"}. Let's think about it step by step and show your reasoning before showing the final result. Below is the table data. T4 "Instruction: Given input table data and a list of candidate headers, determine the most appropriate column headers for each column in the table. Select only the column headers from the candidate list. Return only the most appropriate column headers for each column. Return the selected column headers from the list. Do not return the entire table. Return the final result as JSON in the format {"column headers": "<list of headers for each column selected from the candidate list>"}. T5 "instruction": "Task description: Please convert the following table with multiple columns into plain text and return the table. Use vertical lines to separate columns and horizontal lines to separate new rows. Return the final result in JSON format {\"table\": \"\" T6 "Instruction: Task description: Please check the table below. There is a missing cell in the table. When you find the missing cell, please point it out using its column name. The final result is returned in JSON format {\"missing_col\": \"<missing column name>\"}. T7 "Instruction: Task description: Please check the table below. There is a missing cell in the table. When you find the missing cell, please point it out using its column name. The final result is returned in JSON format {\"missing_col\": \"<missing column name>\"}. T8 "Instruction: Task Description: Please check the table below. There is a missing cell in the table. When you find this missing cell, please point it out using the row id shown in the first column. The final result is returned in JSON format {\"row_id\": \"<row_id of the row with the missing cell>\"}. T9 "Instruction: Task Description: Please check the table below. There is a missing cell in the table. When you find this missing cell, please point it out using the row id shown in the first column. The final result is returned in JSON format {\"row_id\": \"<row_id of the row with the missing cell>\"}. T10 "Task Description: Generate new rows for the input table and append them to the bottom of the table. Return the final result in JSON format." T11 "Task description: The nth row in the table has been selected. Please return the table of the selected row. Return the final result in JSON format." T12 "Instruction: Please swap rows 2 and 3 in the table and return the swapped table. T13 "Task description: Given a two-column table of inputs and outputs, find the pattern between the inputs and outputs from the first few rows, determine the output value of the last row, and return only the output value. Return the final result in JSON format {\"output_value\": \"<output value>\"}."
[0057] The following describes the process of generating the second training data set in step S120:
[0058] Exemplarily, any training sample in the second training data set includes a second table processing instruction, a second table (ie, input data or input table of the model) and a second table processing result (ie, reference output data or reference output result of the model).
[0059] Exemplarily, the plurality of second table processing instructions may partially overlap with the plurality of second table processing instructions mentioned above. For example, the second table processing instruction may be:
[0060] "instruction":"Task description:Please refer to the table below and generate descriptive text. The final result is returned in JSON format {\"result\":\"<>\"}.".
[0061] Exemplarily, the format of the second table processing instruction may be the same as the format of the first table processing instruction.
[0062] Exemplarily, the second table may come from tables in various documents, such as tables in HowNet papers.
[0063] Exemplarily, the second table processing result may be a manually annotated table processing result obtained by processing the second table based on the second table processing instruction in a manually annotated manner. The table processing result is a descriptive text.
[0064] Exemplarily, a manual annotation method is used to search for descriptive text in the paper after the original author processes the second table according to the second table processing instruction as the second table processing result. The descriptive text is the reference text.
[0065] Exemplarily, since there are currently few table-text data sets, the present invention adopts pure manual annotation plus reconstruction of the input and output in the training samples based on existing structured data. Similarly, to facilitate training, the training samples are uniformly converted into two-segment forms, using the consistent format as above. The second table (Table) is a given table, in which cells are separated by '|' and line breaks are represented by '\n'. The second table feasible processing result (Completion) is manually annotated, and the descriptive text of the table made by the original author in the paper is manually annotated as a reference text. Figure 3 As shown, an example of training samples in a second training data set is shown.
[0066] Exemplarily, the first language model may be an untrained basic model or a trained model.
[0067] Exemplarily, multiple factors need to be considered, including the performance of the model, the complexity of the model, the interpretability and scalability of the model, to determine the base model of the above-mentioned large language model. Taking all factors into consideration, the ChatGLM3-6B model is more suitable. In terms of the deployment of the base model, the present invention chooses local installation to ensure that the selected environment can support the stable operation of the model and provide sufficient computing resources and memory. Install pytorch corresponding to the Python version and graphics card version. By creating a virtual environment, install the dependency packages required for the large model to run. Use the command line conda create to create a virtual environment, activate it using the command activate conda, and finally install the dependency packages in batches.
[0068] For example, from the perspective of the hardware environment, model training and testing require powerful computing power to process large-scale data, so it will be performed on a high-performance computing cluster. Specifically, the device processor used in the embodiment of the present disclosure is Intel64 Family 6 Model 183 Stepping 1 GenuineIntel ~3000 Mhz, the total physical memory is 65277 MB, the maximum value of virtual memory is 116540 MB, and the graphics card is NVIDIA GeForce RTX 4090.
[0069] In terms of software environment, Python, as a powerful and easy-to-learn programming language, and with its own deep learning library, has become the preferred programming language for this article. The disclosed embodiment builds and trains the target model by using Python's existing basic machine learning framework. The Python software version is 3.11, and the CUDA corresponding version pytorch2.0.0 is installed. The maximum GPU power is 170w, the maximum video memory is 24564MiB, the driver version is 552.22, and the CUDA version is 12.4.
[0070] Among them, the base model can select ChatGLM as the basic model.
[0071] Exemplarily, the above step S130 may be: a LoRA (low rank adaptive fine-tuning method) fine-tuning for table understanding.
[0072] For example, the main idea of LoRA fine-tuning adopts low-rank decomposition, namely:
[0073]
[0074] in , , then the rank , during fine-tuning, Frozen unchanged, A and B contain training parameters, only A and B are trained.
[0075] In reasoning, the new and old parameters are directly merged by using the re-parameter method, that is, The forward pass of is rewritten as:
[0076]
[0077] A is initialized by a random Gaussian, B is 0, and ΔW is 0. During the training process, it is scaled by the ratio of α / r, where α is a constant in r. Figure 4 As shown, this is the code for the above LoRA fine-tuning.
[0078] For example, the model training parameters are set by referring to the official training parameters of ChatGLM3-6B for the instructive dataset Alpaca (52k). After repeated experiments, the training parameters for the best convergence of loss values are determined. The specific parameters are as follows:
[0079] LoRA corresponding training parameters:
[0080] Table 3: LoRA training parameters
[0081] symbol explain value r rank 8 lora_alpha Scaling factor 16 lora_dropout Dropout ratio 0.1
[0082] The model is fine-tuned using the official default parameter r=8, and lora_alpha is set to 16, which is twice r. Setting alpha to twice r is a common rule of thumb, and the disclosed embodiment tests alpha=32, 64, 128, etc. In fact, alpha=16 works best.
[0083] For example, the data set input and output text parameters are set as follows:
[0084] Table 4: Input and output parameters
[0085] symbol explain value max_input_length Maximum input length 4096 max_output_length Maximum output length 2048 max_new_tokens Maximum generated text 1024
[0086] For example, the 13 tasks of table comprehension have a small overall volume and a strong regularity in the data set, so the parameters are set as follows considering the training efficiency and accuracy:
[0087] Table 5: Core training parameters
[0088] symbol explain value num_proc The number of processes used for data loading 12 epoch Total training rounds 10 learning_rate Learning Rate 5e-4 per_device_train_batch_size The training batch size per device 4 per_device_eval_batch_size Evaluation batch size per device 16 task_type Task type: Causal language model CAUSAL_LM loss Deep learning effect /
[0089] For example, the total training time of the 2600 data set is 5 hours. The data smoothing operation of the exponential moving average is plotted on the data. The final smooth loss curve with circle identifiers is the result. The loss value loss converges as follows Figure 5 The vertical axis is the loss value, and the horizontal axis is the batch. Each time a batch of data is trained, the corresponding loss is recorded. It can be found that the loss converges quickly, and the final loss value converges to 0.0097. The average memory usage is 23502 MiB, and the memory utilization rate is 95.68%.
[0090] Exemplarily, the above step S130 may be: secondary fine-tuning of LoRA for table-text data generation.
[0091] For example, on the large model that has completed the table understanding fine-tuning task, the model is fine-tuned to generate descriptive text for the table. The table-text data generation training data set has a total of 1,800 items, which contains a wide range of content and little data regularity. It requires multiple iterations of repeated learning. Considering the training efficiency and accuracy, the parameters are set as follows:
[0092] Table 6: Core training parameters
[0093] symbol explain value num_proc The number of processes used for data loading 12 epoch Total training rounds 20 learning_rate Learning Rate 5e-5 per_device_train_batch_size The training batch size per device 2 per_device_eval_batch_size Evaluation batch size per device 8 task_type Task type: Causal language model CAUSAL_LM loss Deep learning effect /
[0094] This example uses a single machine with multiple GPUs for fine-tuning and uses DeepSpeed for acceleration to improve the training efficiency of the ChatGLM3-6B model. The total training time is 11.5 hours, with an average memory usage of 23883 MiB and a memory utilization rate of 97.23%. The data is smoothed by exponential moving average and the loss value converges. Figure 7 The loss continues to decrease and converge, approaching 0, but the oscillation is obvious. Its final convergence value is between 0.1 and 0.2, and the training result is good.
[0095] Exemplarily, in the above step S150, the target table processing instruction and the target table may be test data or data for actual application.
[0096] According to the technology disclosed in the present invention, a first training data set is constructed based on each first table processing instruction, each first table, and each first table processing result obtained by processing each first table using each first table processing instruction, wherein the first table processing result is any one of a table and a descriptive text. A second training data set is constructed based on each second table processing instruction, each second table, and each second table processing result obtained by processing each second table using each second table processing instruction, wherein the second table processing result is a descriptive text. Then, the first training data set is used to train the first large language model to obtain the second large language model; and the second training data set is used to train the second large language model to obtain a table text data generation model. Thus, the target table processing instruction and the target table are input into the table text data generation model to obtain the target table processing result expressed as a descriptive text output by the table text data generation model. Therefore, in the disclosed technology, the model can be first trained for the basic table comprehension ability of the large language model, and then the model can be trained again for the table processing and descriptive text generation capabilities of the large language model. In this way, the basic understanding ability of the large language model for the table can be improved and the processing results of the table can be further generated in the form of descriptive text, so that the table comprehension ability of the large model can achieve an automated and efficient effect.
[0097] In one embodiment, in the above step S130, the first training data set is used to train the first largest language model to obtain the second largest language model, including: based on a low-rank adaptive fine-tuning method, the model parameters of the first largest language model are fine-tuned using the first training data set to obtain the second largest language model.
[0098] For example, the main idea of LoRA fine-tuning adopts low-rank decomposition, namely:
[0099]
[0100] in , , then the rank , during fine-tuning, Frozen unchanged, A and B contain training parameters, only A and B are trained.
[0101] In reasoning, the new and old parameters are directly merged by using the re-parameter method, that is, The forward pass of is rewritten as:
[0102]
[0103] A is initialized by a random Gaussian, B is 0, and ΔW is 0. During the training process, it is scaled by the ratio of α / r, where α is a constant in r. Figure 4 As shown, this is the code for the above LoRA fine-tuning.
[0104] According to the above implementation, based on the low-rank adaptive fine-tuning method, the model parameters of the first largest language model are fine-tuned using the first training data set to obtain the second largest language model, which can improve the speed of model training while maintaining the model accuracy.
[0105] Of course, in some embodiments, other training methods may also be used to fine-tune the model parameters of the first large language model using the first training data set to obtain the above-mentioned second large language model.
[0106] In one embodiment, in the above step S140, the second largest language model is trained with the second training data set to obtain a tabular text data generation model, including: merging the first largest language model and the second largest language model to obtain a third largest language model; and training the third largest language model with the second training data set to obtain a tabular text data generation model.
[0107] It can be understood that after the first fine-tuning of the model, the original model is combined with the fine-tuned model to obtain the third language model mentioned above, thereby preventing the model from being over-trained.
[0108] For example, the process of model merging can be as follows: first load the base model, then call out the fine-tuned LoRA model, including the low-rank matrix and the fine-tuned embedding layer parameters. Merge the embedding layer parameters and the low-rank matrix into the corresponding base model for subsequent model reasoning and parameter decomposition. Figure 6 , which shows the pseudo code for model merging.
[0109] Exemplarily, the merged model is subjected to secondary fine-tuning of the above-mentioned LoRA table-oriented text generation.
[0110] According to the above implementation, after the model is fine-tuned for the first time, the original model is combined with the fine-tuned model to obtain the above third language model, thereby preventing the model from being over-trained.
[0111] In one embodiment, the above-mentioned use of the second training data set to train the third largest language model to obtain the tabular text data generation model includes: based on a low-rank adaptive fine-tuning method, using the second training data set to fine-tune the model parameters of the third largest language model to obtain the tabular text data generation model.
[0112] For example, the main idea of LoRA fine-tuning adopts low-rank decomposition, namely:
[0113]
[0114] in , , then the rank , during fine-tuning, Frozen unchanged, A and B contain training parameters, only A and B are trained.
[0115] In reasoning, the new and old parameters are directly merged by using the re-parameter method, that is, The forward pass of is rewritten as:
[0116]
[0117] A is initialized by a random Gaussian, B is 0, and ΔW is 0. During the training process, it is scaled by the ratio of α / r, where α is a constant in r. Figure 4 As shown, this is the code for the above LoRA fine-tuning.
[0118] According to the above implementation, based on the low-rank adaptive fine-tuning method, the model parameters of the third language model are secondarily fine-tuned using the second training data set to obtain a tabular text data generation model, which can improve the speed of model training while maintaining the model accuracy.
[0119] Of course, in some embodiments, other training methods may also be used to fine-tune the model parameters of the third language model using the second training data set to obtain the above-mentioned tabular text data generation model.
[0120] In some implementations, the type of the first form processing instruction includes the type of the second form processing instruction.
[0121] It can be understood that the instruction set in the second training data set is a subset of the instruction set in the first training data set.
[0122] According to the above implementation, the model is trained to understand the table using various forms of first table processing instructions and various forms of first table processing results, which can improve the model's ability to understand the table. After improving the model's ability to understand the table, the model is trained to generate table-text using the second table processing result in the form of descriptive text and some of the same instructions as the aforementioned understanding training. In this way, the table can be processed using the trained model to accurately obtain the table processing result output in the form of descriptive text.
[0123] In one embodiment, the above method may further include:
[0124] Determine a score of the table text data generation model based on the target table processing result and the reference table processing result corresponding to the target table processing instruction executed on the target table;
[0125] When the score of the tabular text data generation model does not meet the preset score target, the first largest language model is updated based on the tabular text data generation model, and based on the updated first largest language model, the above training steps are returned to be executed until the score of the updated tabular text data generation model meets the score target and the above training steps are stopped.
[0126] For example, Figure 8 and Fig. 9 The final training results of this method are shown. The two types of generated texts are evaluated by BLEU indicators. The evaluation results are shown in Table 7, where Model 1 is a large model that has been fine-tuned for the table comprehension task, Model 2 is a model that has not been fine-tuned for the table comprehension task, and Model 3 is the original model that has not been fine-tuned:
[0127] Table 7: BLEU index of the corresponding model generation results
[0128] serial number Model 1 BLEU Score Model 2 BLEU Score Model 3 BLEU Score 1 0.428182 0.487774 0 2 0.701688 0.701688 0.528993 3 0.442897 0.119756 0.054003 4 0.531697 0.492027 0 5 0.537117 0.275556 0.167321 6 0.634047 0.634047 0 7 0.457883 0.701688 0 8 0.634047 0.634047 0 9 0.634047 0.634047 0 … … … … 200 0.531697 0.583351 0 average 0.65982 0.502983 0.117538
[0129] Model 3: The large model without any fine-tuning often has difficulty in giving satisfactory answers to specified questions. In fact, in most cases, the model fails to understand the instruction itself and simply returns an empty answer. The large model fine-tuned with table comprehension ability is closer to the original reference text and has been significantly improved. Therefore, based on the above experiments, this paper believes that the table-text data generation method based on table comprehension ability is effective.
[0130] Fig.10 It is a structural block diagram of a table-text data generation device based on a large language model according to an embodiment of the present disclosure.
[0131] like Fig.10 As shown, the table-text data generating device based on the large language model includes:
[0132] A first training set determination module 1110, configured to construct a first training data set based on each first table processing instruction, each first table, and each first table processing result obtained by processing each first table using each first table processing instruction, wherein the first table processing result is any one of a table and a descriptive text;
[0133] A second training set determination module 1120, configured to construct a second training data set based on each second table processing instruction, each second table, and each second table processing result obtained by processing each second table using each second table processing instruction, wherein the second table processing result is a descriptive text;
[0134] A first training module 1130, configured to train the first large language model using the first training data set to obtain a second large language model;
[0135] A second training module 1140 is used to train the second language model using the second training data set to obtain a table text data generation model;
[0136] The model application module 1150 is used to input the target table processing instruction and the target table into the table text data generation model to obtain the target table processing result output by the table text data generation model, wherein the target table processing result is a descriptive text.
[0137] In one implementation, the first training module 1130 is specifically configured to:
[0138] Based on a low-rank adaptive fine-tuning method, the first training data set is used to fine-tune the model parameters of the first largest language model to obtain the second largest language model.
[0139] In one embodiment, the second training module 1140 includes:
[0140] a model merging unit, configured to merge the first largest language model and the second largest language model to obtain a third largest language model;
[0141] A model training unit is used to train the third language model using the second training data set to obtain the tabular text data generation model.
[0142] In one implementation, the model training unit is specifically used to:
[0143] Based on a low-rank adaptive fine-tuning method, the model parameters of the third language model are fine-tuned using the second training data set to obtain the tabular text data generation model.
[0144] In one implementation, the type of the first form processing instruction includes the type of the second form processing instruction.
[0145] In one embodiment, the above device further comprises:
[0146] A model scoring module, used to determine the score of the table text data generation model based on the target table processing result and the reference table processing result corresponding to the target table processing instruction executed on the target table;
[0147] The model loop training module is used to update the first large language model based on the tabular text data generation model when the score of the tabular text data generation model does not meet the preset score target, and return to execute the above training steps based on the updated first large language model until the score of the updated tabular text data generation model meets the score target and stop executing the above training steps.
[0148] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.
[0149] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0150] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product. The electronic device may be a server, and the server may also include any power supply control device in the embodiment of the present disclosure.
[0151] Fig.11 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0152] like Fig.11 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 607 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0153] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 606, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0154] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a task processing method. For example, in some embodiments, a task processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 607. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps in the task processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform a task processing method in any other appropriate manner (e.g., by means of firmware).
[0155] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0156] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0157] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0158] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0159] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0160] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0162] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for generating table-text data based on a large language model, characterized in that: include: Based on each first table processing instruction, each first table, and each first table processing result obtained by processing each first table using each first table processing instruction, construct each training sample to obtain a first training data set, wherein the first training data set includes at least one training sample whose first table processing result is a table and at least one training sample whose first table processing result is a descriptive text; Based on each second table processing instruction, each second table, and each second table processing result obtained by processing each second table using each second table processing instruction, construct each training sample to obtain a second training data set, wherein the second training data set only includes training samples whose second table processing result is a descriptive text; Using the first training data set to train the first language model to obtain a second language model; Using the second training data set to train the second language model to obtain a tabular text data generation model; The target table processing instruction and the target table are input into the table text data generation model to obtain the target table processing result output by the table text data generation model, wherein the target table processing result is a descriptive text.
2. The method according to claim 1, characterized in that: The step of training the first language model using the first training data set to obtain the second language model includes: Based on a low-rank adaptive fine-tuning method, the first training data set is used to fine-tune the model parameters of the first largest language model to obtain the second largest language model.
3. The method according to claim 1, characterized in that The step of training the second language model using the second training data set to obtain a table text data generation model includes: Merging the first largest language model and the second largest language model to obtain a third largest language model; The third language model is trained using the second training data set to obtain the tabular text data generation model.
4. The method according to claim 3, characterized in that The step of using the second training data set to train the third language model to obtain the table text data generation model includes: Based on a low-rank adaptive fine-tuning method, the model parameters of the third language model are fine-tuned using the second training data set to obtain the tabular text data generation model.
5. The method according to claim 1, characterized in that The type of the first form processing instruction includes the type of the second form processing instruction.
6. The method according to claim 1, characterized in that Also includes: Determining a score of the table text data generation model based on the target table processing result and a reference table processing result corresponding to the target table processing instruction executed on the target table; In the case where the score of the tabular text data generation model does not meet the preset score target, updating the first large language model based on the tabular text data generation model, and returning to execute the training step based on the updated first large language model until the score of the updated tabular text data generation model meets the score target and stopping executing the training step; The training step includes: using the first training data set to train the first language model to obtain the second language model, and using the second training data set to train the second language model to obtain a table text data generation model.
7. A table-text data generation device based on a large language model, characterized in that: include: A first training set determination module, configured to construct respective training samples based on respective first table processing instructions, respective first tables, and respective first table processing results obtained by processing respective first tables using respective first table processing instructions to obtain a first training data set, wherein the first training data set includes at least one training sample whose first table processing result is a table and at least one training sample whose first table processing result is a descriptive text; A second training set determination module, configured to construct respective training samples based on respective second table processing instructions, respective second tables, and respective second table processing results obtained by processing respective second tables using respective second table processing instructions to obtain second training data sets, wherein the second training data sets only include training samples whose second table processing results are descriptive texts; A first training module, configured to train the first language model using the first training data set to obtain a second language model; A second training module, used to train the second language model using the second training data set to obtain a table text data generation model; The model application module is used to input the target table processing instruction and the target table into the table text data generation model to obtain the target table processing result output by the table text data generation model, wherein the target table processing result is a descriptive text.
8. The device according to claim 7, characterized in that The second training module comprises: a model merging unit, configured to merge the first largest language model and the second largest language model to obtain a third largest language model; A model training unit is used to train the third language model using the second training data set to obtain the tabular text data generation model.
9. An electronic device, characterized in that: include: at least one central processing unit; as well as A memory in communication with the at least one central processing unit; wherein, The memory stores instructions that can be executed by the at least one central processor, and the instructions are executed by the at least one central processor to enable the at least one central processor to perform the method according to any one of claims 1 to 6.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
Citation Information
Patent Citations
Text description generation method and system for table data
CN116484837A
Robot control method and device
CN117283549A
Data processing method and related device
CN117828049A
Intelligent session method and system suitable for vertical field based on large model
CN118035408A
Document processing method, device and equipment
CN118095433A
