Method, device, storage medium and electronic device for training table model

Abstract

The embodiments of this specification disclose a method, apparatus, storage medium, and electronic device for training a table model. The method first obtains a mask matrix corresponding to the table training data, then obtains corresponding masked text sequence data based on the table training data and the mask matrix, and then pre-trains based on the masked text sequence data to obtain a general table model. This scheme uses masking and then prediction to capture the complex relationship between table features using a mask matrix, and integrates different table tasks into one training objective by using different mask matrices, thereby realizing a new unified training framework.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a method, device, storage medium, and electronic device for training a table model. Background Art

[0002] Tabular data is widely used in various fields, such as finance, healthcare, and scientific research. Across these fields, tasks related to tabular data, such as prediction, generation, and interpolation, are common and crucial for users. The efficiency of processing these tasks can have a profound impact on data analysis and decision-making. In existing technologies, different downstream tasks related to tabular data are typically handled separately. For example, pre-training is performed for table prediction and table generation tasks, respectively, to produce models specifically for table prediction and table generation. Summary of the Invention

[0003] The purpose of the embodiments of this specification is to provide a method, device, storage medium and electronic device for training a table model.

[0004] The embodiments of this specification provide a method for training a table model, which uses a masking-first-then-prediction approach to unify various table tasks by using a mask matrix to pre-train a general table model. The method includes:

[0005] Obtaining a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data;

[0006] Obtaining corresponding masked text sequence data according to the table training data and the mask matrix, wherein the masked text sequence data includes a masked table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the masked table item is replaced with a first preset mark;

[0007] Pre-training is performed based on the masked text sequence data to obtain a general table model, wherein a training goal is to predict the content of the masked table item corresponding to the at least one masked position.

[0008] Furthermore, the method further comprises:

[0009] According to the task type of the downstream task, fine-tuning the general table model by adjusting the mask matrix and the training target to obtain a downstream table model corresponding to the downstream task;

[0010] For a target downstream task, the target downstream task is executed according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data, by using the downstream table model corresponding to the target downstream task, wherein the mask position corresponding to the target mask matrix is ​​determined according to the target downstream task.

[0011] Furthermore, the step of obtaining corresponding masked text sequence data according to the table training data and the mask matrix includes:

[0012] Serializing the table training data to obtain corresponding table text training data;

[0013] According to the tabular text training data and the mask matrix, corresponding masked text sequence data is obtained.

[0014] Furthermore, the serializing the table training data to obtain corresponding table text training data further includes:

[0015] If the content of the target table item in the table training data is missing, the target table item does not participate in serialization.

[0016] Furthermore, the step of obtaining corresponding masked text sequence data according to the table training data and the mask matrix includes:

[0017] Obtaining table mask training data according to the table training data and the mask matrix;

[0018] The table mask training data is serialized to obtain corresponding mask text sequence data.

[0019] Furthermore, the task type of the downstream task includes any one of the following:

[0020] Table prediction tasks;

[0021] Table generation tasks;

[0022] Table missing value imputation task.

[0023] Furthermore, the task type of the downstream task includes a table prediction task;

[0024] The step of fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes:

[0025] Adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table entry corresponding to the target column to be predicted in the downstream table training data;

[0026] The training objective is adjusted to predict the content of the table item corresponding to the target column in the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

[0027] Furthermore, the task type of the downstream task includes a table generation task;

[0028] The step of fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes:

[0029] Adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table entry to be generated in the downstream table training data;

[0030] The training objective is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive than the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

[0031] Furthermore, the table generation task is a conditional table generation task based on a label column, and the mask positions indicated by the adjusted mask matrix are positions of table items that need to be generated in the downstream table training data except for the label column;

[0032] Adjusting the training objective to generate table data having a distribution similar to and without duplication of the downstream table training data, and fine-tuning the general table model based on the downstream table training data, including:

[0033] The training objective is adjusted to generate, based on the label column, table data whose corresponding distribution is similar to and non-repeated than the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

[0034] Furthermore, the task type of the downstream task includes a table missing value interpolation task;

[0035] The step of fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes:

[0036] For downstream table training data containing missing table entries, adjusting the mask matrix according to a preset missing rate and the position of the missing table entry in the downstream table training data, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table entry in the downstream table training data that needs to be interpolated, and the table entry that needs to be interpolated is not the missing table entry;

[0037] The training objective is adjusted to interpolate the contents of table items that need to be interpolated in the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

[0038] Furthermore, the content of the missing table entry in the downstream table training data is set as a second preset mark.

[0039] The present invention also provides a method for training a table model, including:

[0040] A first obtaining module is configured to obtain a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data;

[0041] a second obtaining module, configured to obtain corresponding masked text sequence data based on the table training data and the mask matrix, wherein the masked text sequence data includes a masked table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the masked table item is replaced with a first preset mark;

[0042] A pre-training module is used to perform pre-training based on the masked text sequence data to obtain a general table model, wherein the training goal is to predict the content of the masked table item corresponding to the at least one mask position.

[0043] An embodiment of this specification further provides a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0044] An embodiment of this specification further provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0045] An embodiment of this specification also provides a computer program product having at least one instruction stored thereon, wherein the at least one instruction implements the steps of the above method when executed by a processor.

[0046] In an embodiment of the present specification, corresponding masked text sequence data is obtained based on table training data and its corresponding mask matrix, and pre-training is performed based on the masked text sequence data. A general table model suitable for various table tasks can be obtained. This scheme provides a unified pre-training paradigm, that is, masking first and then predicting. The mask matrix can be used to capture the complex relationship between table features. This method can enhance the model's understanding and operation of table data, thereby improving the processing performance of various table-related tasks. Moreover, by integrating different table tasks into one training target using different mask matrices, a new unified training framework is also implemented, which not only keeps the pre-training tasks consistent with downstream tasks, but also significantly improves the versatility and practicality of the general table model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a method for training a table model provided in an embodiment of this specification;

[0048] Figure 2 A schematic diagram of a process for executing a target downstream task provided in an embodiment of this specification;

[0049] Figure 3 A schematic diagram of the structure of a device for training a table model provided in an embodiment of this specification;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of this specification more clear, the following will clearly and completely describe the technical solutions of this specification in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0052] See Figure 1 , is a flow chart of a method for training a table model provided in an embodiment of this specification. In an embodiment of this specification, the method for training a table model is applied to a device for training a table model (hereinafter referred to as a "table model training device") or an electronic device equipped with a table model training device. Figure 1 The process shown in FIG. 1 is described in detail. The method for training a table model may include the following steps:

[0053] S102: Obtain a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data.

[0054] In some embodiments, the mask matrix includes at least one mask indication information (mask indication information is used to indicate that masking is required). For example, for a mask matrix P, the elements in P are expressed as 0 or 1, 1 represents mask indication information, and 0 represents non-mask indication information, that is, the position where the element is expressed as 1 corresponds to the position of the table training data as the mask position. For example, all elements in the third column of P are 1, and the remaining elements are 0. Then, the mask matrix indicates that the mask position corresponding to the table training data is the position of the table entry in the third column of the table training data, that is, the third column of the table training data needs to be masked. In some embodiments, the mask matrix corresponding to the table training data is randomly generated. For example, for the table T=∈R n ×m , where n represents the number of rows in table T, m represents the number of columns in table T, and R represents the matrix corresponding to the table. The mask matrix M corresponding to the table T can be randomly generated, M∈{0,1} n×m , the positions where the elements in the mask matrix M are 1 correspond to the mask positions in the table T. In some embodiments, the mask matrix corresponding to the table training data can be generated based on a pre-set mask rate. For example, if the mask rate is 30%, 30% of the element values ​​in the generated mask matrix are 1 (representing mask indication information), and the other 70% of the element values ​​are 0 (representing non-mask indication information). It should be noted that the above implementation method for obtaining the mask matrix is ​​only an example and not a limitation of the present application. Those skilled in the art should understand that any implementation method for obtaining the mask matrix corresponding to the table training data should be included in the protection scope of the present application.

[0055] In some embodiments, the tabular training data includes tabular data samples corresponding to various downstream task types. In some embodiments, tabular training data for different types of tasks can be collected from different sources. For example, the tabular training data includes tabular data samples from different sources for various types of tasks, such as binary, multi-class classification, and regression tasks. The embodiments of this specification are not limited to tabular training data.

[0056] S104. Obtain corresponding mask text sequence data based on the table training data and the mask matrix, wherein the mask text sequence data includes a mask table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the mask table item is replaced with a first preset mark.

[0057] In some embodiments, the masked text sequence data represents text data obtained by masking and serializing the table training data based on a masking matrix. The masked text sequence data represents the contents of each table item after masking the table training data in text form, facilitating understanding by a large language model. In some embodiments, the masked table item is obtained by masking the table item corresponding to the masked position in the table training data. The content of the masked table item is replaced with a first predetermined marker, which indicates that the table item is masked. For example, a 1 is used in the masking matrix to represent masking indication information. After masking the table training data based on the masking matrix, the content of each masked table item is marked with a first predetermined marker (e.g., [MASK]) to indicate that the masked table item is masked. Table items other than the masked position in the table training data, i.e., those that do not need to be masked, remain unchanged because they are not masked. In some embodiments, a table item represents the position (or feature name) and content of the item in the table. In some embodiments, serialization is achieved by connecting the feature names and values ​​in the table into sentences, for example, in the form of "[feature] is [Value]", where "feature" represents the feature name corresponding to the table item, "Value" represents the value corresponding to the table item, and "is" is a conjunction. For example, given a table T={t ij}∈R n×m , where n represents the number of rows in table T, m represents the number of columns in table T, R represents the matrix corresponding to the table, and t ij It is used to represent the jth eigenvalue of the i-th sample (the i-th sample in this example is also the i-th row of table T), then F j Represents a feature column, which can convert the i-th sample into, for example, "F1 is t i1 , F2 is t i2 ,…,F m It is t im ”.

[0058] In some embodiments, the table training text can be serialized first, and then the serialized text information can be modified based on at least one mask position indicated by the mask matrix (equivalent to modifying the content of the table item corresponding to the mask position in the table training data) to obtain masked text sequence data. Alternatively, the table training data can be masked based on the mask matrix to obtain masked table data, and then the masked table data can be serialized to obtain masked text sequence data. The above two implementation methods will be described in detail in subsequent embodiments and will not be repeated here. Those skilled in the art should understand that the above implementation method of obtaining masked text sequence data is only an example and not a limitation of the present application. Any implementation method of obtaining corresponding masked text sequence data based on table training data and its corresponding mask matrix should be included in the scope of inclusion of the present application.

[0059] S106 , performing pre-training based on the masked text sequence data to obtain a general table model, wherein a training goal is to predict the content of the masked table item corresponding to the at least one masked position.

[0060] In some embodiments, a prompt text is generated based on the masked text sequence data (i.e., a pre-training task is constructed), and the prompt text is input into a large language model for training to obtain a general table model; in some embodiments, the large language model used for pre-training uses the default cross entropy loss. In some embodiments, a prompt text is generated based on the masked text sequence data and the training target. For example, given a table T∈R n×m , where n represents the number of rows in table T, m represents the number of columns in table T, R represents the matrix corresponding to the table, and the mask matrix M∈{0,1} n×m , mask the content corresponding to the mask position in table T according to the following formula:

[0061]

[0062] Where X is a first preset mark used to indicate the mask position in the table T, T(i,j) represents the original value of the i-th row and j-th column in the table before masking, and T m (i, j) represents the value of the i-th row and j-th column after masking, M(i, j) represents the value of the i-th row and j-th column in the mask matrix, and "if" is used to determine whether the given condition is met. Based on the above formula, the position of the element with a value of 1 in the mask matrix also indicates the mask position in table T. After masking, the content of the table item corresponding to the mask position in table T is replaced by X, and the remaining positions keep their original values ​​unchanged, then F is used. j Represents the jth feature name, then for the i-th row T(i, :), the mask position of the i-th row can be defined as:

[0063] Ф i={k|M(i,k) = 1}

[0064] Among them, Ф i Indicates the mask position of the i-th row, k∈Ф i , then the form of "[feature] is [Value]" is used to serialize and obtain the masked text sequence data. Based on the masked text sequence data and the training target, the prompt text is generated as follows: "The question is "F1 is T m (i,1), F2 is T m (i,2),…,F m It's T m (i,m), interpolate missing values", for all k∈Ф i , the answer is "F k is T(i,k)". In some embodiments, the pre-trained framework adopts an autoregressive architecture, which adopts appropriately labeled tabular data input (that is, tabular data input determined based on masked text sequence data) and learns to use the contextual information of the unmasked value to predict the masked value (that is, the content corresponding to the masked table item). By using an autoregressive model and a variable mask matrix, various types of table-related tasks can be integrated into a unified framework, which not only improves the performance of the general table model in tabular data tasks, but also enhances its adaptability to subsequent tasks.

[0065] According to the scheme of the embodiments of this specification, by obtaining corresponding masked text sequence data based on table training data and its corresponding mask matrix, and pre-training based on the masked text sequence data, a general table model suitable for various table tasks can be obtained. This scheme provides a unified pre-training paradigm, that is, masking first and then predicting. The mask matrix can be used to capture the complex relationship between table features. This method can enhance the model's understanding and operation of table data, thereby improving the processing performance of various table-related tasks. Moreover, by integrating different table tasks into one training target using different mask matrices, a new unified training framework is also implemented, which not only keeps the pre-training tasks consistent with downstream tasks, but also significantly improves the versatility and practicality of the general table model.

[0066] In some embodiments, the obtaining of corresponding masked text sequence data according to the table training data and the mask matrix includes: serializing the table training data to obtain corresponding table text training data; obtaining corresponding masked text sequence data according to the table text training data and the mask matrix. In some embodiments, the table text training data is text data obtained by directly serializing the table training data, and the table text training data includes text data corresponding to each table item of the table training data. In some embodiments, according to the mask position indicated by the mask matrix, the text data to be masked corresponding to the mask position in the table text training data is determined, and then the text data is replaced with a first preset mark, thereby obtaining masked text sequence data corresponding to the table training data. For example, directly serializing the table T can convert the i-th row in the table into the text data "F1 is t i1 , F2 is t i2 ,…,F m It is t im ", based on the mask matrix to determine t i1 Need to cover (ie t i1 is the text data to be masked), then the “t i1 " is replaced by the first preset mark " [MASK] ". In some embodiments, the tabular text training data is modified according to the tabular text training data, the mask matrix and the preset masking rules, and the modified text data is used as the masked text sequence data. The implementation method of serializing the tabular training data is the same as or similar to the implementation method of the aforementioned serialization, and will not be repeated here.

[0067] In some embodiments, the serialization of the table training data to obtain the corresponding table text training data further includes: if the content of the target table item in the table training data is missing, the target table item is not serialized. In some embodiments, if the content of the target table item in the table training data is missing, the target table item may not be serialized, that is, the serialized table text training data is obtained only for the non-missing table items; for example, for the i-th row of table T, since the value of the second feature column is missing, the second feature column may be skipped during the serialization process, and the serialized text "F1 is t i1 , F3 is t i3 ,…,F m It is t im ”.

[0068] In some embodiments, if the content of the target table item in the table training data is missing, the content of the target table item can be set to a second preset mark in the pre-training stage, and then serialized based on this to obtain the corresponding table text training data. The second preset mark is used to indicate that the content of the table item is missing. Continuing with the above example, for the i-th row of table T, since the value of the second feature column is missing, the value of the second feature column is set to the second preset mark "[Impute]" during the serialization process, and the serialized text "F1 is t i1 , F2 is [Impute], F3 is t i3 ,…,F m It is t im ". In some embodiments, if the content of the target table item in the table training data is missing, the mask position indicated by the mask matrix is ​​not the missing table item, that is, the missing table item does not need to be masked; when obtaining the mask matrix corresponding to the table training data based on the preset mask rate, the number of masks can be determined based on the product of the number of non-missing table items and the preset mask rate, and then the mask matrix is ​​randomly generated. For example, if the mask rate is 30%, 30% of the elements corresponding to the non-missing table items in the mask matrix have element values ​​of 1 (that is, mask indication information), and the element values ​​corresponding to the remaining non-missing table items and missing table items are all 0.

[0069] In some embodiments, the step of obtaining corresponding masked text sequence data based on the table training data and the mask matrix includes: obtaining table mask training data based on the table training data and the mask matrix; and serializing the table mask training data to obtain corresponding masked text sequence data. In some embodiments, the table mask training data represents table data obtained by masking the table training data based on the mask matrix. For example, for a given table T∈R n×m , where n represents the number of rows in table T, m represents the number of columns in table T, R represents the matrix corresponding to the table, and the mask matrix M∈{0,1} is used. n×m Mask at least one mask position in table T to obtain the masked table T m In some embodiments, the implementation method of serializing the table mask training data is the same as or similar to the aforementioned implementation method of serialization, and will not be repeated here.

[0070] Figure 2 A flowchart of a method for executing a target downstream task provided in an embodiment of this specification includes the following steps:

[0071] S202: Obtain a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data. The implementation of step S202 is the same or similar to the implementation of step S102, and will not be repeated here.

[0072] S204: Obtain corresponding masked text sequence data based on the table training data and the mask matrix, wherein the masked text sequence data includes a masked table entry after masking the mask position in the table training data, and other table entries in the table training data except for the mask position, wherein the contents of the masked table entry are replaced with a first preset marker. The implementation of step S204 is the same or similar to the implementation of step S104, and is not further described here.

[0073] S206: Pre-training is performed based on the masked text sequence data to obtain a general table model, wherein the training objective is to predict the content of the masked table item corresponding to the at least one masked position. The implementation of step S206 is the same or similar to the implementation of step S106, and will not be repeated here.

[0074] S208 , fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task, to obtain a downstream table model corresponding to the downstream task.

[0075] In some embodiments, the task type of the downstream task includes any one of a table prediction task, a table generation task, and a table missing value interpolation task. In some embodiments, the goal of the table prediction task is to predict the value of each sample target column based on existing data. In some embodiments, the goal of the table generation task is to generate table data with a distribution similar to the original data but without duplication; in some embodiments, the table generation task is conditionally generated based on the label column, that is, its goal is to generate table data with a distribution similar to the original data but without duplication based on the known label column. In some embodiments, the purpose of the table missing value interpolation task is to interpolate missing values ​​in the table.

[0076] In some embodiments, after pre-training, the general table model is fine-tuned by adjusting the mask matrix and training objective to adapt to specific downstream tasks. For example, for a table prediction task, the mask matrix is ​​adjusted to mask the target value to be predicted. Prompt text is generated based on the adjusted mask matrix and training objective to fine-tune the model for the prediction task, thereby obtaining a downstream table model adapted for the table prediction task. The specific implementation of fine-tuning the general table model by adjusting the mask matrix and training objective, depending on the type of downstream task, will be described in detail in subsequent embodiments and will not be repeated here.

[0077] S210, for the target downstream task, according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data, by using the downstream table model corresponding to the target downstream task, the target downstream task is executed, wherein the mask position corresponding to the target mask matrix is ​​determined according to the target downstream task. In some embodiments, the input of the downstream table model is the input information corresponding to the target table data (for example, the corresponding mask text sequence data can be obtained based on the target table data and the target mask matrix, and then the corresponding prompt question is generated as the input information in combination with the training target), and the output of the downstream table model is the execution result of the target downstream task, such as the value that needs to be predicted, generated or interpolated. In some embodiments, the target downstream task is a table prediction task, and the mask position indicated by the target mask matrix corresponds to the target column that needs to be predicted in the target table data. Based on the target mask matrix, the target column that needs to be predicted in the target table data can be masked to output the predicted value of the target column through the corresponding downstream table model. In some embodiments, the target downstream task is a conditional table generation task based on a label column, and the mask positions indicated by the target mask matrix correspond to values ​​other than the label column in the target table data. Based on the target mask matrix, other values ​​other than the label column that need to be generated in the target table data can be masked, so that the values ​​of other columns generated based on the label column can be output through the corresponding downstream table model. In some embodiments, the target downstream task is an unconditional table generation task, and the mask positions indicated by the target mask matrix correspond to all values ​​in the target table data. Based on the target mask matrix, all values ​​can be masked to generate table data with a distribution similar to the original data but without duplication. In some embodiments, the target downstream task is a table missing value interpolation task, and the mask positions indicated by the target mask matrix correspond to missing values ​​in the target table data (i.e., positions that need to be interpolated). Based on the target mask matrix, the missing values ​​in the target table data can be masked, so that the corresponding downstream table model can output predicted values ​​for the missing values ​​to complete the interpolation of the missing values.

[0078] The embodiments of this specification can fine-tune the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task to obtain a downstream table model corresponding to the downstream task. This new pre-training paradigm enhances the ability of the downstream table model for specific downstream tasks and improves the performance and efficiency of the data analysis and decision-making process.

[0079] In some embodiments, the task type of the downstream task includes a table prediction task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item corresponding to the target column to be predicted in the downstream table training data; adjusting the training target to predict the content of the table item corresponding to the target column in the downstream table training data, and fine-tuning the general table model based on the downstream table training data. In some embodiments, for the table prediction task, the training target is to predict the value of the target column for each sample. Based on the training target, the value of the target column to be predicted needs to be masked, and the general table model is fine-tuned on the prediction task to adapt it to the table prediction task. In some embodiments, the downstream table training data can be derived in whole or in part from the above-mentioned table training data, or it can be table training data collected outside the above-mentioned table training data.

[0080] As an example, for the downstream table prediction task, the adjusted mask matrix is ​​as follows. Based on this mask matrix and the training objective of the table prediction task, the model can be fine-tuned on the prediction task (that is, generating prompt text suitable for the table prediction task):

[0081]

[0082] Among them, M(i,j) represents the value of the i-th row and j-th column in the mask matrix M, W represents the condition that (i,j) is the target value, (i,j) is the target value, that is, it corresponds to a known label column, and "else" represents other situations except W; in this example, if (i,j) is the target value, that is, for the target column to be predicted, the value of M(i,j) is set to 1, otherwise it is set to 0, thereby masking the value of the target column to be predicted. Based on the adjusted mask matrix and the training target, the corresponding prompt text can be generated (the implementation method of generating the prompt text here is similar to the implementation method of generating the prompt text in the aforementioned pre-training stage, which will not be repeated here), so as to fine-tune the model on the prediction task.

[0083] In some embodiments, the downstream task includes a table generation task. Fine-tuning the general table model by adjusting the mask matrix and the training objective based on the downstream task type includes: adjusting the mask matrix, wherein the mask positions indicated by the adjusted mask matrix correspond to the positions of table entries to be generated in the downstream table training data; adjusting the training objective to generate table data with a distribution similar to that of the downstream table training data and without duplication, and fine-tuning the general table model based on the downstream table training data. In some embodiments, for the table generation task, the training objective is to generate table data with a distribution similar to that of the original data but without duplication. In some embodiments, the table generation task includes an unconditional table generation task and a conditional table generation task based on a label column. For the unconditional table generation task, all elements in the adjusted mask matrix have a mask indication (e.g., 1). Based on the adjusted mask matrix, all values ​​in the downstream table training data can be set to a first predetermined marker (e.g., a [MASK] character), thereby masking all values. Fine-tuning methods for the conditional table generation task based on the label column will be described in detail in subsequent embodiments and are not further elaborated here.

[0084] In some embodiments, the table generation task is a conditional table generation task based on a label column, and the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be generated in the downstream table training data other than the label column; the training objective is adjusted to generate table data whose corresponding distribution is similar to and non-repeating than the downstream table training data, and the general table model is fine-tuned based on the downstream table training data, including: adjusting the training objective to generate table data whose corresponding distribution is similar to and non-repeating than the downstream table training data based on the label column, and fine-tuning the general table model based on the downstream table training data. In some embodiments, for the conditional table generation task based on the label column, in the adjusted mask matrix, except for the elements corresponding to the label column whose values ​​are non-mask indication information (e.g., 0), the values ​​of the remaining elements are all mask indication information (e.g., 1). Based on the adjusted mask matrix, all other values ​​in the downstream table training data other than the label column can be set to a first preset mark (e.g., the [MASK] character), that is, the values ​​other than the label column are masked. As an example, for the table generation task based on the label column, the adjusted mask matrix can be expressed as:

[0085]

[0086] In this example, W represents the condition that (i, j) is the target value. (i, j) is the target value, which means it corresponds to a known label column. "else" represents other situations except W, thereby masking values ​​outside the label column.

[0087] In some embodiments, the task type of the downstream task includes a table missing value interpolation task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: for downstream table training data containing missing table items, according to a preset missing rate and the position of the missing table items in the downstream table training data, adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be interpolated in the downstream table training data, and the table item that needs to be interpolated is not the missing table item; adjusting the training target to interpolate the content of the table item that needs to be interpolated in the downstream table training data, and fine-tuning the general table model based on the downstream table training data. In some embodiments, for the table missing value interpolation task, the training target is to interpolate the missing values ​​in the table. In some embodiments, for the task of interpolating missing values ​​in a table, a method of first masking and then predicting is used to process values ​​other than missing values ​​(i.e., non-missing values); for tables without missing values, a mask matrix can be directly constructed based on the missing rate. For example, if the missing rate is 30%, a mask matrix corresponding to the table is constructed, wherein 30% of the elements in the mask matrix have mask indication information (e.g., 1), and the values ​​of the remaining elements are non-mask indication information (e.g., 0); for tables with original missing values, the non-missing values ​​in the table need to be masked according to the missing rate. For example, if the missing rate is 30%, the mask position indicated by the constructed mask matrix corresponds to at least one non-missing value in the table, and the ratio of the number of mask positions to the number of non-missing table items in the table is equal to the missing rate.

[0088] In some embodiments, the content of the missing table entries in the downstream table training data is set to a second preset mark. As an example, given a table T, the missing table entries in the table T are represented by a binary mask matrix M ori ∈{0,1} n×m Indicates that, if M ori (i,j)=0 means the value of row i and column j is not missing. ori (i,j)=1 means that the value of row i and column j is missing (that is, the corresponding table entry is missing); a new mask matrix M∈{0,1} is randomly selected according to the preset missing rate n×m , during the training process, the table T is masked based on the following formula:

[0089]

[0090] in," " represents the intersection operation, T(i,j) represents the original value of the i-th row and j-th column in the table T before masking, Tm (i, j) represents the value of the i-th row and j-th column in the masked table, Y is the second preset mark used to indicate a missing table item, and X is the first preset mark used to indicate a masked item. Thus, the original missing values ​​in the table T are filled with Y, and the values ​​of the table items that need to be masked but are not missing are replaced with X. Then, each row in the masked table can be serialized into text data, that is, the corresponding masked text sequence data is obtained for fine-tuning the model. It should be noted that in the above example, since the masked items are the contents of the table items that are not missing in the table and the corresponding elements in the mask matrix are the mask indication information, for the i-th row T(i, :) of the table T, the mask position of the i-th row is defined as:

[0091] Ф i ={k|M(i,k) = 1 ∧M ori (i,k)=0}

[0092] Among them, Ф i Indicates the mask position of the i-th row, k∈Ф i .

[0093] In some embodiments, after adjusting the mask matrix and the training target according to the task type of the downstream task, the mask text sequence data corresponding to the downstream table training data is obtained based on the adjusted mask matrix, and then a prompt text is generated based on the mask text sequence data and the adjusted training target, and the prompt text is input into the large language model for training to adapt to the corresponding task type; wherein, the implementation method of generating the prompt text is similar to the implementation method of generating the prompt text in the pre-training stage mentioned above, and will not be repeated here. It should be noted that in some embodiments, for downstream table training data with missing table items, the value of the missing table item can be set to a second preset mark before serialization, or it can be not serialized.

[0094] It should be noted that the method of the embodiments of this specification can be used in all application scenarios related to table tasks, and this application does not limit this. For example, in risk control scenarios, to address the problem of sparse black samples, the method of the embodiments of this specification can be used to process related downstream tasks, which can play a role in data enhancement.

[0095] Figure 3 The following is a schematic diagram of a device for training a table model (hereinafter referred to as "table model training device 1") provided in embodiments of this specification. This device can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the table model training device 1 includes a first acquisition module 11, a second acquisition module 12, and a pre-training module 13.

[0096] A first obtaining module 11 is configured to obtain a mask matrix corresponding to the table training data, wherein the mask matrix is ​​configured to indicate at least one mask position corresponding to the table training data;

[0097] a second obtaining module 12, configured to obtain corresponding masked text sequence data based on the table training data and the mask matrix, wherein the masked text sequence data includes a masked table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the masked table item is replaced with a first preset mark;

[0098] The pre-training module 13 is configured to perform pre-training based on the masked text sequence data to obtain a general table model, wherein a training goal is to predict the content of the masked table item corresponding to the at least one masked position.

[0099] In some embodiments, the table model training device 1 is also used to: fine-tune the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task to obtain the downstream table model corresponding to the downstream task; for the target downstream task, according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data, perform the target downstream task by using the downstream table model corresponding to the target downstream task, wherein the mask position corresponding to the target mask matrix is ​​determined according to the target downstream task.

[0100] In some embodiments, the second obtaining module 12 is used to: serialize the table training data to obtain corresponding table text training data; and obtain corresponding mask text sequence data according to the table text training data and the mask matrix.

[0101] In some embodiments, serializing the table training data to obtain corresponding table text training data further includes: if the content of the target table item in the table training data is missing, making the target table item not participate in serialization.

[0102] In some embodiments, the second obtaining module 12 is configured to: obtain table mask training data according to the table training data and the mask matrix; and serialize the table mask training data to obtain corresponding mask text sequence data.

[0103] In some embodiments, the task type of the downstream task includes any one of the following: a table prediction task; a table generation task; a table missing value interpolation task.

[0104] In some embodiments, the task type of the downstream task includes a table prediction task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item corresponding to the target column to be predicted in the downstream table training data; adjusting the training target to predict the content of the table item corresponding to the target column in the downstream table training data, and fine-tuning the general table model based on the downstream table training data.

[0105] In some embodiments, the task type of the downstream task includes a table generation task; wherein, fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be generated in the downstream table training data; adjusting the training target to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and fine-tuning the general table model based on the downstream table training data.

[0106] In some embodiments, the table generation task is a conditional table generation task based on a label column, and the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be generated in the downstream table training data except for the label column; the training objective is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data, including: adjusting the training objective to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data based on the label column, and fine-tuning the general table model based on the downstream table training data.

[0107] In some embodiments, the task type of the downstream task includes a table missing value interpolation task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: for downstream table training data containing missing table items, according to a preset missing rate and the position of the missing table items in the downstream table training data, adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be interpolated in the downstream table training data, and the table item that needs to be interpolated is not the missing table item; adjusting the training target to interpolate the content of the table item that needs to be interpolated in the downstream table training data, and fine-tuning the general table model based on the downstream table training data.

[0108] In some embodiments, the content of the missing table entry in the downstream table training data is set as a second preset flag.

[0109] The above-mentioned device embodiments correspond to the method embodiments. For detailed descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For detailed descriptions, please refer to the corresponding method embodiments.

[0110] The embodiments of this specification also provide a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the method of the embodiments of this specification.

[0111] An embodiment of the present specification further provides a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor to execute the method of the embodiment of the present specification.

[0112] The embodiments of this specification also provide Figure 4 The structural diagram of the electronic device shown in FIG. Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for its operations. The processor reads the corresponding computer program from the non-volatile storage into the memory and then runs it to implement the above method.

[0113] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0114] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0118] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0119] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0120] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0121] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A method for training a table model, comprising: Obtaining a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data; Obtaining corresponding masked text sequence data according to the table training data and the mask matrix, wherein the masked text sequence data includes a masked table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the masked table item is replaced with a first preset mark; Performing pre-training based on the masked text sequence data to obtain a general table model, wherein a training goal is to predict the content of the masked table item corresponding to the at least one masked position; The method further comprises: According to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target to obtain a downstream table model corresponding to the downstream task, wherein the task type of the downstream task includes any one of a table prediction task, a table generation task, and a table missing value interpolation task: For a target downstream task, the target downstream task is executed according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data, by using the downstream table model corresponding to the target downstream task, wherein the mask position corresponding to the target mask matrix is ​​determined according to the target downstream task.

2. The method according to claim 1, wherein obtaining corresponding masked text sequence data based on the tabular training data and the mask matrix comprises: Serializing the table training data to obtain corresponding table text training data; According to the tabular text training data and the mask matrix, corresponding masked text sequence data is obtained.

3. The method according to claim 2, wherein the serializing the table training data to obtain corresponding table text training data further comprises: If the content of the target table item in the table training data is missing, the target table item does not participate in serialization.

4. The method according to claim 1, wherein obtaining corresponding masked text sequence data based on the tabular training data and the mask matrix comprises: Obtaining table mask training data according to the table training data and the mask matrix; The table mask training data is serialized to obtain corresponding mask text sequence data.

5. The method according to claim 1, wherein the task type of the downstream task comprises a table prediction task; in, Fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: Adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table entry corresponding to the target column to be predicted in the downstream table training data; The training objective is adjusted to predict the content of the table item corresponding to the target column in the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

6. The method according to claim 1, wherein the task type of the downstream task comprises a table generation task; in, Fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: Adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table entry to be generated in the downstream table training data; The training objective is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive than the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

7. The method according to claim 6, wherein the table generation task is a conditional table generation task based on a label column, and the mask positions indicated by the adjusted mask matrix are positions of table entries to be generated in downstream table training data other than the label column; Adjusting the training objective to generate table data having a distribution similar to and without duplication of the downstream table training data, and fine-tuning the general table model based on the downstream table training data, including: The training objective is adjusted to generate, based on the label column, table data whose corresponding distribution is similar to and non-repeated than the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

8. The method according to claim 1, wherein the task type of the downstream task includes a table missing value interpolation task; in, Fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: For downstream table training data containing missing table entries, adjusting the mask matrix according to a preset missing rate and the position of the missing table entry in the downstream table training data, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table entry in the downstream table training data that needs to be interpolated, and the table entry that needs to be interpolated is not the missing table entry; The training objective is adjusted to interpolate the contents of table items that need to be interpolated in the downstream table training data, and the general table model is fine-tuned based on the downstream table training data. 9 . The method according to claim 8 , wherein the content of the missing table entry in the downstream table training data is set as a second preset mark.

10. A device for training a table model, comprising: A first obtaining module is configured to obtain a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data; a second obtaining module, configured to obtain corresponding masked text sequence data based on the table training data and the mask matrix, wherein the masked text sequence data includes a masked table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the masked table item is replaced with a first preset mark; a pre-training module, configured to perform pre-training based on the masked text sequence data to obtain a general table model, wherein a training goal is to predict the content of the masked table item corresponding to the at least one masked position; Wherein, the device is also used for: According to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target to obtain a downstream table model corresponding to the downstream task, wherein the task type of the downstream task includes any one of a table prediction task, a table generation task, and a table missing value interpolation task: For a target downstream task, the target downstream task is executed according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data, by using the downstream table model corresponding to the target downstream task, wherein the mask position corresponding to the target mask matrix is ​​determined according to the target downstream task.

11. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method according to any one of claims 1 to 9.

13. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images