Method and device for training table model, storage medium and electronic equipment

Abstract

The embodiment of the invention discloses a method and device for training a table model, a storage medium and electronic equipment, and the method comprises the steps: firstly obtaining a mask matrix corresponding to table training data, then obtaining corresponding mask text sequence data according to the table training data and the mask matrix, and obtaining the table model according to the mask text sequence data. According to the scheme, the mask text sequence data is masked and then predicted, the complex relation between table features can be captured by utilizing the mask matrixes, and different table tasks are integrated into one training target by utilizing different mask matrixes, so that the training efficiency is improved. And a brand-new unified training framework is realized.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a method, device, storage medium and electronic device for training a table model. Background Art

[0002] Tabular data is widely used in various fields, such as finance, medical treatment, scientific research, etc. In different fields, tasks such as prediction, generation, and interpolation related to tabular data are common and crucial for users. The processing efficiency of such tasks may have a profound impact on data analysis and decision-making. In the prior art, different downstream tasks related to tabular data are usually processed separately, for example, pre-training is performed for table prediction tasks and table generation tasks respectively, so as to obtain a model dedicated to table prediction tasks and a model dedicated to table generation tasks respectively. Summary of the invention

[0003] The purpose of the embodiments of this specification is to provide a method, device, storage medium and electronic device for training a table model.

[0004] The embodiment of this specification provides a method for training a table model, which adopts a method of first masking and then predicting, and can unify various table tasks by using a mask matrix to pre-train a general table model. The method includes: Obtaining a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data; Obtaining corresponding mask text sequence data according to the table training data and the mask matrix, wherein the mask text sequence data includes a mask table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the mask table item is replaced with a first preset mark; Pre-training is performed based on the masked text sequence data to obtain a general table model, wherein a training goal is to predict the content of the masked table item corresponding to the at least one masked position.

[0005] Furthermore, the method further comprises: According to the task type of the downstream task, fine-tune the general table model by adjusting the mask matrix and the training target to obtain a downstream table model corresponding to the downstream task; For a target downstream task, the target downstream task is executed according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data, by using the downstream table model corresponding to the target downstream task, wherein the mask position corresponding to the target mask matrix is ​​determined based on the target downstream task.

[0006] Furthermore, the step of obtaining corresponding masked text sequence data according to the table training data and the mask matrix includes: Serializing the table training data to obtain corresponding table text training data; According to the tabular text training data and the mask matrix, corresponding masked text sequence data is obtained.

[0007] Furthermore, the serializing the table training data to obtain corresponding table text training data also includes: If the content of the target table item in the table training data is missing, the target table item does not participate in serialization.

[0008] Furthermore, the step of obtaining corresponding masked text sequence data according to the table training data and the mask matrix includes: Obtaining table mask training data according to the table training data and the mask matrix; The table mask training data is serialized to obtain corresponding mask text sequence data.

[0009] Furthermore, the task type of the downstream task includes any of the following: Table prediction tasks; Table generation tasks; Table missing value imputation task.

[0010] Further, the task type of the downstream task includes a table prediction task; Wherein, fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: Adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item corresponding to the target column to be predicted in the downstream table training data; The training objective is adjusted to predict the content of the table item corresponding to the target column in the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

[0011] Further, the task type of the downstream task includes a table generation task; Wherein, fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: Adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table entry to be generated in the downstream table training data; The training objective is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

[0012] Furthermore, the table generation task is a conditional table generation task based on the label column, and the mask position indicated by the adjusted mask matrix is ​​the position of the table item to be generated in the downstream table training data except the label column; The training objective is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data, including: The training objective is adjusted to generate, based on the label column, table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

[0013] Furthermore, the task type of the downstream task includes a table missing value interpolation task; Wherein, fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: For downstream table training data containing missing table items, adjusting the mask matrix according to a preset missing rate and the position of the missing table items in the downstream table training data, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be interpolated in the downstream table training data, and the table item that needs to be interpolated is not the missing table item; The training objective is adjusted to interpolate the contents of table items that need to be interpolated in the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

[0014] Further, the content of the missing table item in the downstream table training data is set as a second preset mark.

[0015] The embodiment of this specification also provides a device for training a table model, including: A first obtaining module, used to obtain a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data; a second obtaining module, configured to obtain corresponding mask text sequence data according to the table training data and the mask matrix, wherein the mask text sequence data includes a mask table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the mask table item is replaced with a first preset mark; A pre-training module is used to perform pre-training based on the masked text sequence data to obtain a general table model, wherein the training goal is to predict the content of the masked table item corresponding to the at least one masked position.

[0016] The embodiments of the present specification also provide a storage medium, wherein the storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the steps of the above method.

[0017] An embodiment of the present specification also provides an electronic device, comprising: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the above method.

[0018] The embodiments of the present specification also provide a computer program product having at least one instruction stored thereon, wherein the at least one instruction implements the steps of the above method when executed by a processor.

[0019] In an embodiment of the present specification, by obtaining corresponding masked text sequence data based on table training data and its corresponding mask matrix, and pre-training based on the masked text sequence data, a general table model suitable for various table tasks can be obtained. This scheme provides a unified pre-training paradigm, that is, masking first and then predicting. The mask matrix can be used to capture the complex relationship between table features. This method can enhance the model's understanding and operation of table data, thereby improving the processing performance of various table-related tasks. In addition, by using different mask matrices to integrate different table tasks into one training target, a new unified training framework is also implemented, which not only makes the pre-training tasks consistent with downstream tasks, but also significantly improves the versatility and practicality of the general table model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of a method for training a table model provided in an embodiment of this specification; Figure 2 A schematic diagram of a process for executing a target downstream task provided in an embodiment of this specification; Figure 3 A schematic diagram of the structure of a device for training a table model provided in an embodiment of this specification; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0022] See also Figure 1 , is a flowchart of a method for training a table model provided in an embodiment of this specification. In an embodiment of this specification, the method for training a table model is applied to a device for training a table model (hereinafter referred to as a "table model training device") or an electronic device equipped with a table model training device. Figure 1 The process shown is described in detail, and the method for training the table model may specifically include the following steps: S102, obtaining a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data.

[0023] In some embodiments, the mask matrix includes at least one mask indication information (mask indication information is used to indicate that masking is required). For example, for a mask matrix P, the elements in P are expressed as 0 or 1, 1 represents mask indication information, and 0 represents non-mask indication information, that is, the position where the element is expressed as 1 corresponds to the position of the table training data as the mask position. For example, all elements in the third column of P are 1, and the remaining elements are 0. Then the mask matrix indicates that the mask position corresponding to the table training data is the position of the table entry in the third column of the table training data, that is, the third column of the table training data needs to be masked. In some embodiments, the mask matrix corresponding to the table training data is randomly generated. For example, for the table T=∈R n ×m , where n represents the number of rows in table T, m represents the number of columns in table T, and R represents the matrix corresponding to the table. The mask matrix M corresponding to the table T can be randomly generated, M∈{0,1} n×m, the positions where the elements in the mask matrix M are 1 correspond to the mask positions in the table T. In some embodiments, the mask matrix corresponding to the table training data can be generated based on a preset mask rate. For example, if the mask rate is 30%, 30% of the element values ​​in the generated mask matrix are 1 (representing mask indication information), and the other 70% of the element values ​​are 0 (representing non-mask indication information). It should be noted that the above implementation method for obtaining the mask matrix is ​​only an example, not a limitation of the present application. Those skilled in the art should understand that any implementation method for obtaining the mask matrix corresponding to the table training data should be included in the protection scope of the present application.

[0024] In some embodiments, the tabular training data includes tabular data samples corresponding to various downstream task types. In some embodiments, tabular training data of different types of tasks from different sources can be collected. For example, the tabular training data includes tabular data samples of various types including binary, multi-class classification, regression tasks, etc. from different sources. The embodiments of this specification do not limit the tabular training data.

[0025] S104, obtaining corresponding mask text sequence data according to the table training data and the mask matrix, wherein the mask text sequence data includes mask table items after masking the mask positions in the table training data, and other table items in the table training data except the mask positions, and the content of the mask table items is replaced with a first preset mark.

[0026] In some embodiments, the masked text sequence data is used to represent text data after the table training data is masked and serialized based on the mask matrix, and the masked text sequence data expresses the content of each table item after the table training data is masked in a textual manner, so as to facilitate the understanding of the large language model. In some embodiments, the masked table item is obtained by masking the table item corresponding to the mask position in the table training data, and the content of the masked table item is replaced by a first preset mark, which represents that the table item is masked. For example, the mask indication information is represented by 1 in the mask matrix. After the table training data is masked based on the mask matrix, the content of each masked table item is marked as a first preset mark (such as [MASK]) to represent that the masked table item is masked; wherein, the other table items in the table training data except the mask position are also table items that do not need to be masked, and the content (or value) of these table items remains unchanged because they are not masked. In some embodiments, a table item is used to represent the position (or feature name) and content of the item in the table. In some embodiments, serialization is achieved by connecting the feature names and values ​​in the table into sentences, such as in the form of "[feature] is [Value]", where "feature" represents the feature name corresponding to the table item, "Value" represents the value corresponding to the table item, and "is" is a conjunction. For example, given a table T = {t ij}∈R n×m , where n represents the number of rows in table T, m represents the number of columns in table T, R represents the matrix corresponding to the table, and t ij It is used to represent the jth eigenvalue of the ith sample (the ith sample in this example is the ith row of table T), then F j Represents a feature column, which can convert the i-th sample into, for example, “F1 is t i1 , F2 is t i2 , …, F m Yes im ”.

[0027] In some embodiments, the table training text may be serialized first, and then the serialized text information may be modified based on at least one mask position indicated by the mask matrix (equivalent to modifying the content of the table item corresponding to the mask position in the table training data) to obtain masked text sequence data, or the table training data may be masked based on the mask matrix to obtain masked table data, and then the masked table data may be serialized to obtain masked text sequence data. The above two implementation methods will be described in detail in subsequent embodiments and will not be repeated here. Those skilled in the art should understand that the above implementation method of obtaining masked text sequence data is only an example, not a limitation of the present application, and any implementation method of obtaining corresponding masked text sequence data based on table training data and its corresponding mask matrix should be included in the scope of the present application.

[0028] S106, performing pre-training based on the masked text sequence data to obtain a general table model, wherein a training goal is to predict the content of the masked table item corresponding to the at least one masked position.

[0029] In some embodiments, a prompt text is generated based on the masked text sequence data (i.e., a pre-training task is constructed), and the prompt text is input into a large language model for training to obtain a general table model; in some embodiments, the large language model used for pre-training uses a default cross entropy loss. In some embodiments, a prompt text is generated based on the masked text sequence data and the training target. For example, given a table T∈R n×m , where n represents the number of rows in table T, m represents the number of columns in table T, R represents the matrix corresponding to the table, and the mask matrix M∈{0,1} n×m , mask the content corresponding to the mask position in table T according to the following formula: Where X is a first preset mark used to indicate the mask position in table T, T(i,j) represents the original value of the i-th row and j-th column in the table before masking, and T m (i,j) represents the value of the i-th row and j-th column after masking, M(i,j) represents the value of the i-th row and j-th column in the mask matrix, and "if" is used to determine whether the given condition is met. Based on the above formula, the element position with a value of 1 in the mask matrix also indicates the mask position in table T. After masking, the content of the table item corresponding to the mask position in table T is replaced by X, and the remaining positions remain unchanged. Then F j Represents the jth feature name, then for the i-th row T(i, :), the mask position of the i-th row can be defined as: Ф i ={k|M(i,k) = 1} Among them, Фi Represents the mask position of the i-th row, k∈Ф i , the form of "[feature] is [Value]" is used to serialize and obtain the masked text sequence data. Based on the masked text sequence data and the training target, the prompt text is generated as follows: "The question is "F1 is T m (i,1), F2 is T m (i,2),…,F m It is T m (i,m), interpolate missing values”, for all k∈Ф i The answer is "F k is T(i,k)”. In some embodiments, the pre-trained framework adopts an autoregressive architecture, which adopts appropriately labeled tabular data input (that is, tabular data input determined based on masked text sequence data) and learns to use the contextual information of unmasked values ​​to predict masked values ​​(that is, the content corresponding to the masked table item). By using an autoregressive model and a variable mask matrix, various types of table-related tasks can be integrated into a unified framework, which not only improves the performance of the general table model in tabular data tasks, but also enhances its adaptability to subsequent tasks.

[0030] According to the scheme of the embodiments of the present specification, by obtaining corresponding masked text sequence data based on table training data and its corresponding mask matrix, and pre-training based on the masked text sequence data, a general table model suitable for various table tasks can be obtained. The scheme provides a unified pre-training paradigm, that is, masking first and then predicting. The mask matrix can be used to capture the complex relationship between table features. This method can enhance the model's understanding and operation of table data, thereby improving the processing performance of various table-related tasks. In addition, by integrating different table tasks into one training target using different mask matrices, a new unified training framework is also implemented, which not only keeps the pre-training tasks consistent with downstream tasks, but also significantly improves the versatility and practicality of the general table model.

[0031] In some embodiments, the obtaining of the corresponding masked text sequence data according to the table training data and the mask matrix includes: serializing the table training data to obtain the corresponding table text training data; obtaining the corresponding masked text sequence data according to the table text training data and the mask matrix. In some embodiments, the table text training data is text data obtained by directly serializing the table training data, and the table text training data includes text data corresponding to each table item of the table training data. In some embodiments, according to the mask position indicated by the mask matrix, the text data to be masked in the table text training data corresponding to the mask position is determined, and then the text data is replaced with a first preset mark, so as to obtain the masked text sequence data corresponding to the table training data. For example, directly serializing the table T can convert the i-th row in the table into the text data "F1 is t i1 , F2 is t i2 , …, F m Yes im ”, based on the mask matrix to determine t i1 Need to cover (that is, t i1 is the text data to be masked), then the “t i1 " is replaced with the first preset mark " [MASK] ". In some embodiments, the table text training data is modified according to the table text training data, the mask matrix and the preset mask rule, and the modified text data is used as the mask text sequence data. The implementation method of serializing the table training data is the same or similar to the implementation method of the aforementioned serialization, which will not be repeated here.

[0032] In some embodiments, the serializing the table training data to obtain the corresponding table text training data further includes: if the content of the target table item in the table training data is missing, the target table item is not serialized. In some embodiments, if the content of the target table item in the table training data is missing, the target table item may not be serialized, that is, the serialized table text training data is obtained only for the non-missing table items; for example, for the i-th row of table T, since the value of the second feature column is missing, the second feature column may be skipped during the serialization process, and the serialized text "F1 is t i1 , F3 is t i3 , …, F m Yes im ”.

[0033] In some embodiments, if the content of the target table item in the table training data is missing, the content of the target table item can be set to a second preset mark in the pre-training stage, and then serialized based on this to obtain the corresponding table text training data. The second preset mark is used to indicate that the content of the table item is missing. Continuing with the above example, for the i-th row of table T, since the value of the second feature column is missing, the value of the second feature column is set to the second preset mark "[Impute]" during the serialization process, and the serialized text "F1 is t i1 , F2 is [Impute], F3 is t i3 , …, F m Yes im ". In some embodiments, if the content of the target table item in the table training data is missing, the mask position indicated by the mask matrix is ​​not the missing table item, that is, the missing table item does not need to be masked; when the mask matrix corresponding to the table training data is obtained based on the preset mask rate, the number of masks can be determined based on the product of the number of non-missing table items and the preset mask rate, and then the mask matrix is ​​randomly generated. For example, if the mask rate is 30%, 30% of the elements corresponding to the non-missing table items in the mask matrix have element values ​​of 1 (that is, mask indication information), and the element values ​​corresponding to the remaining non-missing table items and the missing table items are all 0.

[0034] In some embodiments, the step of obtaining the corresponding masked text sequence data according to the table training data and the mask matrix includes: obtaining table mask training data according to the table training data and the mask matrix; and serializing the table mask training data to obtain the corresponding masked text sequence data. In some embodiments, the table mask training data represents the table data obtained after masking the table training data based on the mask matrix. For example, for a given table T∈R n×m , where n represents the number of rows in table T, m represents the number of columns in table T, R represents the matrix corresponding to the table, and the mask matrix M∈{0,1} is used. n×m Mask at least one mask position in table T to obtain the masked table T m In some embodiments, the implementation method of serializing the table mask training data is the same as or similar to the implementation method of the aforementioned serialization, and will not be repeated here.

[0035] Figure 2 A flowchart of a method for executing a target downstream task provided in an embodiment of this specification, the method comprising the following steps: S202, obtaining a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data. The implementation of step S202 is the same or similar to the implementation of the aforementioned step S102, and will not be repeated here.

[0036] S204, according to the table training data and the mask matrix, obtain corresponding mask text sequence data, wherein the mask text sequence data includes a mask table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the mask table item is replaced with a first preset mark. The implementation method of the step S204 is the same or similar to the implementation method of the aforementioned step S104, and will not be repeated here.

[0037] S206, pre-training is performed based on the masked text sequence data to obtain a general table model, wherein the training goal is to predict the content of the masked table item corresponding to the at least one masked position. The implementation of step S206 is the same or similar to the implementation of the aforementioned step S106, and will not be repeated here.

[0038] S208, according to the task type of the downstream task, fine-tune the general table model by adjusting the mask matrix and the training target to obtain a downstream table model corresponding to the downstream task.

[0039] In some embodiments, the task type of the downstream task includes any one of a table prediction task, a table generation task, and a table missing value interpolation task. In some embodiments, the goal of the table prediction task is to predict the value of each sample target column based on existing data. In some embodiments, the goal of the table generation task is to generate table data whose distribution is similar to but not repeated as the original data; in some embodiments, the table generation task is conditionally generated based on the label column, that is, its goal is to generate table data whose distribution is similar to but not repeated as the original data based on the known label column. In some embodiments, the purpose of the table missing value interpolation task is to interpolate missing values ​​in the table.

[0040] In some embodiments, after pre-training, the general table model is fine-tuned by adjusting the mask matrix and the training target to adapt to specific downstream tasks. For example, for a table prediction task, the mask matrix is ​​adjusted to mask the target value to be predicted, and a prompt text is generated based on the adjusted mask matrix and the training target to fine-tune the model on the prediction task, thereby obtaining a downstream table model adapted to the table prediction task. Depending on the task type of the downstream task, the specific implementation method of fine-tuning the general table model by adjusting the mask matrix and the training target will be described in detail in subsequent embodiments and will not be repeated here.

[0041] S210, for the target downstream task, according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data, by using the downstream table model corresponding to the target downstream task, the target downstream task is executed, wherein the mask position corresponding to the target mask matrix is ​​determined according to the target downstream task. In some embodiments, the input of the downstream table model is the input information corresponding to the target table data (for example, the corresponding mask text sequence data can be first obtained based on the target table data and the target mask matrix, and then the corresponding prompt question is generated as the input information in combination with the training target), and the output of the downstream table model is the execution result of the target downstream task, such as the value that needs to be predicted, generated or interpolated. In some embodiments, the target downstream task is a table prediction task, and the mask position indicated by the target mask matrix corresponds to the target column that needs to be predicted in the target table data. Based on the target mask matrix, the target column that needs to be predicted in the target table data can be masked to output the predicted value of the target column through the corresponding downstream table model. In some embodiments, the target downstream task is a conditional table generation task based on the label column, and the mask position indicated by the target mask matrix corresponds to other values ​​in the target table data except the label column. Based on the target mask matrix, other values ​​in the target table data except the label column that need to be generated can be masked, so as to output the values ​​of other columns generated based on the label column through the corresponding downstream table model. In some embodiments, the target downstream task is an unconditional table generation task, and the mask position indicated by the target mask matrix corresponds to all values ​​in the target table data. Based on the target mask matrix, all values ​​can be masked to generate table data with a distribution similar to the original data but without duplication. In some embodiments, the target downstream task is a table missing value interpolation task, and the mask position indicated by the target mask matrix corresponds to the missing values ​​in the target table data (that is, the positions that need to be interpolated). Based on the target mask matrix, the missing values ​​in the target table data can be masked, so as to output the predicted values ​​for the missing values ​​through the corresponding downstream table model, so as to complete the interpolation of the missing values.

[0042] The embodiments of the present specification can fine-tune the general table model according to the task type of the downstream task by adjusting the mask matrix and the training target to obtain a downstream table model corresponding to the downstream task. This new pre-training paradigm enhances the capabilities of the downstream table model for specific downstream tasks and improves the performance and efficiency of the data analysis and decision-making process.

[0043] In some embodiments, the task type of the downstream task includes a table prediction task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item corresponding to the target column to be predicted in the downstream table training data; adjusting the training target to predict the content of the table item corresponding to the target column in the downstream table training data, and fine-tuning the general table model based on the downstream table training data. In some embodiments, for the table prediction task, the training target is to predict the value of the target column of each sample, based on the training target, it is necessary to mask the value of the target column to be predicted, and fine-tune the general table model on the prediction task to adapt it to the table prediction task. In some embodiments, the downstream table training data may be derived in whole or in part from the above-mentioned table training data, or may be table training data collected outside the above-mentioned table training data.

[0044] As an example, for the downstream table prediction task, the adjusted mask matrix is ​​as follows. Based on the mask matrix and the training target of the table prediction task, the model can be fine-tuned on the prediction task (that is, generating prompt text suitable for the table prediction task): Among them, M(i,j) represents the value of the i-th row and j-th column in the mask matrix M, W represents the condition that (i,j) is the target value, (i,j) is the target value, that is, it corresponds to the known label column, and "else" represents other situations except W; in this example, if (i,j) is the target value, that is, for the target column to be predicted, the value of M(i,j) is set to 1, otherwise it is set to 0, thereby masking the value of the target column to be predicted. Based on the adjusted mask matrix and the training target, the corresponding prompt text can be generated (the implementation method of generating the prompt text here is similar to the implementation method of generating the prompt text in the aforementioned pre-training stage, which will not be repeated here), so as to fine-tune the model on the prediction task.

[0045] In some embodiments, the task type of the downstream task includes a table generation task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item to be generated in the downstream table training data; adjusting the training target to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and fine-tuning the general table model based on the downstream table training data. In some embodiments, for the table generation task, the training target is to generate table data whose distribution is similar to but non-repetitive to the original data. In some embodiments, the table generation task includes an unconditional table generation task and a conditional table generation task based on a label column; for the unconditional table generation task, the values ​​of all elements in the adjusted mask matrix are mask indication information (e.g., 1), and all values ​​in the downstream table training data can be set to a first preset mark (e.g., [MASK] character) based on the adjusted mask matrix, that is, all values ​​are masked; the fine-tuning method for the conditional table generation task based on the label column will be described in detail in subsequent embodiments, and will not be repeated here.

[0046] In some embodiments, the table generation task is a conditional table generation task based on the label column, and the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be generated in the downstream table training data except the label column; the training target is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data, including: adjusting the training target to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data based on the label column, and fine-tuning the general table model based on the downstream table training data. In some embodiments, for the conditional table generation task based on the label column, except for the value of the element corresponding to the label column in the adjusted mask matrix, which is non-mask indication information (for example, 0), the values ​​of the remaining elements are all mask indication information (for example, 1). Based on the adjusted mask matrix, other values ​​in the downstream table training data except the label column can be set to the first preset mark (for example, [MASK] character), that is, the values ​​other than the label column are masked. As an example, for the table generation task based on the label column, the adjusted mask matrix can be expressed as: In this example, W represents the condition that (i, j) is the target value. (i, j) is the target value, which means it corresponds to a known label column. "else" represents other situations except W, thereby masking values ​​outside the label column.

[0047] In some embodiments, the task type of the downstream task includes a table missing value interpolation task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: for downstream table training data containing missing table items, according to a preset missing rate and the position of the missing table items in the downstream table training data, adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item to be interpolated in the downstream table training data, and the table item to be interpolated is not the missing table item; adjusting the training target to interpolate the content of the table item to be interpolated in the downstream table training data, and fine-tuning the general table model based on the downstream table training data. In some embodiments, for the table missing value interpolation task, the training target is to interpolate the missing values ​​in the table. In some embodiments, for the task of interpolating missing values ​​in a table, a method of first masking and then predicting is used to process values ​​other than missing values ​​(i.e., non-missing values); for tables without missing values, a mask matrix can be directly constructed based on the missing rate. For example, if the missing rate is 30%, a mask matrix corresponding to the table is constructed, wherein 30% of the elements in the mask matrix have mask indication information (e.g., 1), and the values ​​of the remaining elements are non-mask indication information (e.g., 0); for tables with original missing values, the non-missing values ​​in the table need to be masked according to the missing rate. For example, if the missing rate is 30%, the mask position indicated by the constructed mask matrix corresponds to at least one non-missing value in the table, and the ratio of the number of mask positions to the number of non-missing table items in the table is equal to the missing rate.

[0048] In some embodiments, the content of the missing table items in the downstream table training data is set to a second preset mark. As an example, given a table T, the missing table items in the table T are represented by a binary mask matrix M ori ∈{0,1} n×m Indicates that, if M ori (i,j)=0 means the value of row i and column j is not missing. ori (i,j)=1 means that the value of the i-th row and j-th column is missing (that is, it corresponds to a missing table entry); a new mask matrix M∈{0,1} is randomly selected according to the preset missing rate n×m , during the training process, the table T is masked based on the following formula: in," " represents the intersection operation, T(i,j) represents the original value of the i-th row and j-th column in the table T before masking, T m(i,j) represents the value of the i-th row and j-th column in the masked table, Y is the second preset mark for indicating missing table items, and X is the first preset mark for indicating masked items. Thus, the original missing values ​​in table T are filled with Y, and the values ​​of the table items that need to be masked but are not missing are replaced with X. Then, each row in the masked table can be serialized into text data, that is, the corresponding masked text sequence data is obtained for fine-tuning the model. It should be noted that in the above example, since the masked items are the contents of the table items that are not missing in the table and the corresponding elements in the mask matrix are the mask indication information, for the i-th row T (i, :) of table T, the mask position of the i-th row is defined as: Ф i ={k|M(i,k) = 1 ∧M ori (i,k)=0} Among them, Ф i Represents the mask position of the i-th row, k∈Ф i .

[0049] In some embodiments, after adjusting the mask matrix and the training target according to the task type of the downstream task, the mask text sequence data corresponding to the downstream table training data is obtained based on the adjusted mask matrix, and then a prompt text is generated according to the mask text sequence data and the adjusted training target, and the prompt text is input into the large language model for training to adapt to the corresponding task type; wherein, the implementation method of generating the prompt text is similar to the aforementioned implementation method of generating the prompt text in the pre-training stage, and will not be repeated here. It should be noted that in some embodiments, for downstream table training data with missing table items, the value of the missing table item can be set to a second preset mark before serialization, or not participate in serialization.

[0050] It should be noted that the method of the embodiments of this specification can be used in all application scenarios related to table tasks, and this application does not impose any restrictions on this. For example, in a risk control scenario, to address the problem of sparse black samples, the method of the embodiments of this specification can be used to process related downstream tasks, which can play a role in data enhancement.

[0051] Figure 3 The structural diagram of a device for training a table model provided in an embodiment of this specification (hereinafter referred to as "table model training device 1") can be implemented as all or part of an electronic device through software, hardware or a combination of both. According to some embodiments, the table model training device 1 includes a first acquisition module 11, a second acquisition module 12 and a pre-training module 13.

[0052] A first obtaining module 11 is used to obtain a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data; A second obtaining module 12 is used to obtain corresponding mask text sequence data according to the table training data and the mask matrix, wherein the mask text sequence data includes a mask table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the mask table item is replaced with a first preset mark; The pre-training module 13 is used to perform pre-training based on the masked text sequence data to obtain a general table model, wherein the training goal is to predict the content of the masked table item corresponding to the at least one masked position.

[0053] In some embodiments, the table model training device 1 is also used to: fine-tune the general table model according to the task type of the downstream task by adjusting the mask matrix and the training target to obtain a downstream table model corresponding to the downstream task; for a target downstream task, perform the target downstream task according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data by using the downstream table model corresponding to the target downstream task, wherein the mask position corresponding to the target mask matrix is ​​determined according to the target downstream task.

[0054] In some embodiments, the second obtaining module 12 is used to: serialize the table training data to obtain corresponding table text training data; and obtain corresponding mask text sequence data according to the table text training data and the mask matrix.

[0055] In some embodiments, the serializing the table training data to obtain the corresponding table text training data further includes: if the content of the target table item in the table training data is missing, making the target table item not participate in the serialization.

[0056] In some embodiments, the second obtaining module 12 is used to: obtain table mask training data according to the table training data and the mask matrix; and serialize the table mask training data to obtain corresponding mask text sequence data.

[0057] In some embodiments, the task type of the downstream task includes any one of the following: a table prediction task; a table generation task; a table missing value interpolation task.

[0058] In some embodiments, the task type of the downstream task includes a table prediction task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item corresponding to the target column to be predicted in the downstream table training data; adjusting the training target to predict the content of the table item corresponding to the target column in the downstream table training data, and fine-tuning the general table model based on the downstream table training data.

[0059] In some embodiments, the task type of the downstream task includes a table generation task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be generated in the downstream table training data; adjusting the training target to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and fine-tuning the general table model based on the downstream table training data.

[0060] In some embodiments, the table generation task is a conditional table generation task based on a label column, and the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be generated in the downstream table training data except the label column; the training goal is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data, including: adjusting the training goal to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data based on the label column, and fine-tuning the general table model based on the downstream table training data.

[0061] In some embodiments, the task type of the downstream task includes a table missing value interpolation task; wherein, according to the task type of the downstream task, the general table model is fine-tuned by adjusting the mask matrix and the training target, including: for downstream table training data containing missing table items, according to a preset missing rate and the position of the missing table items in the downstream table training data, adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be interpolated in the downstream table training data, and the table item that needs to be interpolated is not the missing table item; adjusting the training target to interpolate the content of the table item that needs to be interpolated in the downstream table training data, and fine-tuning the general table model based on the downstream table training data.

[0062] In some embodiments, the content of the missing table entry in the downstream table training data is set to a second preset flag.

[0063] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0064] The embodiment of the present specification also provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor to execute the method of the embodiment of the present specification.

[0065] The embodiments of the present specification also provide a computer program product, which stores at least one instruction, and the at least one instruction is loaded by the processor to execute the method of the embodiments of the present specification.

[0066] The embodiments of this specification also provide Figure 4 The structural diagram of the electronic device shown in FIG. Figure 4 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above method.

[0067] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0068] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0073] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0074] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0075] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for training a table model, comprising: Obtaining a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data; Obtaining corresponding mask text sequence data according to the table training data and the mask matrix, wherein the mask text sequence data includes a mask table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the mask table item is replaced with a first preset mark; Pre-training is performed based on the masked text sequence data to obtain a general table model, wherein a training goal is to predict the content of the masked table item corresponding to the at least one masked position.

2. The method according to claim 1, further comprising: According to the task type of the downstream task, fine-tune the general table model by adjusting the mask matrix and the training target to obtain a downstream table model corresponding to the downstream task; For a target downstream task, the target downstream task is executed according to the target table data corresponding to the target downstream task and the target mask matrix corresponding to the target table data, by using the downstream table model corresponding to the target downstream task, wherein the mask position corresponding to the target mask matrix is ​​determined based on the target downstream task.

3. The method according to claim 1, wherein obtaining corresponding masked text sequence data according to the table training data and the mask matrix comprises: Serializing the table training data to obtain corresponding table text training data; According to the tabular text training data and the mask matrix, corresponding masked text sequence data is obtained.

4. The method according to claim 3, wherein the step of serializing the table training data to obtain corresponding table text training data further comprises: If the content of the target table item in the table training data is missing, the target table item does not participate in serialization.

5. The method according to claim 1, wherein obtaining corresponding masked text sequence data according to the table training data and the mask matrix comprises: Obtaining table mask training data according to the table training data and the mask matrix; The table mask training data is serialized to obtain corresponding mask text sequence data.

6. According to the method of claim 2, the task type of the downstream task includes any one of the following: Table prediction tasks; Table generation tasks; Table missing value imputation task.

7. The method according to claim 6, wherein the task type of the downstream task comprises a table prediction task; in, The method of fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: Adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item corresponding to the target column to be predicted in the downstream table training data; The training objective is adjusted to predict the content of the table item corresponding to the target column in the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

8. The method according to claim 6, wherein the task type of the downstream task comprises a table generation task; in, The method of fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: Adjusting the mask matrix, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table entry to be generated in the downstream table training data; The training objective is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

9. The method according to claim 8, wherein the table generation task is a conditional table generation task based on a label column, and the mask position indicated by the adjusted mask matrix is ​​the position of the table item to be generated in the downstream table training data except for the label column; The training objective is adjusted to generate table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data, including: The training objective is adjusted to generate, based on the label column, table data whose corresponding distribution is similar to and non-repetitive to the downstream table training data, and the general table model is fine-tuned based on the downstream table training data.

10. The method according to claim 6, wherein the task type of the downstream task comprises a table missing value interpolation task; in, The method of fine-tuning the general table model by adjusting the mask matrix and the training target according to the task type of the downstream task includes: For downstream table training data containing missing table items, adjusting the mask matrix according to a preset missing rate and the position of the missing table items in the downstream table training data, wherein the mask position indicated by the adjusted mask matrix is ​​the position of the table item that needs to be interpolated in the downstream table training data, and the table item that needs to be interpolated is not the missing table item; The training objective is adjusted to interpolate the contents of table items that need to be interpolated in the downstream table training data, and the general table model is fine-tuned based on the downstream table training data. 11 . The method according to claim 10 , wherein the content of the missing table entry in the downstream table training data is set as a second preset mark.

12. A device for training a table model, comprising: A first obtaining module, used to obtain a mask matrix corresponding to the table training data, wherein the mask matrix is ​​used to indicate at least one mask position corresponding to the table training data; a second obtaining module, configured to obtain corresponding mask text sequence data according to the table training data and the mask matrix, wherein the mask text sequence data includes a mask table item after masking the mask position in the table training data, and other table items in the table training data except the mask position, and the content of the mask table item is replaced with a first preset mark; A pre-training module is used to perform pre-training based on the masked text sequence data to obtain a general table model, wherein the training goal is to predict the content of the masked table item corresponding to the at least one masked position.

13. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

14. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the steps of the method as claimed in any one of claims 1 to 11.

15. A computer program product having at least one instruction stored thereon, characterized in that: When the at least one instruction is executed by the processor, the steps of the method described in any one of claims 1 to 11 are implemented.

Images