A language model based cross-table multi-task pre-training method and device

Abstract

The application discloses a cross-table multi-task pre-training method and device based on a language model, comprising the following steps: obtaining table data with labels; performing semantic correspondence on discrete feature values; performing normalization processing on continuous feature values, and then performing discretization processing on the continuous feature values by adopting a C4.5 algorithm of a decision tree, so that the continuous feature values are divided into multiple non-equidistant intervals, and the continuous feature values are divided into corresponding intervals; each interval is identified as a token and placed in a vocabulary of a language model; performing regularization processing on the intervals corresponding to the continuous feature values to construct a regularization loss function capable of preserving the size relationship of the intervals; and inputting the semantic text corresponding to the table data and the tokens of the intervals into a RoBERTa language model, and pre-training classification tasks and regression tasks based on the labels and the regularization loss function, so that the prediction effect of the model is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning on tabular data, and particularly relates to a cross-table multi-task pre-training method and device based on a language model. BACKGROUND

[0002] In machine learning, a label refers to the category or numerical value of an instance. A tabular data usually contains two parts: features and labels. Features are used to describe the properties of instances, while labels are used to train and evaluate machine learning models. The goal of supervised learning algorithms is to find a function or model that can automatically learn rules and patterns from input data and map new input data to the correct label. In order to train this model, a set of training data with labels is needed to adjust the parameters of the model so that it can accurately predict the label to the maximum extent.

[0003] Although deep learning has made some breakthroughs in computer vision, natural language processing and speech recognition, it is still a research field that can be further explored on tabular data. In 2017, Google proposed the Transformer model, which made deep learning enter the era of large models, and it has been widely applied in natural language processing and computer vision. Recently, many deep learning architectures based on Transformer have been applied to tabular data and have achieved label prediction results comparable to traditional machine learning models.

[0004] Structured data is a very common type of data storage, including data in databases or Excel tables. In recent years, using deep learning methods to process tabular data has become a new research direction. Tabular learning methods based on deep learning have achieved results comparable to GBDT, a machine learning method. At the same time, in the field of natural language processing, some large-scale pre-training models have emerged. In tabular data, the feature names are usually described using natural language. Combined with language models, the relationships between multiple features can be learned. In the content of the table, there are also many descriptions using natural language, which is a problem that traditional machine learning methods cannot solve. However, combined with natural language processing methods, these feature contents can be well explained, and ultimately benefit the prediction of tabular labels.

[0005] Generally, there are three types of features of table data, numerical type, category type and text type. Traditional machine learning models understand numerical type features well, but some cannot handle category type features. For example, XGBoost does not support category type features. Before training the model, preprocessing such as One-Hot Encoding is required before label prediction, so a model capable of understanding text semantic information is needed to solve these problems. In the field of natural language processing, models such as BERT and RoBERTa have been produced, which can learn contextual information by Masked Language Model pre-training strategy to predict the corresponding Masked Token. This feature can be used in table label prediction field. SUMMARY

[0006] In view of the above, the purpose of the present application is to provide a language model based cross-table multi-task pre-training method and device, which solves the problem of inaccurate table label prediction containing text features in practical application.

[0007] In order to achieve the above purpose, the language model based cross-table multi-task pre-training method provided by the present application comprises the following steps:

[0008] Obtain table data with labels for classification tasks and regression tasks, the table data including discrete feature values and continuous feature values, and filter missing values from the table data;

[0009] Perform semantic correspondence on the discrete feature values to convert the discrete feature values into corresponding semantic text;

[0010] After normalizing the continuous feature values, use the C4.5 algorithm of decision tree to discretize the continuous feature values, divide the continuous feature values into multiple non-equidistant intervals, and divide these continuous feature values into corresponding intervals, and each interval name is identified as a token and put into the vocabulary of the language model;

[0011] Regularize the intervals corresponding to the continuous feature values to construct a regularization loss function capable of preserving the numerical size relationship of the intervals;

[0012] Input the semantic text and the token of the interval corresponding to the table data into the RoBERTa language model, and pre-train the classification task and the regression task based on the label and the regularization loss function;

[0013] Fine-tune the pre-trained RoBERTa language model based on the downstream task, and use the fine-tuned RoBERTa language model to predict the downstream task.

[0014] Preferably, the missing value filtering of the table data comprises:

[0015] If the feature names in the table data exceeding the first threshold value do not contain semantic information, the table data is deleted;

[0016] If the number of features in the table data exceeds the second threshold value or the text in the features exceeds the third threshold value, the table data is deleted;

[0017] If the missing values of the features in the table data exceed the fourth threshold value, the table data is deleted;

[0018] If there are less than the fifth threshold value of features with missing values in the table data, the features with missing values are judged to be continuous type features, and if they are judged to be discrete type features, the missing values are treated as text;

[0019] If the feature names in the table data are composed of multiple words and use camel case, Pascal case, underscore case or underscore case, regular expression is used to detect and split each word.

[0020] Preferably, the discrete type features include numerical type features and category type features, and for the category type features, if their original representation is numerical and there is information explaining the correspondence between the numerical value and the category information, the category information corresponding to the numerical value is taken as semantic text when semantic correspondence is performed, and the discrete type features presented in the form of semantic text do not need to be subjected to semantic correspondence.

[0021] Preferably, the C4.5 algorithm of the decision tree is used to discretize the continuous type feature numerical value features, comprising:

[0022] Defining the basic case of the C4.5 algorithm: if the features of a table data set belong to the same category, a leaf node is directly created to represent the selection of the category; if all features do not provide any information gain, the C4.5 uses the expected value of the class to create a decision node at the upper layer of the tree; if a feature that has not been seen before is encountered, the C4.5 uses the expected value to create a decision node at the upper layer of the tree;

[0023] Decision process: 1) check the above basic cases; 2) for each feature a, calculate the information gain of the division a; 3) record a_best as the feature with the highest information gain; 4) create a decision node that divides a_best; 5) use the divided features to create child nodes of the current decision node, and recursively process on these child nodes.

[0024] Preferably, the intervals corresponding to the continuous type feature numerical values are subjected to regularization processing to construct a regularization loss function that can preserve the numerical size relationship of the intervals, comprising:

[0025] Randomly select token in embedding vectors of RoBERTa language model in three intervals, first select token of middle size, then make original values close to each other, and values far from each other, expressed as formula:

[0026] L=max(d(a,p)-d(a,n)+margin,0)

[0027] Wherein, L is a regularization loss function, a represents token of middle size, p represents token close to a and belonging to the same category as a, n represents token far from a and belonging to different categories as a, d(·) represents distance between embedding vectors, margin represents a constant greater than 0, max(·) represents maximum operation, using the regularization loss function L, the size relationship between embedding vectors of token in multiple intervals is clear.

[0028] Preferably, in the process of inputting semantic text corresponding to table data and interval token into the RoBERTa language model for pre-training, when performing multi-head self-attention operation in the model, discard the position label between features, and use self-attention within the feature, generate a [CLS] token after self-attention operation of the feature name and value of each feature, to represent the feature and its value, and finally perform multi-head self-attention operation on the [CLS] token equal to the number of features in the language model to obtain the final representation result.

[0029] Preferably, in the pre-training process, the regularization loss function is combined with the task loss as the total loss to update the parameters of the RoBERTa language model.

[0030] To achieve the above-mentioned purposes, the embodiment also provides a cross-table multi-task pre-training device based on a language model, comprising a table data collection and preprocessing module, a discrete feature value semantic correspondence module, a continuous feature value discretization processing module, a continuous value feature regularization processing module, a cross-table multi-task pre-training module based on a language model, a downstream table fine-tuning and result testing module;

[0031] The table data collection and preprocessing module is used to obtain table data with labels for classification tasks and regression tasks, and the table data includes discrete feature values and continuous feature values, and the table data is subjected to missing value filtering.

[0032] The discrete feature value semantic correspondence module is used to perform semantic correspondence on the discrete feature values, and convert the discrete feature values into corresponding semantic text.

[0033] The continuous numerical feature discretization processing module is configured to normalize continuous numerical features, and then discretize the continuous numerical features by using a C4.5 algorithm of a decision tree, divide the continuous numerical features into multiple non-equidistant intervals, and put the continuous numerical features into corresponding intervals, and each interval is identified as a token and put into a vocabulary of a language model.

[0034] The continuous numerical feature regularization processing module is configured to perform regularization processing on intervals corresponding to continuous numerical features to construct a regularization loss function capable of preserving the numerical size relationship of the intervals.

[0035] The cross-table multi-task pre-training module based on a language model is configured to input semantic text corresponding to table data and tokens of intervals into a RoBERTa language model, and pre-train classification tasks and regression tasks based on labels and a regularization loss function.

[0036] The downstream table fine-tuning and result testing module is configured to fine-tune the pre-trained RoBERTa language model based on downstream tasks, and predict downstream tasks by using the fine-tuned RoBERTa language model.

[0037] To achieve the above-mentioned purposes, the embodiments further provide a cross-table multi-task pre-training device based on a language model, which comprises a memory configured to store computer executable instructions and data used or produced when the computer executable instructions are executed, and a processor communicatively connected with the memory and configured to execute the computer executable instructions stored in the memory, wherein the computer executable instructions are executed to implement the above-mentioned cross-table multi-task pre-training method based on a language model.

[0038] To achieve the above-mentioned purposes, the embodiments further provide a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the above-mentioned cross-table multi-task pre-training method based on a language model.

[0039] Compared with the prior art, the present application has at least the following beneficial effects:

[0040] 1) A large number of labeled table data sets are collected and preprocessed, and some tables with rich semantic information of feature names are screened out, which can be used for training in the field of table deep learning combined with large-scale language models.

[0041] 2) In the case of text and numerical values in the table, some ordinary table deep learning models are better than the present application. Because the numerical interval operation of continuous numerical feature discretization processing and the regularization of continuous numerical features play a good role.

[0042] 3) The same model can be used for pre-training across different domains of tables, which has certain advantages compared to previous models that can only be pre-trained in a specific domain, and can be applied to binary classification, multi-classification and regression tasks. For regression prediction tasks, it is also better than some traditional machine learning models and previous table deep learning models. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0044] Figure 1 The flowchart of the language model-based cross-table multi-task pre-training method in the embodiment of the present application;

[0045] Figure 2 The structural schematic diagram of the language model-based cross-table multi-task pre-training device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application more clear and explicit, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0047] As shown in Figure 1 The language model-based cross-table multi-task pre-training method provided by the embodiment includes the following steps:

[0048] S110, collecting and preprocessing table data.

[0049] In the embodiment, a large amount of labeled table data is collected. These table data sets contain three types of binary classification, multi-classification and regression. Because supervised learning on table data is mainly applied to classification problems and regression problems, and classification problems can be further divided into binary classification problems and multi-classification problems. The table data includes discrete feature values and continuous feature values.

[0050] A large amount of labeled table data is collected from Kaggle, OpenML and UCI machine learning data websites, about 1500 tables, including binary classification, multi-classification and regression tasks. At the same time, the description information of each table and the specific description of each feature in the table are collected. If it is a discrete feature represented by a numerical value, the corresponding relationship between the numerical value and the text is also collected.

[0051] In the embodiment, the table data is also preprocessed, some table data with many missing values in the table data set is processed, some tables with ambiguous column names are removed, so that the column names in the tables can retain certain semantic information, and the language model can learn the semantics of the features and the mutual relationship between the features. Specifically, the preprocessing process includes:

[0052] (1) If more than a first threshold of feature names in the table data do not contain semantic information, the table data is deleted, that is, if most of the feature names in the table do not contain some semantic information, they are some uncommon and meaningless words, the table is discarded, because this is not conducive to the understanding ability of the language model.

[0053] (2) If the number of features in the table data exceeds a second threshold or the text in the features exceeds a third threshold, the table data is deleted, that is, if the feature is too long or the text in the feature is too long, it may cause too many tokens to be generated, so the table needs to be discarded, because it may cause the video memory to be too large and the training process to overflow.

[0054] (3) If the missing values of the features in the table data exceed a fourth threshold, the table data is deleted, that is, for the features in the table, if the missing values exceed a certain range, the table is discarded.

[0055] (4) If less than a fifth threshold of features in the table data have missing values, the features with missing values are judged to be continuous features, and the features are treated as continuous features, if they are judged to be discrete features, the missing values are treated as text, that is, if some features have a small amount of missing values, it is judged whether these features are continuous numerical features, if so, it is still treated as continuous values, if they are category features, the missing values Nan are treated as text.

[0056] (5) If the feature name in the table data is composed of multiple words and uses camel case, Pascal case, underscore case or underscore case, it is detected by regular expression and each word is split to facilitate the understanding of the language model.

[0057] S120, semantic corresponding processing of discrete feature values.

[0058] In the embodiment, the discrete feature values are semantically corresponding, and the discrete feature values are converted into corresponding semantic text. The discrete features include numerical features and categorical features. For the categorical features, if the original representation is numerical and the information has explained the corresponding relationship between the numerical and categorical information, the corresponding categorical information of the numerical value is replaced into the corresponding table value as the semantic text in the semantic correspondence. For example, the name of a certain feature is color, and the corresponding relationship is 1: red, 2: green, and 3: blue. The corresponding numerical value is converted into a feature text and stored in the table, because in the language model, it can well understand the text information, but the understanding of the numerical value is not very accurate. For the text representation of the discrete feature, it is not necessary to process, and the text can be retained. Another advantage of using the language model is that semantically similar words can be easily identified, so that the model can have better semantic understanding ability and generalization ability between different data sets.

[0059] In S130, the continuous feature values are discretized.

[0060] In the embodiment, first, the continuous feature values are normalized. For each continuous feature value, a certain normalization processing is used, and experiments show that this processing is beneficial to the experimental results. After the normalization processing, the interval operation is performed, and the effect is improved more obviously.

[0061] Then, the C4.5 algorithm of the decision tree is used to discretize the continuous feature values, and the continuous feature values are divided into multiple unequal intervals, and the continuous feature values are divided into corresponding intervals. Each interval name is identified as a token and placed in the vocabulary of the language model, so that the language model can understand the continuous numerical features.

[0062] Specifically, the continuous feature values are divided into 256 unequal intervals, and the sizes of the intervals are unequal. However, by dividing the intervals, the feature gain ratio of the entire decision tree can be improved. Then each interval is regarded as a token, and these tokens are added to some units reserved in the original vocabulary of the language model RoBERTa, so that the original numerical values can be understood and distinguished by the language model. In the pre-training process, a certain regularization strategy is used to ensure that the size relationship of the numerical features can be preserved, which is very important for numerical features.

[0063] Definition of the base case of the C4.5 algorithm: if all features of a table data set belong to the same category, a leaf node is created directly, indicating the selection of the category; if all features do not provide any information gain, C4.5 uses the expected value of the class to create a decision node at the upper layer of the tree; if a feature that has not been seen before is encountered, C4.5 uses the expected value to create a decision node at the upper layer of the tree;

[0064] Decision process: 1) check the above base case; 2) for each feature a, calculate the information gain of the division a; 3) record a_best as the feature with the highest information gain; 4) create a decision node that divides a_best; 5) use the divided features to create child nodes of the current decision node, and recursively process on these child nodes.

[0065] The C4.5 algorithm, like the ID3 algorithm, uses information entropy to construct a decision tree from a training table data set. The training table data is a set of samples S = s1, s2,... that have been classified. Each sample s i is composed of a p-dimensional vector (x 1,i , x 2,i ,..., x p,i ), where x j represents the attribute value or feature of the sample, and of course includes the category of the sample s i . At each node of the tree, the C4.5 algorithm selects the attribute of the data that most effectively divides its sample set into subsets that are concentrated in one class or another. The division criterion is the normalized information gain, i.e. the difference in entropy. The attribute with the maximum information gain is selected for decision-making, and then the subsets after division are recursively processed.

[0066] S140, continuous feature value normalization processing.

[0067] In the embodiment, the intervals corresponding to the continuous feature values are normalized to construct a normalized loss function that can preserve the numerical size relationship of the intervals. Specifically, the size of the feature value is normalized, because if the numerical feature is directly input into the language model, the numerical feature will lose its size relationship in the numerical value and only exist as a token. In order to preserve the numerical size relationship of the intervals, a normalized loss function Triplet Loss is used, which randomly selects the embedding vectors of three interval tokens, first selects the token of the middle size, and then makes the tokens with similar original values close to each other and the tokens with distant values far away from each other. After this normalization processing, the size relationship between the embedding vectors of the tokens with numerical values divided into multiple intervals is very significant. The formula is:

[0068] L = max (d (a, p) - d (a, n) + margin, 0)

[0069] Triplet loss is a loss function of deep learning, which is mainly used to train samples with small differences, such as human faces; secondly, in the task of obtaining sample embedding, triplet loss is also often used, such as text and picture embedding. The input is a triplet, including anchor example, positive example and negative example. By optimizing the distance between anchor example and positive example to be smaller than the distance between anchor example and negative example, the similarity calculation between samples is realized. a: anchor, anchor example; p: positive, feature of the same category as a; n: negative, feature of different category from a; margin is a constant greater than 0. The final optimization goal is to shorten the distance between a and p, and lengthen the distance between a and n. The regularization processing added here is to judge the size relationship of two numerical features, so that the method based on language model can well preserve the relative size information of some numerical features while understanding the text features.

[0070] S150, cross-table multi-task pre-training based on language model.

[0071] In the embodiment, the semantic text and interval token corresponding to the table data are input into the RoBERTa language model, and the pre-training of classification task and regression task is carried out based on the label combined with the regularization loss function. The pre-training parameters use the default parameters of RoBERTa. After a period of pre-training on the binary classification, multi-classification and regression data sets, the corresponding pre-trained model can be obtained.

[0072] In the pre-training process, self-attention mechanism is used. Because the features of the table are usually independent of the order, randomly arranging the feature order of the table has no effect on the final prediction result, so the position label between the features is discarded when performing multi-head self-attention operation in the model; at the same time, because if the feature is very long and includes many tokens, it will take a long time to calculate the final self-attention result, so the self-attention within the feature is used to generate a [CLS] token after the self-attention operation of the feature name and the value of each feature, which is used to represent the feature and its value. Finally, the [CLS] token equal to the number of features is subjected to multi-head self-attention operation in the language model to obtain the final representation result; the regularization loss function is combined with the task loss as the total loss to update the parameters of the RoBERTa language model. This method can greatly preserve the relationship between the feature and the value, while reducing the information interference of other feature names on the feature value, and speed up the operation speed of the model.

[0073] Specifically, RoBERTa language model is used, and Prompt is used to make binary classification prediction on label when facing binary classification task. The prediction result is false or true according to the label is_label. About 1000 binary classification and regression data sets with more than 10000 rows are used for pre-training, and the feature number of most tables is less than 32. The model converges well. The network architecture of BERT uses the multi-layer Transformer structure proposed in Attention is all you need. Its biggest feature is to abandon the traditional RNN and CNN, and convert the distance between two words at any position to 1 through the Attention mechanism, effectively solving the long-term dependence problem in NLP. In terms of model structure, compared with BERT, RoBERTa language model is more like a further exploration of BERT in pre-training. It improves many pre-training strategies of BERT, and the results show that the original BERT may not be trained enough and has not fully learned the language knowledge in the training data.

[0074] Since many features in table data have no sequence, in order to avoid using position encoding Position Embedding information in RoBERTa language model, position encoding Position Embedding information is cancelled, and token encoding Token Embedding information is used directly.

[0075] The pre-training experiment is carried out on 4 Nvidia A100 GPUs, the batch size is set to 384, the learning rate is set to 3e-5, the warm up is set every 6000 epochs, 30 epochs are trained, 5% of the data is selected as the validation set every epoch, and the number of validation times is set to 5 every epoch.

[0076] The loss function of model training is divided into three categories according to different table label prediction tasks, which are cross entropy loss, mean square error loss (MSE), and the upper limit of the number of iterations is set to 20000 times. One batch of data is input each time, the loss is calculated and back propagated, the model parameters are updated, and the training is completed until the iteration is completed. The model with the best verification effect is saved during the iteration process. If the classification effect of the verification set is continuously decreasing for 16 times, the current training is exited, and finally a new model is obtained. By modifying the hyperparameters, including learning rate (learning rate, lr) and learning rate reduction speed (K% reduction every N rounds), the loss value, accuracy, recall rate and other indicators on the verification set are optimized, so that the generalization performance of the model is better.

[0077] At the same time, in the pre-training process, the early stopping strategy is also used. The number of training rounds of the neural network is a very important problem. If the number of training rounds is too small, it will be under-fitted. If the number of training rounds is too large, over-fitting phenomenon will occur. The role of early stopping is to stop training when the performance of the model on the verification set does not increase within a certain number of training rounds, so as to achieve the purpose of sufficient training and avoid the occurrence of over-fitting phenomenon.

[0078] S160, downstream table fine-tuning and result testing.

[0079] In the embodiment, the pre-trained RoBERTa language model is fine-tuned based on the downstream task, and the fine-tuned RoBERTa language model is used for downstream task prediction.

[0080] Specifically, each table is divided into a training set, a verification set and a test set according to the proportions of 64%, 16% and 20%. The training set and the verification set are fine-tuned and the hyperparameters are searched to obtain a better model. Then the test set is tested to obtain the corresponding result score. The data set is fine-tuned in binary classification, multi-classification or regression task. After achieving a certain effect, it can be used for table label prediction to obtain corresponding loss value, accuracy, recall rate and other indicators. The test results show that the table pre-training method based on the language model is very effective. Compared with some machine learning or deep learning methods, the pre-trained language model can obtain very good results on the three tasks, especially for tables with more discrete features or large total Shapley value of discrete features.

[0081] In the embodiment, on the binary classification task, the loss function L = L clswhere cls represents a classification task, L cls The classification loss is constructed according to the prediction result of the classification task table and the true value label, and the cross-entropy loss is used, and the specific formula is as follows:

[0082] L cls = -log(S(X y ))

[0083]

[0084] where S represents a Softmax normalization function, the purpose of which is to convert the output result into a probability value between 0 and 1, and the exp() function is used to perform exponential transformation on X y to convert them into exponential form. And the sum of the probability values of all categories is 1. X represents the binary vector predicted by the classification task table, y represents the corresponding true label, and the result is false or true using is_label when using the prediction head of Prompt, and X y represents the value of the yth element of the vector X. The loss function takes the minimum value when X and y completely correspond, so it can be used to optimize the parameters of the classification model.

[0085] In the regression prediction task, the commonly used evaluation index is the mean square error loss, also known as the quadratic loss or L2 loss. The mean square error function measures the goodness of the model by calculating the square of the distance (i.e. error) between the predicted value and the actual value. That is, the closer the predicted value and the true value, the smaller the mean square error of the two.

[0086] Compared with some common machine learning or table deep learning methods, such as XGBoost, CatBoost, MLP, VIME, AutoINT, FT-Transformer, TransTab, etc., this model has obtained good results.

[0087] Based on the same inventive concept, as Figure 2As shown, the embodiment also provides a language model based cross-table multi-task pre-training device, comprising a table data collection and preprocessing module, a discrete feature value semantic corresponding module, a continuous feature value discretization processing module, a continuous value feature regularization processing module, a language model based cross-table multi-task pre-training module, and a downstream table fine-tuning and result testing module. The table data collection and preprocessing module is used for table data collection and preprocessing. The discrete feature value semantic corresponding module is used for semantic corresponding processing of discrete feature values. The continuous feature value discretization processing module is used for continuous feature value discretization processing. The continuous value feature regularization processing module is used for continuous feature value regularization processing. The language model based cross-table multi-task pre-training module is used for language model based cross-table multi-task pre-training. The downstream table fine-tuning and result testing module is used for downstream table fine-tuning and result testing.

[0088] It should be noted that the language model based cross-table multi-task pre-training device provided by the above embodiment should be illustrated by the division of the above functional modules when performing cross-table multi-task pre-training. The above functions can be completed by different functional modules as needed, i.e., the internal structure of the terminal or server is divided into different functional modules to complete all or part of the above described functions. In addition, the cross-table multi-task pre-training device and the cross-table multi-task pre-training method provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the cross-table multi-task pre-training method embodiment, which will not be described here.

[0089] The embodiment also provides a language model based cross-table multi-task pre-training device, comprising a memory, a processor, and a computer program. The memory stores computer executable instructions and data used or produced when the computer executable instructions are executed. The processor is communicatively connected with the memory and is configured to execute the computer executable instructions stored in the memory. The computer executable instructions, when executed, implement S110-S160 of the above language model based cross-table multi-task pre-training method.

[0090] The embodiment also provides a computer readable storage medium having a computer program stored thereon. The computer program, when executed by a processor, implements S110-S160 of the above language model based cross-table multi-task pre-training method.

[0091] The memory can be a volatile memory close to the terminal, such as RAM, and can also be a non-volatile memory, such as ROM, FLASH, floppy disk, mechanical hard disk, etc. The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA).

[0092] The above detailed description of the specific embodiments of the present application has described the technical solutions and beneficial effects of the present application, and it should be understood that the above description is only the most preferred embodiment of the present application and is not intended to limit the present application. Any modifications, supplements and equivalent replacements made within the principle range of the present application shall be included in the protection range of the present application.

Claims

1. A cross-table multi-task pre-training method based on a language model, characterized in that, Includes the following steps: Obtain labeled tabular data for classification and regression tasks. The tabular data includes discrete and continuous feature values. Filter out missing values ​​from the tabular data. Semantic mapping is performed on discrete feature values, converting discrete feature values ​​into corresponding semantic text; After normalizing the continuous feature values, the C4.5 algorithm of decision tree is used to discretize the continuous feature values, dividing them into multiple unequal intervals. These continuous feature values ​​are then assigned to the corresponding intervals, and each interval name is identified as a token and added to the vocabulary of the language model. Regularization is applied to the intervals corresponding to continuous feature values ​​to construct a regularization loss function that can preserve the numerical magnitude relationship within the intervals; The semantic text and interval tokens corresponding to the table data are input into the RoBERTa language model, and pre-training is performed on classification and regression tasks based on labels and regularization loss functions respectively. The pre-trained RoBERTa language model is fine-tuned based on downstream tasks, and the fine-tuned RoBERTa language model is then used for downstream task prediction.

2. The cross-table multi-task pre-training method based on language model according to claim 1, characterized in that, The process of filtering out missing values ​​from the table data includes: If the feature names in the table data that exceed the first threshold do not contain semantic information, then the table data will be deleted. If the number of features in the table data exceeds the second threshold or the text in the features exceeds the third threshold, then the table data will be deleted. If the number of missing values ​​corresponding to the features in the table data exceeds the fourth threshold, the table data will be deleted. If there are features in the table data that have fewer than the fifth threshold with missing values, then the feature with missing values ​​is determined to be a continuous feature and will be treated as a continuous feature. If it is determined to be a discrete feature, then the missing values ​​will be treated as text. If the feature names in the table data consist of multiple words and use camelCase, PascalCase, hyphen, or underscore naming conventions, then regular expressions will be used to detect and separate each word.

3. The cross-table multi-task pre-training method based on language models according to claim 1, characterized in that, The discrete features include numerical features and categorical features. For categorical features, if their original representation is numerical and there is already information explaining the correspondence between numerical and categorical information, then during semantic correspondence, the categorical information corresponding to the numerical value is used as semantic text. Discrete features presented as semantic text do not need to be semantically corresponded.

4. The cross-table multi-task pre-training method based on a language model according to claim 1, characterized in that, The C4.5 algorithm using decision trees is used to discretize continuous feature values, including: The basic conditions for defining the C4.5 algorithm are as follows: If all features of a tabular dataset belong to the same category, a leaf node is created directly to indicate the selection of that category; if none of the features provide any information gain, C4.5 uses the expected value of the class to create a decision node at the upper level of the tree; if a previously unseen feature is encountered, C4.5 uses the expected value to create a decision node at the upper level of the tree. Decision-making process: 1) Check the above basic information; 2) For each feature a, calculate the information gain of the partition a; 3) Denote a_best as the feature with the highest information gain; 4) Create a decision node that is partitioned on a_best; 5) Use the partitioned features to create child nodes as the current decision node, and process these child nodes recursively.

5. The cross-table multi-task pre-training method based on a language model according to claim 1, characterized in that, The step of performing regularization processing on the intervals corresponding to continuous feature values ​​to construct a regularization loss function that can preserve the numerical magnitude relationship within the intervals includes: The embedding vectors of tokens in the RoBERTa language model are randomly selected from three intervals. First, the token with the middle value is selected. Then, tokens with similar original values ​​are moved closer together, while tokens with different original values ​​are moved further apart. This can be expressed by the formula: L=max(d(a,p)-d(a,n)+margin,0) Where L is the regularization loss function, a represents the token of the middle size, p represents the token that belongs to the same category as a and is close to a, n represents the token that belongs to a different category as a and is far from a, d(·) represents the distance between the embedding vectors, margin represents a constant greater than 0, and max(·) represents the maximum operation. Using this regularization loss function L, the size relationship between the embedding vectors of tokens whose values ​​are divided into multiple intervals is made clear.

6. The cross-table multi-task pre-training method based on language model according to claim 1, characterized in that, During the pre-training process of inputting the semantic text corresponding to the table data and the token of the interval into the RoBERTa language model, when performing multi-head self-attention operations within the model, the positional labels between features are discarded, while the self-attention within the features is used. After performing self-attention operations on the feature name and value of each feature, a [CLS] token is generated to represent the feature and its value. Finally, the [CLS] tokens, which are equal to the number of features, are subjected to multi-head self-attention operations in the language model to obtain the final representation result.

7. The cross-table multi-task pre-training method based on a language model according to claim 1, characterized in that, During pre-training, the regularized loss function is combined with the task loss as the total loss to update the parameters of the RoBERTa language model.

8. A cross-table multi-task pre-training device based on a language model, characterized in that, It includes a table data collection and preprocessing module, a discrete feature numerical semantic correspondence module, a continuous feature numerical discretization processing module, a continuous numerical feature regularization processing module, a language model-based cross-table multi-task pre-training module, and a downstream table fine-tuning and result testing module. The tabular data collection and preprocessing module is used to acquire labeled tabular data for classification and regression tasks. The tabular data includes discrete and continuous feature values, and missing values ​​are filtered out from the tabular data. The discrete feature numerical semantic correspondence module is used to perform semantic correspondence on discrete feature numerical values ​​and convert discrete feature numerical values ​​into corresponding semantic text. The continuous numerical feature discretization processing module is used to normalize the continuous feature values ​​and then use the C4.5 algorithm of decision tree to discretize the continuous feature values, dividing the continuous feature values ​​into multiple unequal intervals and assigning these continuous feature values ​​to the corresponding intervals. Each interval name is identified as a token and placed in the vocabulary of the language model. The continuous feature value regularization processing module is used to perform regularization processing on the intervals corresponding to continuous feature values ​​in order to construct a regularization loss function that can preserve the numerical size relationship of the intervals. The language model-based cross-table multi-task pre-training module is used to input the semantic text and interval tokens corresponding to the table data into the RoBERTa language model, and perform pre-training for classification and regression tasks based on labels and regularization loss functions respectively. The downstream table fine-tuning and result testing module is used to fine-tune the pre-trained RoBERTa language model based on downstream tasks, and to use the fine-tuned RoBERTa language model to predict downstream tasks.

9. A language model-based cross-table multi-task pre-training device, comprising: A memory that stores computer-executable instructions and data used or produced in executing the computer-executable instructions; A processor, communicatively connected to the memory and configured to execute computer-executable instructions stored in the memory, characterized in that, when executed, the computer-executable instructions implement the language model-based cross-table multi-task pre-training method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is processed and executed, it implements the steps of the cross-table multi-task pre-training method based on a language model as described in any one of claims 1-6.

Images