Classification model building method, data classification method, medium and equipment

Abstract

The present application provides a classification model construction method, a data classification method, a medium and a device, and belongs to the field of information classification technology. The classification model construction method includes: obtaining a sample set consisting of tabular data, extracting the real content of non-text fields in the tabular data to pre-train multiple Transformer models to be trained, and obtaining multiple pre-trained Transformer models; extracting the first tabular data containing the real data type in the sample set to fine-tune each pre-trained Transformer model to form a trained Transformer model; performing hybrid expert model fusion on each trained Transformer model, determining the final weight of each trained Transformer model in the hybrid expert model, and forming a classification model for classifying tabular data. The present application can improve the accuracy of data classification.

Description

Technical Field

[0001] The present application relates to the field of information classification technology, and in particular to a classification model construction method, a data classification method, a medium and a device. Background Art

[0002] In data classification scenarios, processing tabular data is an important topic. The common processing method is to use supervised learning to classify or regress these data. Among them, the XGBoost model has become the preferred model for processing tabular data due to its advantages in efficiency and accuracy. However, the XGBoost model also has some limitations, such as the inability to effectively support task transfer learning, high sensitivity to input data, and difficulty in directly processing large-scale text data.

[0003] In order to overcome these limitations, the Transformer model has attracted attention in recent years due to its wide application in natural language processing and computer vision. With its powerful feature extraction capabilities and good generalization performance, the Transformer model has gradually become a powerful tool for processing tabular data. Despite this, most of the existing methods based on the Transformer model do not fully consider the ability of model transfer learning, and are still insufficient in processing large-scale text features. The accuracy of using the Transformer model to classify tabular data still needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to provide a classification model construction method, a data classification method, a medium and a device to improve the accuracy of tabular data classification using a Transformer model.

[0005] In a first aspect, the present application provides a classification model construction method, the method comprising:

[0006] Acquire a sample set consisting of tabular data, and extract the real content of non-text fields in the tabular data;

[0007] Pre-training a plurality of Transformer models to be trained based on the real content to obtain a plurality of pre-trained Transformer models, each Transformer model to be trained having different parameters;

[0008] Extracting first tabular data containing real data types from the sample set, and fine-tuning each pre-trained Transformer model based on the first tabular data to form a trained Transformer model;

[0009] Each trained Transformer model is fused with a hybrid expert model, and a final weight of each trained Transformer model in the hybrid expert model is determined to form a classification model for classifying the tabular data.

[0010] Optionally, the tabular data in the sample set includes first tabular data containing a real data type and second tabular data lacking the real data type.

[0011] Optionally, pre-training a plurality of Transformer models to be trained based on the real content to obtain a plurality of pre-trained Transformer models includes:

[0012] Using the real content as the input of the Transformer model under the current first iteration training, outputting the predicted content of the non-text field and the first predicted data type to which the tabular data belongs;

[0013] Based on the predicted content and the first predicted data type, a first comprehensive loss value of the Transformer model under the current first iterative training is calculated, and based on the first comprehensive loss value, the Transformer model under the current first iterative training is updated, thereby finally forming the pre-trained Transformer model.

[0014] Optionally, the using the real content as the input of the Transformer model under the current first iteration training, and outputting the predicted content of the non-text field and the first predicted data type to which the tabular data belongs, includes:

[0015] Performing a partial random mask replacement on the real content of the non-text field to form a non-text field feature after replacement;

[0016] The replaced non-text field features are input into the encoding layer in the Transformer model under the current first iteration training, the encoding layer forms a corresponding embedding vector for each replaced non-text field feature, the embedding vector is input into the multi-layer perceptron layer in the Transformer model under the current first iteration training, the multi-layer perceptron layer forms the predicted content and the first predicted data type based on the embedding vector.

[0017] Optionally, the calculating the first comprehensive loss value of the Transformer model under the current first iteration training based on the predicted content and the first predicted data type includes:

[0018] Calculate the first cross entropy loss value of the Transformer model under the current first iteration training under the field type based on the predicted content;

[0019] Calculate a second cross entropy loss value of the Transformer model under the current first iteration training under the data type based on the first predicted data type;

[0020] The first comprehensive loss value is calculated based on the first cross entropy loss value and the second cross entropy loss value.

[0021] Optionally, fine-tuning each pre-trained Transformer model based on the first tabular data to form a trained Transformer model includes:

[0022] Using the first tabular data as the input of the Transformer model in the current second iteration training, and outputting the second predicted data type of the first tabular data;

[0023] Calculating a second comprehensive loss value of the Transformer model under the current second iteration training based on the second predicted data type of the first tabular data and the real data type, and updating the Transformer model under the current second iteration training based on the second comprehensive loss value, thereby finally forming a fine-tuned Transformer model;

[0024] Combine the fine-tuned Transformer model with the preset BERT model for fields that process text features to form a trained Transformer model.

[0025] Optionally, the step of fusing each trained Transformer model with a hybrid expert model to determine a final weight of each trained Transformer model in the hybrid expert model includes:

[0026] Get the initial weights of each trained Transformer model;

[0027] Using the first tabular data as input to each trained Transformer model, and outputting the corresponding third predicted data type;

[0028] The initial weights are updated according to the actual data type in the first tabular data and the third predicted data type output by each trained Transformer model to form final weights.

[0029] In a second aspect of the present application, a data classification method is provided, the method comprising:

[0030] Obtaining data to be classified, and inputting the data to be classified into the classification model described in any one of the embodiments of the present application;

[0031] Outputting the initial data type of the data to be classified through each Transformer model in the classification model;

[0032] The final data type of the data to be classified is calculated based on each initial data type and the final weight of each Transformer model.

[0033] In a third aspect of the present application, an electronic device is provided, including: one or more processors;

[0034] A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the classification model building method and / or data classification method as described in any embodiment of the present application.

[0035] In a fourth aspect of the present application, a computer storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the classification model construction method and / or data classification method as described in any embodiment of the present application.

[0036] The classification model construction method, data classification method, medium and device in the present application set a pre-training stage and a fine-tuning stage for multiple Transformer models. In the pre-training stage, the content in the non-text field in the tabular data is used for iterative training to obtain each pre-trained Transformer model; in the fine-tuning stage, the first tabular data containing the data type in the sample set is extracted, and the pre-trained Transformer model is iteratively fine-tuned based on the first tabular data to form each trained Transformer model; finally, each trained Transformer model is integrated into the hybrid expert model, which can give full play to the performance of each trained Transformer model and realize the learning transfer ability of the Transformer model; by determining the final weights of each Transformer model in the hybrid expert model, a classification model of the hybrid expert model for data classification of tabular data is formed, so that the accuracy of classification of tabular data can be improved based on the classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope of the present application.

[0038] Figure 1 A schematic diagram of a process for constructing a classification model in one embodiment;

[0039] Figure 2 A schematic diagram of a process for calculating a first comprehensive loss value of a Transformer model under current first iteration training based on predicted content and a first predicted data type in one embodiment;

[0040] Figure 3 A schematic diagram of a process of fine-tuning each pre-trained Transformer model based on first table data to form a trained Transformer model in one embodiment;

[0041] Figure 4 A schematic diagram of a process of fusing each trained Transformer model with a hybrid expert model to determine a final weight of each trained Transformer model in the hybrid expert model in an embodiment;

[0042] Figure 5 is a flow chart of a data classification method in one embodiment;

[0043] Figure 6 FIG. 1 is a schematic diagram of the structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] All terms (including technical and scientific terms) used in this application have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0046] For example, the terms "first", "second", etc. used in this application are only used to distinguish similar objects and to differentiate the first object from another object, rather than to describe a specific order or sequence, and cannot be understood as indicating or implying relative importance.

[0047] This application proposes a classification model construction method, combining Figure 1 As shown, the method includes:

[0048] Step 110 , obtaining a sample set consisting of tabular data, and extracting the actual content of non-text fields in the tabular data.

[0049] In this embodiment, tabular data refers to data stored in a tabular form, usually consisting of multiple fields (or columns) and several records (or rows). Each field corresponds to a specific type of data attribute, such as name, age, income range, number of visits, etc., and each record (i.e., a piece of tabular data) contains specific values ​​of multiple fields, representing an independent data entity. Each piece of tabular data is also provided with a corresponding data type, and the data type and one or more fields of some tabular data may be null values ​​(i.e., the corresponding content is empty). The field content in the tabular data is its real content, and the data type of the record is the real type of the tabular data. The data type is a type set for classifying the tabular data, and the data type divided from the tabular data has multiple types, for example, there are k categories. The data type of each piece of tabular data can be divided into one or more of the k categories. Among them, k can be any suitable value set in advance.

[0050] In one embodiment, the tabular data constituting the sample set are all tabular data including complete content and data types, and may also include tabular data with complete content and tabular data with some field content and / or data types missing.

[0051] Taking customer data in the financial field as an example, tabular data may include the following fields: customer ID, name, gender, age, occupation, loan amount, loan term, repayment status, etc. A specific tabular data may be the detailed loan information of a certain customer, such as "Customer ID: 001, Name: Zhang San, Gender: Male, Age: 35, Occupation: Corporate White Collar, Loan Amount: 500,000 Yuan, Loan Term: 24 Months, Repayment Status: Normal Repayment".

[0052] In one embodiment, the fields of the tabular data include non-text fields and text fields. For example, the content in the text field (such as the name field) is related text annotation content, and the content in the non-text field can be discrete content (such as income range) or continuous content (such as age), etc. Discrete content can be related to level, data range, etc., and continuous content can be any number, for example.

[0053] As shown in Table 1 below, it lists 7 sets of table data, each of which includes the contents of 5 fields, namely field name, field classification, 7-day access count, 30-day field access count, and field comment, as well as the data type of the table data (i.e., "field classification" in Table 1) and field ID. Among them, the contents of the two fields, field classification and 30-day field access count, are discrete contents; the contents of the 7-day access count field are continuous contents; the contents of the fields in the field content and field comment are text contents. "NaN" in the table means that the corresponding field content is missing and is a null value.

[0054] Table 1

[0055]

[0056] There are multiple tabular data in the sample set, and the number can be determined according to the model to be trained to ensure that the training and construction requirements of the classification model can be met. For example, the tabular data in the sample set includes the 7 tabular data mentioned above. For the content of the field classification, there are 4 values ​​​​including L1, L2, L3, and L4; for the field content of the number of field visits in 30 days, there are 3 values ​​​​including "<100", "100-200", ">200"; the data type of discrete data (that is, the "field classification") has 4 data types including 1, 2, 3, and 4.

[0057] Step 120, pre-training multiple Transformer models to be trained based on real content to obtain multiple pre-trained Transformer models.

[0058] In this embodiment, the classification model to be constructed is a mixture of experts (Mixture of Experts, Mixture of Experts Model, referred to as "MoE model") composed of multiple Transformer models. The number of Transformer models to be combined can be set as needed, such as 3, 5, 8, etc. Each Transformer model to be trained has different parameters.

[0059] Specifically, each Transformer model can have different initial parameters or the same initial parameters, and can use completely identical, partially identical, or completely different samples to train itself, so that the Transformer model in the finally constructed MoE model has corresponding different parameters.

[0060] For example, before training each Transformer model, at least one initial parameter is different between each two Transformer models, and the initial parameter may be a hyperparameter in the Transformer model, the number of layers of the coding layer, the number of layers of the multi-layer perceptron layer, the dimension of the embedding vector, the parameter in the loss function, the weight corresponding to each loss function or loss value, the generation method of the random mask vector, etc. Furthermore, assuming that the number of Transformer models is N (respectively the 1st Transformer model to the Nth Transformer model), the sample set can also be split into N sample subsets (respectively the 1st sample subset to the Nth sample subset), and each sample subset may have an intersection or may not have an intersection at all, that is, the tabular data in the same sample set is split into both the i-th sample subset and the j-th sample subset. The tabular data in each two sample subsets are not exactly the same, and the i-th sample subset is used to train the i-th Transformer model.

[0061] In the pre-training stage, self-supervised learning can be used to pre-train the model, which does not rely on supervised labels (such as data types in tabular data), thereby improving the model's transfer learning ability. For each Transformer model to be trained, only the actual content of the non-text fields in the tabular data is used to iteratively train the Transformer model to be trained.

[0062] During each iterative training process, the Transformer model to be trained randomly replaces and restores the actual content of (part of) the non-text fields in the input tabular data, and outputs the content of the non-text fields in the corresponding tabular data (i.e., the predicted content) and predicts the data type of the tabular data (i.e., the first predicted data type).

[0063] The first comprehensive loss value is calculated for the obtained prediction content and the first prediction content, and the Transformer model to be trained is back-propagated based on the calculated first comprehensive loss value to update the relevant parameters therein. Multiple iterations are performed until the first termination iteration condition is met, and the Transformer model formed after the termination iteration is used as the pre-trained Transformer model.

[0064] Among them, the first termination iteration conditions of each Transformer model to be trained can be the same or different. For example, a corresponding first iteration number threshold can be set. When the iteration number of the Transformer model to be trained in the pre-training stage reaches the corresponding first iteration number threshold, the pre-training of the Transformer model to be trained is completed to form a pre-trained Transformer model. Or further set a first loss threshold. When the calculated first comprehensive loss value of the Transformer model to be trained in the current iteration is less than the first loss threshold, the pre-training of the Transformer model to be trained is completed to form a pre-trained Transformer model.

[0065] Step 130 , extracting first tabular data containing real data types from the sample set, and fine-tuning each pre-trained Transformer model based on the first tabular data to form a trained Transformer model.

[0066] In this embodiment, after completing the pre-training of each Transformer model, multiple corresponding pre-trained Transformer models are obtained. For each pre-trained Transformer model, a supervised learning method is used to fine-tune the model, that is, the pre-trained Transformer model is fine-tuned in combination with the real content and real data type of the non-text field in the tabular data (which may also include the real content of the text field).

[0067] In each fine-tuning phase, the real content of the non-text fields in the tabular data (which may also include the real content of the text fields) is used as the input of the corresponding pre-trained Transformer model. The pre-trained Transformer model predicts the data type of the corresponding tabular data (i.e., the second predicted data type) based on the real content of the input, and calculates the second comprehensive loss value of the pre-trained Transformer model based on the second predicted data type and the real data type. The pre-trained Transformer model is back-propagated based on the calculated loss value to update the relevant parameters. Multiple iterations are performed until the second termination condition is met, and the Transformer model formed after the termination iteration is used as the trained Transformer model.

[0068] Similarly, the second termination iteration conditions of each pre-trained Transformer model can be the same or different. For example, a corresponding second iteration number threshold can be set. When the number of iterations of the pre-trained Transformer model in the fine-tuning stage reaches the corresponding second iteration number threshold, the fine-tuning of the pre-trained Transformer model is completed to form a trained Transformer model. Or a second loss threshold can be further set. When the calculated second comprehensive loss value of the pre-trained Transformer model in the current iteration is less than the second loss threshold, the fine-tuning of the pre-trained Transformer model is completed to form a trained Transformer model.

[0069] The first comprehensive loss value and the second comprehensive loss value may be loss values ​​calculated by corresponding loss functions, and the loss function may be a combination of one or more of the cross entropy loss function, the mean absolute error loss function, the mean square error loss function, the Huber loss function, etc.

[0070] In one embodiment, the tabular data in the sample set includes first tabular data including real data types and second tabular data lacking real data types.

[0071] The first tabular data records the data type to which the data belongs, while the second tabular data does not record the data type to which the data belongs. In the pre-training stage of step 120, the first tabular data and / or the second tabular data can be used to pre-train the Transformer model, and in the fine-tuning stage of step 130, the first tabular data with the real data type is used to fine-tune the pre-trained Transformer model.

[0072] Step 140 , each trained Transformer model is fused with the hybrid expert model, and the final weight of each trained Transformer model in the hybrid expert model is determined to form a classification model for classifying tabular data.

[0073] In this embodiment, each trained Transformer model serves as an expert model in the hybrid expert model. A corresponding gating network layer is further set in the hybrid expert model. The gating network can be a lightweight neural network. The weights of each trained Transformer model are output through the gating network layer. The output results of each trained Transformer model are weighted summed through the gating network. The result of the weighted sum is the final output result of the classification model.

[0074] The final weight may be a fixed weight that is set in advance, for example, the final weights of each Transformer model may be set to be the same. The final weight may also be a weight formed after training through the above-mentioned gated network layer. Specifically, the initial weight of each Transformer model may be set, and training is performed in the gated network through the initial weight, and the weight after training is used as the final weight. After the final weight is determined, the above-mentioned classification model is formed.

[0075] The classification model construction method in the present application sets a pre-training stage and a fine-tuning stage for multiple Transformer models. In the pre-training stage, the content in the non-text field in the tabular data is used for iterative training to obtain each pre-trained Transformer model; in the fine-tuning stage, the first tabular data containing the data type in the sample set is extracted, and the pre-trained Transformer model is iteratively fine-tuned based on the first tabular data to form each trained Transformer model; finally, each trained Transformer model is integrated into the hybrid expert model, which can give full play to the performance of each trained Transformer model and realize the learning transfer ability of the Transformer model; by determining the final weights of each Transformer model in the hybrid expert model, a classification model of the hybrid expert model for data classification of the tabular data is formed, so that the accuracy of classification of the tabular data can be improved based on the classification model.

[0076] In one embodiment, step 120 includes: taking the real content as the input of the Transformer model under the current first iteration training, outputting the predicted content of the non-text field and the first predicted data type to which the tabular data belongs; calculating the first comprehensive loss value of the Transformer model under the current first iteration training based on the predicted content and the first predicted data type, and updating the Transformer model under the current first iteration training based on the first comprehensive loss value, thereby finally forming a pre-trained Transformer model.

[0077] In this embodiment, the real content in the non-text field is used as the input of the Transformer model under the current first iteration training. After receiving the input data, the model processes part or all of the real content in the column embedding layer. The processing process includes modifying the real data, random mask replacement and embedding, and then the processed real content is restored through the encoding layer and the MLP layer. Finally, the MLP layer outputs the restored data (i.e., the predicted content) and the prediction of the data type of the tabular data corresponding to the content (i.e., the first predicted data type).

[0078] Specifically, the real content after being processed by the column embedding layer can be a corresponding vector, such as an a-dimensional vector, where a can be any appropriate value set. For each real content, the feature vector converted into the feature vector represents the feature representation of the input real data. The feature vector is input into the encoding layer of the Transformer model, and one or more of the processes such as self-attention mechanism, position encoding, stacking and normalization are performed in the encoding layer to convert the input feature vector into an encoded vector (for example, a vector of the same dimension a, i.e., an embedding vector), and the encoded vector is further input into the MLP (Multi-Layer Perceptron) layer in the Transformer model. The MLP layer first concatenates the encoded vectors corresponding to the real contents of multiple non-text fields in a tabular data to form a concatenated vector (for example, if there are 3 non-text fields in a tabular data, the above 3 encoded a-dimensional vectors are concatenated to form a 3a-dimensional embedding vector), and one or more of the processes such as linear transformation, activation function, stacking and output, Softmax, etc. are performed on the concatenated vector, and finally the predicted contents corresponding to each real content and the predicted result of the data type of the tabular data (i.e., the first predicted data type) are output.

[0079] Optionally, the multiple vectors input to the MLP layer may not be concatenated but directly processed correspondingly for the three a-dimensional vectors to obtain an output result, that is, output corresponding prediction content and the first prediction data type.

[0080] After the output results of all input tabular data are completed, the first comprehensive loss value can be calculated based on the output results to obtain the first comprehensive loss value under the current first iteration training, and one or more parameters in the Transformer model under the current first iteration training are updated based on the first comprehensive loss value to complete an iterative update. Specifically, relevant optimization algorithms (such as Adam, SGD, etc.) can be used to update the parameters in the Transformer model.

[0081] Then, taking the updated Transformer model as the benchmark, repeat the above iterative training process, re-use the tabular data as input, perform corresponding processing, obtain the output result, and calculate the new first comprehensive loss value to further update the corresponding parameters.

[0082] When the number of pre-training iterations reaches a preset first number threshold, or the calculated first comprehensive loss value is less than the preset first loss threshold, the pre-training of the corresponding Transformer model is terminated to form a pre-trained Transformer model. The first number threshold and the first loss threshold can be any appropriate pre-set values.

[0083] In one embodiment, real content is used as input of a Transformer model under current first iteration training, and predicted content of a non-text field and a first predicted data type to which tabular data belongs are output, including: performing partial random mask replacement on the real content of the non-text field to form replaced non-text field features; inputting the replaced non-text field features into an encoding layer in a Transformer model under current first iteration training, the encoding layer forming a corresponding embedding vector for each replaced non-text field feature, inputting the embedding vector into a multi-layer perceptron layer in a Transformer model under current first iteration training, and the multi-layer perceptron layer forming predicted content and a first predicted data type based on the embedding vector.

[0084] Taking the above-mentioned sample of tabular data with field ID "2" as an example, its non-text fields include field rating, 7-day visit count, and 30-day visit count, and the corresponding real contents are L2, 57, and >200 respectively. L2, 57, and >200 are used as the input of the Transformer model under the current first iteration training. The column embedding layer in the Transformer model generates a random mask vector, and the length of the random mask vector is the same as the number of non-text fields (for example, the length is 3). Each element in the random mask vector corresponds to a non-text field in the tabular data, and the value of the element is any value of 0 or 1, where 0 indicates that the real content of the corresponding non-text field needs to be retained, and 1 indicates that the real content of the corresponding non-text field needs to be replaced. By setting the probability of generating 1 (for example, 0.3), for example, a random mask vector of [0,1,0] is finally generated, which means that the real content L2 and ">200 are retained, and the real content "57" of the 7-day visit count field is replaced.

[0085] For the real content that needs to be replaced, it can be selected from the real content of the same non-text field (such as "7-day visit times") in other tabular data. Specifically, it can be preferentially selected randomly from tabular data with missing data types but the same non-text fields. For example, the real content "61" of "7-day visit times" in the tabular data with field ID "4" is selected, that is, the above real content "57" is replaced with "61". That is, after the real content in the three non-text fields is replaced, the content of the replaced non-text fields is "L2, 61, >200".

[0086] A corresponding feature vector is generated for the replaced non-text field content. For example, each of the three non-text features is mapped into a corresponding a-dimensional feature vector, that is, three a-dimensional feature vectors (i.e., non-text field features) are obtained and used as the input of the encoding layer.

[0087] In the encoding layer, one or more of the processes such as self-attention mechanism, position encoding, stacking and normalization are performed on the three a-dimensional feature vectors to convert the input feature vector into an encoded vector (also a three a-dimensional vector, i.e., an embedded vector).

[0088] Further, the three a-dimensional embedding vectors are concatenated to form a 3a-dimensional concatenated vector as the input of the multi-layer perceptron layer. In the MLP layer, the concatenated vector is subjected to one or more processes such as linear transformation, activation function, stacking and output, and Softmax (or the three a-dimensional vectors may be directly processed without concatenation), and finally the corresponding predicted content of each non-text field and the first predicted data type are output.

[0089] Among them, in addition to reflecting the predicted value of the non-text field, the predicted content can also further reflect the probability of the corresponding predicted value. The probability represents the probability of each predicted value (including the predicted value being empty) in the corresponding non-text field. Specifically, for discrete fields (with a limited number of numerical values), the output prediction content is the probability of each predicted value, and the sum of the probabilities of each predicted value is 1; for continuous fields, the output prediction content is the predicted value. For example, for the above-mentioned "L2, 57, >200", the predicted content corresponding to the actual content "L2" of the non-text field of field classification can be any one of L1~L2 or NaN, or [0.1, 0.8, 0.05, 0.03, 0.02]. When it is [0.1,0.8, 0.05, 0.03, 0.02], it indicates that the probability of the value of the field classification being L1 is 0.1, the probability of the value being L2 is 0.8, the probability of the value being L3 is 0.05, the probability of the value being L4 is 0.03, and the probability of the value being empty is 0.02; the predicted content corresponding to the true content "57" of the field of the number of visits in 7 days is "57"; the predicted content corresponding to the true content ">200" of the field of the number of visits in 30 days can be any one of "<100", "100-200", ">200", and NaN, or [0.1, 0.06, 0.82, 0.02]. When [0.1, 0.06, 0.82, 0.02], it indicates that the probability that the value of the number of visits to the field in the 30 days is "<100" is 0.1, the probability that the value is 100-200 is 0.06, the probability that the value is >200 is 0.82, and the probability that the value is empty is 0.02.

[0090] Similar to the prediction content, the first predicted data type not only reflects the data type to which the tabular data belongs, but also further reflects the probability of the corresponding predicted data type. For example, the predicted first predicted data type of the field ID "2" can be any one of 1, 2, 3, and 4, or [0.01, 0.95, 0.02, 0.02], which means that the probability of the data type to which the tabular data belongs being type 1 is 0.01, the probability of 2 is 0.95, the probability of 3 is 0.02, and the probability of 4 is 0.02.

[0091] In one embodiment, in addition to outputting the first predicted data type and predicted content, a predicted mask vector corresponding to the random mask vector is further output, and the predicted mask vector is used to indicate which non-text fields the real content in which is replaced. For example, the predicted mask vector generated by the field ID "2" is [0, 1, 0], which is the same as the random mask vector.

[0092] In one embodiment, Figure 2 As shown, the first comprehensive loss value of the Transformer model under the current first iteration training is calculated based on the predicted content and the first predicted data type, including:

[0093] Step 210, based on the predicted content, calculate the first cross entropy loss value of the Transformer model under the field type in the current first iteration training.

[0094] In this embodiment, the first cross entropy loss value may be a value calculated by a cross entropy loss function, which is used to measure the difference between the predicted content and the real content. The smaller the value, the more accurate the corresponding prediction result.

[0095] The first cross entropy loss value L1 can be calculated using the following formula 1:

[0096] (Formula 1)

[0097] Where N represents the number of samples used by the corresponding Transformer model in the pre-training stage, that is, the number of tabular data used; C represents the number of fields in the tabular data involved in training; Indicates the mask bool value corresponding to the real content of the jth field in the i-th data. The value can be 0 or 1. 1 indicates that the real content is masked, and 0 indicates that the real content maintains the original value. Represents the mask network output corresponding to the true content of the jth field in the i-th data; It is used to reflect the probability of the prediction accuracy for the j-th field in the i-th data. The larger the value, the higher the prediction accuracy.

[0098] For example, the tabular data shown in Table 1 above has three fields involved in the training, namely, field classification, 7-day visit times, and 30-day visit times. Then C=3; Indicates the masked bool value corresponding to the actual content (such as the null value NaN) of the second field (such as "7-day visit count") in the first number (such as the tabular data with ID "1" in Table 1).

[0099] Step 220 , based on the first predicted data type, calculate a second cross entropy loss value of the Transformer model under the data type in the current first iteration training.

[0100] Similar to the first cross entropy loss value, the second cross entropy loss value is also a value calculated by the corresponding cross entropy loss function, which is used to measure the accuracy of the predicted data type. The smaller the value, the more accurate the corresponding prediction result.

[0101] The second cross entropy loss value L2 can be calculated using the following formula 2:

[0102] (Formula 2)

[0103] Where N represents the number of samples used by the corresponding Transformer model in the pre-training stage, that is, the number of tabular data used; Used to reflect the reference data type in the i-th data; A network output for reflecting the predicted data type for the i-th data; It is used to reflect the probability of the prediction accuracy of the data type for the i-th data. The larger the value, the higher the prediction accuracy.

[0104] The reference data type may be determined by the predicted data type based on one or more Transformer models in the current iteration. Since each Transformer model outputs a corresponding predicted data type in each iteration, a weighted sum may be performed based on the predicted data types output by each Transformer model, and the calculated result may be used as the reference data type.

[0105] In one embodiment, it also includes: calculating a third cross entropy loss value of the Transformer model under content replacement under the current first iteration training based on the random mask vector and the predicted mask vector.

[0106] During the input process of the pre-training stage, the Transformer model will generate a corresponding random mask vector to replace the real content, and will output a prediction for the random mask vector (i.e., the predicted mask vector) under the iterative output result. Therefore, the third cross-entropy loss value for the prediction of the random mask vector can be further calculated here, and the accuracy of the prediction for content replacement can be reflected by the third cross-entropy loss value.

[0107] Similarly, the third cross entropy loss value L3 can be calculated using the following formula 3:

[0108] (Formula 3)

[0109] Where N represents the number of samples used by the corresponding Transformer model in the pre-training stage, that is, the number of tabular data used; Used to reflect the random mask vector corresponding to the i-th data; The network output for reflecting the prediction mask vector corresponding to the i-th data; It is used to reflect the probability of the prediction accuracy of the random mask vector for the i-th data. The larger the value, the higher the prediction accuracy.

[0110] Optionally, the order of calculating the cross entropy loss values ​​may not be limited. For example, the calculation of the cross entropy loss values ​​may be performed simultaneously, or the first cross entropy loss value, the second cross entropy loss value, and the third cross entropy loss value may be calculated in sequence.

[0111] Step 230: Calculate a first comprehensive loss value based on the first cross entropy loss value and the second cross entropy loss value.

[0112] In this embodiment, the electronic device pre-sets the loss value weights of the cross entropy loss values ​​of each dimension (including the first cross entropy loss value, the second cross entropy loss value, and the third cross entropy loss value). By weighted summing each cross entropy loss value with the corresponding loss value weight (or using other suitable combinations), the corresponding first comprehensive loss value can be obtained. The loss value weights corresponding to different Transformer models can be the same or different.

[0113] By calculating the first comprehensive loss value of the Transformer model for multiple dimensions such as true content prediction and data type prediction, the accuracy of the model prediction can be improved.

[0114] In one embodiment, Figure 3 As shown, each pre-trained Transformer model is fine-tuned based on the first table data to form a trained Transformer model, including:

[0115] Step 310: Use the first tabular data as input to the Transformer model in the current second iteration training, and output a second predicted data type of the first tabular data.

[0116] Step 320, calculate the second comprehensive loss value of the Transformer model under the current second iteration training based on the second predicted data type and the real data type of the first tabular data, and update the Transformer model under the current second iteration training based on the second comprehensive loss value, so as to finally form a fine-tuned Transformer model.

[0117] In this embodiment, each pre-trained Transformer model is fine-tuned using the first tabular data containing non-text fields and data types, and the parameters in the pre-trained Transformer model are updated. The fine-tuning process is similar to the pre-training process, and the main difference is that the loss value calculation and back propagation are performed using data containing real data types.

[0118] Specifically, in each iteration of the fine-tuning phase, the input and output processing of the pre-trained Transformer model is similar to that of the pre-training phase, where the real content of the non-text field in the first tabular data is used as the input of the pre-trained Transformer model, and the real content is subjected to random mask replacement processing in the column embedding layer, encoding processing in the encoding layer, prediction restoration processing in the multi-layer perceptron layer, etc., to output the predicted content and the second predicted data type. No further details will be given here.

[0119] For the second prediction type output by the pre-trained Transformer model, a second comprehensive loss value is calculated in combination with the real data type of the first tabular data.

[0120] Optionally, the second comprehensive loss value may also be calculated based on the cross entropy loss function. The second comprehensive loss value L4 may be calculated according to the following formula 4:

[0121] (Formula 4)

[0122] Where M represents the number of samples used by the corresponding pre-trained Transformer model in the fine-tuning stage, that is, the number of first table data used; Used to reflect the actual data type corresponding to the i-th data; A network output for reflecting the predicted data type corresponding to the i-th data; It is used to reflect the probability of the prediction accuracy of the data type for the i-th data. The larger the value, the higher the prediction accuracy.

[0123] The parameters in the pre-trained Transformer model are updated according to the calculated second comprehensive loss value, and iteration is continued based on the updated Transformer model until the corresponding second iteration termination condition is met to form a fine-tuned Transformer model.

[0124] Step 330 , combining the fine-tuned Transformer model with the preset BERT model for processing the text feature field to form a trained Transformer model.

[0125] For each fine-tuned Transformer model, combine it with the pre-trained BERT (Bidirectional Encoder Representations from Transformers) model to form a trained Transformer model.

[0126] Among them, the BERT model is a trained model obtained after iterative training based on the content of the text field in the tabular data in the sample set. The BERT model is used to process the field content of the text field in the tabular data, connect the processed content with the output of the encoding layer in the fine-tuned Transformer model, and then output the corresponding second predicted data type together with the MLP layer in the fine-tuned Transformer model.

[0127] By combining the fine-tuned Transformer model with the trained BERT model to form a trained Transformer model, the Transformer model's ability to predict the data type of tabular data can be further improved.

[0128] In one embodiment, Figure 4 As shown, each trained Transformer model is fused with a hybrid expert model to determine the final weight of each trained Transformer model in the hybrid expert model, including:

[0129] Step 410, obtaining the initial weights of each trained Transformer model.

[0130] Step 420 , using the first tabular data as input to each trained Transformer model, and outputting the corresponding third predicted data type.

[0131] Step 430 , updating the initial weights according to the actual data type in the first tabular data and the third predicted data type output by each trained Transformer model to form final weights.

[0132] In this embodiment, the initial weights of each trained Transformer model may be pre-set weights, and these initial weights may be the same value or different values. For example, they may be uniformly set to a fixed weight, or may be pre-set to the corresponding initial value according to the effect of each trained Transformer model predicting the data type of the tabular data.

[0133] The hybrid expert model includes a weight network layer. The column embedding layers in each trained Transformer model can be spliced. Specifically, the column embedding layers in each fine-tuned Transformer model can be spliced. The data output by the column embedding layer of each fine-tuned Transformer model is combined with the data output by the BERT model as the corresponding encoding layer input for further processing.

[0134] During the iterative training of weights, the parameters of each trained Transformer model (such as the weight of the self-attention mechanism, the network parameters of the MLP layer, etc.) remain unchanged, and only the relevant parameters in the weight network are updated. The weights of each Transformer model can be iteratively updated based on the gradient optimization method to obtain the final weight.

[0135] Each time the input data (corresponding to the feature x below) passes through the trained Transformer model, it will be weightedly selected through the weight network to determine which Transformer models participate in the calculation. The probability distribution of these selections (i.e., weights) will be adjusted according to the gradient information during the training process, thereby optimizing the frequency of use of each trained Transformer model and its corresponding weight, so that the model can fully utilize the strengths of each Transformer model while maintaining expert capabilities.

[0136] Optionally, the weight update process can be calculated according to the following formulas 5 to 7.

[0137] (Formula 5)

[0138] (Formula 6)

[0139] (Formula 7)

[0140] Where M represents the total number of trained Transformer models fused with the parameter MoE model; a represents the multiplication of the trainable parameters output by the last MLP layer in the trained Transformer model; b represents the bias parameter output by the prediction result of the trained Transformer model. and Respectively represent the network parameters of the mth trained Transformer model, , They represent the network parameters of the nth trained Transformer model, , , , They represent the network parameters of the mth trained Transformer model when y=k; Represents the final output prediction result output by the MoE model for the feature x in the input tabular data; Represents the third predicted data type y output by the mth trained Transformer model for the input feature x; Represents the weight corresponding to the feature x of the mth trained Transformer model; The output vector representing the concatenated column embedding layer; Used to reflect the output vector after the encoding layer and MLP layer.

[0141] By performing a preset number of iterations, the final weights of each trained Transformer model are formed.

[0142] In one embodiment, a data classification method is provided, combining Figure 5 As shown, the method includes:

[0143] Step 510, obtaining data to be classified, and inputting the data to be classified into a pre-built classification model.

[0144] In this embodiment, the data to be classified is tabular data to be classified, for example, tabular data similar to that shown in the above Table 1. The data to be classified includes non-text fields and text fields, and the real content in some fields may be missing.

[0145] The pre-constructed classification model may be a classification model constructed by the classification model construction method in any one of the above embodiments.

[0146] Step 520: Output the initial data type of the data to be classified through each Transformer model in the classification model.

[0147] In this embodiment, the classification model includes various trained Transformer models. By processing the content of each field in the input tabular data, each trained Transformer model outputs the predicted data type (i.e., the initial data type) of the data to be classified.

[0148] Specifically, each trained Transformer model takes the actual content of the non-text field in the data to be classified as the input of the column embedding layer, and after embedding processing, forms a corresponding feature vector as the input of the encoding layer. The encoding layer encodes the feature vector to form a corresponding embedding vector as the input of the MLP layer, and the MLP layer finally outputs the corresponding initial data type.

[0149] Among them, the initial data type output by each trained Transformer model can be a corresponding vector, and each parameter in the vector reflects the probability that the model to be classified belongs to one of the data types.

[0150] Step 530 , calculating the final data type of the data to be classified based on each initial data type and the final weight of each Transformer model.

[0151] Specifically, similar to the weight training process, when the data to be classified is input into the classification model, the classification model further determines which Transformer models participate in the calculation based on the data to be classified, and these selections correspond to the final weights of the Transformer model.

[0152] After each Transformer model outputs the corresponding initial data type, each initial data type is weighted and summed based on the corresponding final weight, and the final data type is determined from the result obtained after the weighted summation, thereby completing the classification of the data to be classified. Similarly, the final data type can be a corresponding vector used to reflect that the model to be classified belongs to various data types, or it can be a final data type.

[0153] The data classification method in the present application can improve the accuracy of data classification by using the data classification model constructed as above to classify the data to be classified.

[0154] In one embodiment, a computer storage medium is provided, on which executable instructions are stored. When the instructions are executed by a processor, the processor executes the steps in the above-mentioned embodiments of the classification model construction method and the data classification method.

[0155] In one embodiment, an electronic device is proposed, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of each classification model construction method and data classification method in any of the above-mentioned embodiments.

[0156] In one embodiment, an electronic device is provided, which may be a terminal or a server. Figure 6As shown, the electronic device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0157] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0159] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, all of the above embodiments may be used in any combination. The information disclosed in this background technology section is intended only to deepen the understanding of the overall background technology of the present application and should not be regarded as an admission or in any form to imply that the information constitutes prior art known to those skilled in the art.

Claims

1. A classification model construction method, characterized in that: The method comprises: Acquire a sample set consisting of tabular data, and extract the real content of non-text fields in the tabular data; Pre-training a plurality of Transformer models to be trained based on the real content to obtain a plurality of pre-trained Transformer models, each Transformer model to be trained having different parameters; Extracting first tabular data containing real data types from the sample set, and fine-tuning each pre-trained Transformer model based on the first tabular data to form a trained Transformer model; Each trained Transformer model is fused with a hybrid expert model, and a final weight of each trained Transformer model in the hybrid expert model is determined to form a classification model for classifying the tabular data.

2. The method according to claim 1, characterized in that The tabular data in the sample set includes first tabular data containing real data types and second tabular data lacking the real data types.

3. The method according to claim 1, characterized in that The pre-training of multiple Transformer models to be trained based on the real content to obtain multiple pre-trained Transformer models includes: Using the real content as the input of the Transformer model under the current first iteration training, outputting the predicted content of the non-text field and the first predicted data type to which the tabular data belongs; Based on the predicted content and the first predicted data type, a first comprehensive loss value of the Transformer model under the current first iterative training is calculated, and based on the first comprehensive loss value, the Transformer model under the current first iterative training is updated, thereby finally forming the pre-trained Transformer model.

4. The method according to claim 3, characterized in that The method of using the real content as the input of the Transformer model under the current first iteration training and outputting the predicted content of the non-text field and the first predicted data type to which the tabular data belongs includes: Performing a partial random mask replacement on the real content of the non-text field to form a non-text field feature after replacement; The replaced non-text field features are input into the encoding layer in the Transformer model under the current first iteration training, the encoding layer forms a corresponding embedding vector for each replaced non-text field feature, the embedding vector is input into the multi-layer perceptron layer in the Transformer model under the current first iteration training, the multi-layer perceptron layer forms the predicted content and the first predicted data type based on the embedding vector.

5. The method according to claim 3, characterized in that: The calculating the first comprehensive loss value of the Transformer model under the current first iteration training based on the predicted content and the first predicted data type includes: Calculate the first cross entropy loss value of the Transformer model under the current first iteration training under the field type based on the predicted content; Calculate a second cross entropy loss value of the Transformer model under the current first iteration training under the data type based on the first predicted data type; The first comprehensive loss value is calculated based on the first cross entropy loss value and the second cross entropy loss value.

6. The method according to claim 1, characterized in that The fine-tuning of each pre-trained Transformer model based on the first tabular data to form a trained Transformer model includes: Using the first tabular data as the input of the Transformer model in the current second iteration training, and outputting the second predicted data type of the first tabular data; Calculating a second comprehensive loss value of the Transformer model under the current second iteration training based on the second predicted data type of the first tabular data and the real data type, and updating the Transformer model under the current second iteration training based on the second comprehensive loss value, thereby finally forming a fine-tuned Transformer model; Combine the fine-tuned Transformer model with the preset BERT model for fields that process text features to form a trained Transformer model.

7. The method according to claim 1, characterized in that The step of fusing each trained Transformer model with a hybrid expert model and determining a final weight of each trained Transformer model in the hybrid expert model includes: Get the initial weights of each trained Transformer model; Using the first tabular data as input to each trained Transformer model, and outputting the corresponding third predicted data type; The initial weights are updated according to the actual data type in the first tabular data and the third predicted data type output by each trained Transformer model to form final weights.

8. A data classification method, characterized in that: The method comprises: Acquire data to be classified, and input the data to be classified into a classification model constructed according to any one of claims 1 to 7; Outputting the initial data type of the data to be classified through each Transformer model in the classification model; The final data type of the data to be classified is calculated based on each initial data type and the final weight of each Transformer model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores executable instructions, and when the executable instructions are executed by the processor, the processor executes the classification model construction method as described in any one of claims 1 to 7, and / or the data classification method in claim 8.

10. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to execute the classification model building method as described in any one of claims 1 to 7, and / or the data classification method in claim 8.

Images