A tabular data prediction model based on multi-branch parallel transformer, its construction method, and its application.

Abstract

This invention relates to the field of machine learning technology, specifically to tabular data prediction technology in supervised learning. It discloses a tabular data prediction model based on a multi-branch parallel transformer, suitable for classification and regression tasks. Tabular data is highly heterogeneous, making it difficult for existing deep learning models and transformer-like models such as FT-Transformer to adapt effectively, resulting in insufficient prediction accuracy. This model includes an input layer, a multi-branch feature extraction module, and a fusion output layer: the input layer converts numerical and categorical features into embedded representations through a feature segmenter (without CLStoken); the multi-branch module contains parallel transformer branches (alternating stacked intra / inter-sample attention layers) and MLP branches to enhance the diversity of feature extraction; the fusion output layer outputs the results after concatenation, pooling, and a prediction head. This invention captures inter-sample correlations through a hybrid attention mechanism, and the multi-branch structure adapts to heterogeneous features, significantly improving prediction accuracy. It outperforms existing models in both the BankMarketing classification task and the California Housing regression task.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, specifically to tabular data prediction technology in supervised learning, and more particularly to a tabular data prediction model based on a multi-branch parallel transformer, suitable for classification and regression tasks. Background Technology

[0002] Tabular data is one of the most common and fundamental data formats in machine learning and data analysis. It organizes data in a two-dimensional tabular structure, where each row typically represents a sample or observation, and each column represents a feature or variable. Tabular data is widely found in the real world, such as business sales records, financial transaction data, medical examination results, and questionnaires.

[0003] Unlike unstructured data such as images, text, or audio, tabular data has clear structured characteristics: each feature usually has a clear data type (such as numerical, categorical, or temporal), and the relationships between different features are mainly discovered through statistical methods or feature engineering techniques.

[0004] Prediction based on tabular data has long been a major area of ​​research in machine learning, falling under the category of supervised learning. Tabular data prediction can be divided into two categories: classification and regression. Classification tasks output discrete category labels, aiming to determine which class a sample belongs to, such as spam detection, image classification, or sentiment analysis. Models typically learn the relationship between features and categories, outputting the probability distribution for each category and selecting the category with the highest probability as the prediction result. In contrast, regression tasks output continuous numerical values, used to predict specific quantitative results, such as house price prediction, temperature prediction, or sales volume prediction. Models learn the mapping relationship between input features and continuous target variables, outputting a real numerical value as the prediction result.

[0005] Generally, modeling tabular data typically uses traditional machine learning methods. In recent years, an increasing number of deep learning models have also been applied to tabular data prediction tasks. However, tabular data often exhibits strong heterogeneity because different columns (original features) typically have different types, data sources, and numerical ranges. Current deep learning methods lack specific designs to address this heterogeneity, thus failing to effectively handle heterogeneous features and resulting in low prediction accuracy.

[0006] In the fields of natural language processing and computer vision, the Transformer architecture has been widely adopted and has become a mainstream framework due to its excellent performance and scalability. The Transformer is a deep learning architecture based on the attention mechanism. Unlike traditional recurrent neural networks (RNNs) or convolutional neural networks (CNNs), the Transformer relies entirely on the attention mechanism to model the dependencies between elements in sequential data, thereby achieving higher parallel efficiency and stronger long-distance dependency modeling capabilities. The core component of the Transformer is the self-attention mechanism, which dynamically allocates weights based on the relevance of different positions in the input sequence, enabling the model to "focus" on information most relevant to the current word or feature. A classic Transformer structure is shown below. Figure 1 As shown, a single transformer layer can be divided into two modules: a multi-head self-attention (MSA) module and a feed-forward network (FFN) module. Each module consists of several linear layers, each of which performs matrix multiplication on the input and weights to obtain the output. The linear layers in the multi-head self-attention module include (see attached diagram). Figure 1 ): Query, key, value, and output; the linear layers in the feedforward network module include (see appendix) Figure 1 : gate, up, and down.

[0007] Although the transformer model structure itself has some advantages in sequence modeling, it still lacks effective means to handle the heterogeneity of tabular data. Reference [1] proposed a transformer-based tabular data prediction model, FT-Transformer, whose core contribution is a feature tokenizer specifically designed for tabular data, but its prediction accuracy on tabular data is still not high. Summary of the Invention

[0008] To address the technical problems of existing deep learning methods being unable to effectively handle heterogeneous features in tabular data and having low prediction accuracy, this invention provides a tabular data prediction model based on a multi-branch parallel transformer, which can more effectively and accurately handle heterogeneous features in tabular data and improve the model's accuracy in tabular data classification and regression tasks.

[0009] The tabular data prediction model proposed in this invention, based on a multi-branch parallel transformer, includes: an input layer, a multi-branch feature extraction module, and a fusion output layer; wherein,

[0010] The input layer mainly consists of a feature segmenter, which is derived from reference [1] and does not use CLStoken. The feature segmenter divides the input features... Convert to embedded representation ,in The length of the embedded representation, This specifies the number of columns for the input table data.

[0011] For a given feature Its embedding representation is calculated as follows:

[0012]

[0013] in, It is the first The bias term of each feature. Embedding functions can be categorized into numerical feature embedding functions. and categorical feature embedding function : By weight Element-wise multiplication is implemented; By looking up the table Implementation, in which Indicates the first The number of values ​​for each category feature.

[0014] The embedding representations for numerical features and categorical features are as follows:

[0015]

[0016]

[0017]

[0018] in, It is the one-hot vector corresponding to the categorical feature. The number of columns for numerical features. The number of columns for categorical features, and .

[0019] After the feature segmentation unit, the original two-dimensional input [number of samples] , number of columns ] becomes three-dimensional [sample number] , number of columns Embedding length ],in The number of samples for the input table data.

[0020] The multi-branch feature extraction module mainly consists of two types of branches: transformer branches and multilayer perceptron branches.

[0021] Each transformer branch is composed of... It consists of alternating stacks of intra-sample attention transformer layers and inter-sample attention transformer layers.

[0022] (1) In-sample attention transformer: i.e., the ordinary multi-head attention mechanism, for input [number of samples] , number of columns Embedding length Each sample [number of columns] in ] Embedding length The different columns interact with each other through multi-head attention, enabling attention calculation between different feature columns within the sample;

[0023] (2) Inter-sample attention transformer: Before entering the transformer, the input [number of samples] is first processed. , number of columns Embedding length The order is rearranged to [1, number of samples]. , number of columns Embedding length ], so the original Each sample is effectively transformed into a new sample, while each of the original samples becomes a column of length [number of columns]. Embedding length The vector; the rearranged [1, sample number , number of columns Embedding length The data is input into a multi-head attention system, where different samples interact with each other, effectively calculating attention between samples. After multi-head attention, the data is rearranged into [number of samples]. , number of columns Embedding length The shape of the image is used to enter the feedforward network module.

[0024] The above-mentioned hybrid attention transformers are arranged in parallel. These three elements together form the transformer branch; The value range is [1, 10] (with an interval of 1), and the final value is obtained through hyperparameter tuning using a validation set. Ultimately, this... A hybrid attention transformer branch passes through the number of columns Concatenate along the dimension to obtain the output [number of samples]. , number of columns Embedding length ].

[0025] Each multilayer perceptron branch consists of three linear layers and an activation function, namely, an upscaling layer (for embedding length). In proportion (Extending), mapping layer and dimensionality reduction layer (reducing the embedding length) The activation function is generally the rectified linear unit (ReLU).

[0026] After passing through a multilayer perceptron branch, the input [number of samples] , number of columns Embedding length The change is: [sample number] , number of columns Embedding length ]→[sample size] , number of columns Embedding length ]→[sample size] , number of columns Embedding length ], The value range is [1.1, 2] (with an interval of 0.1), and the final value is obtained by hyperparameter tuning through the validation set.

[0027] The above-mentioned multilayer perceptrons are arranged in parallel. These elements together form the multilayer perceptron branch. The value range is [1, 10] (with an interval of 1), and the final value is obtained through hyperparameter tuning using a validation set. Ultimately, this... A multilayer perceptron branch passes through the column number Concatenate along the dimension to obtain the output [number of samples]. , number of columns Embedding length ].

[0028] A multi-branch structure is beneficial for handling heterogeneous original feature columns in tabular data, and can enhance the diversity and richness of features extracted by the network.

[0029] The fusion output layer is used to process the output of the multi-branch feature extraction module, transforming the output into the final predicted shape and providing the prediction result. The specific process is as follows:

[0030] First, the output of the multi-branch feature extraction module, i.e., the output of the transformer branch [number of samples]. , number of columns Embedding length The output of the multilayer perceptron branch [number of samples] , number of columns Embedding length ], along the column number Concatenate along the dimensions to obtain [number of samples]. , number of columns Embedding length ];

[0031] Then, after an average pooling layer, along the column number... Perform average pooling (i.e., calculate the mean) on the dimension to obtain [number of samples]. 1. Embedding length ], then compress the number of columns. This dimension yields [number of samples] Embedding length At this point, each sample is represented as a string of length […]. ;

[0032] Finally, [number of samples] Embedding length The input is processed through a prediction head to obtain the predicted output. The prediction head is a linear layer with an input dimension of... The output dimension is the dimension of the prediction result. That is, if it is a classification task, =Number of categories, if it's a regression task =1, therefore the output shape is [number of samples] Number of categories ].

[0033] Based on the above, this invention also proposes a method for constructing and training a tabular data prediction model based on a multi-branch parallel transformer, comprising the following steps:

[0034] Step 1, Sample data collection and labeling:

[0035] Obtain the original sample dataset for tabular data prediction. The sample dataset includes multiple sample records, each of which includes at least several feature columns and the prediction target label corresponding to the sample.

[0036] The feature column includes at least numerical features and / or categorical features, and the predicted target label is a categorical label or a continuous numerical label.

[0037] Step 2, Data Preprocessing:

[0038] The sample dataset is preprocessed, and the preprocessing includes at least one or more of the following:

[0039] Normalize or standardize numerical features;

[0040] Encode categorical features;

[0041] Missing values ​​are filled in, with missing values ​​for numerical features filled with the mean or median, and missing values ​​for categorical features filled with preset default values.

[0042] Truncate or smooth out outliers.

[0043] Step 3, Dataset Partitioning:

[0044] The preprocessed sample dataset is divided into a training set, a validation set, and a test set, where:

[0045] The training set is used to train the model parameters;

[0046] The validation set is used for selecting model hyperparameters and tuning the training process;

[0047] The test set is used to evaluate the predictive performance of the final model.

[0048] Step 4, Model Building and Initialization:

[0049] Based on the aforementioned tabular data prediction model structure based on multi-branch parallel transformer, a model network is constructed, including an input layer, a multi-branch feature extraction module, and a fusion output layer; and the parameters in the model are initialized, including at least feature embedding parameters, attention weight parameters, feedforward network parameters, multilayer perceptron parameters, and fusion output layer parameters.

[0050] Step 5, Model Training:

[0051] The training set samples are input into the model, and the model is trained using supervised learning, specifically including:

[0052] The input samples are converted into embedded representations using a feature segmenter.

[0053] The embedded representation is input into the multi-branch feature extraction module for feature extraction;

[0054] Calculate the loss function based on the difference between the model output and the corresponding true label;

[0055] Based on the loss function, the model parameters are updated using the backpropagation algorithm.

[0056] The model training process employs a gradient optimization algorithm for parameter updates.

[0057] Step 6, Model Validation and Stopping Training:

[0058] During model training, the model is periodically validated using a validation set. When the model performance on the validation set reaches a preset threshold or no performance improvement is observed in several consecutive training rounds, model training is stopped, and the current model parameters are saved as the target prediction model.

[0059] Step 7, Model Testing and Application:

[0060] Apply the target prediction model obtained in step 6 to the test set or the actual data to be predicted, and output the corresponding classification results or regression prediction results.

[0061] Based on the above, this invention also proposes an application of a tabular data prediction model based on a multi-branch parallel transformer in classification or regression tasks.

[0062] The tabular data prediction model based on multi-branch parallel transformer disclosed in this invention has the following beneficial effects:

[0063] By combining intra-sample attention and inter-sample attention, information interaction between samples is better realized, introducing nearest neighbor information into the model; multiple transformer and multilayer perceptron branches can better handle heterogeneous features, enrich the diversity of model feature extraction, and ultimately improve the model's predictive ability. Attached Figure Description

[0064] To more clearly illustrate the technical inventions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a simplified structural diagram of the classic transformer model;

[0066] Figure 2 This is a schematic diagram illustrating the mechanisms of intra-sample attention and inter-sample attention.

[0067] Figure 3 This is a schematic diagram of the network structure of the multi-branch parallel transformer prediction model proposed in this invention. Detailed Implementation

[0068] The invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the invention are all common knowledge and general knowledge in the art, and the invention does not have any particular limitations.

[0069] This invention proposes a multi-branch parallel transformer tabular data prediction model. In the transformer branch, the in-sample attention mechanism and the inter-sample attention mechanism are used in combination, which can better handle the information interaction between samples and effectively capture the similarity information between samples. Through multiple parallel transformer branches and multilayer perceptron branches, the model can effectively and accurately handle the heterogeneous features in tabular data, and improve the accuracy of the model in tabular data classification and regression tasks.

[0070] Example 1: Classification Task Prediction

[0071] This embodiment uses Bank Marketing (https: / / archive.ics.uci.edu / dataset / 222 / bank+marketing) data as an example to illustrate how to use the method proposed in this invention for classification task prediction.

[0072] Data information and preprocessing

[0073] The bank marketing data relates to a direct marketing campaign (telemarketing) of a Portuguese bank. The campaign involved multiple phone calls to the same customer to determine if they would subscribe to the bank's time deposit products. Therefore, the prediction task is a classification task, predicting whether a customer will subscribe to a time deposit, thus predicting customer behavior. The dataset contains 45,211 samples and 16 feature columns, including 7 numerical features (age, balance, day, duration, campaign, pdays, previous) and 9 categorical features (job, marital, education, default, housing, loan, contact, month, poutcome).

[0074] The dataset was divided into a training set (28,934 samples), a validation set (7,234 samples), and a test set (9,043 samples) in a ratio of 0.64:0.16:0.20. The training set was used to train the model, the validation set was used to find the optimal hyperparameters, and the test set was used to test the model's prediction performance. The QuantileTransformer function from the Scikit-learn library was used to normalize the data. For missing values, missing values ​​for numerical features were filled with the mean of the feature column, while missing values ​​for categorical features were filled with null.

[0075] Model training and testing

[0076] Set the model's hyperparameters to , , This consists of three groups of intra-sample attention transformers and inter-sample attention transformers, each with three transformer branches and one multilayer perceptron branch. Within the transformer branches, the multi-head attention mechanism has eight heads, with embeddings representing lengths. In the multilayer perceptron branch, the coefficients for dimensionality increase... The training learning rate was 0.0008, and the weight decay was 1e-5.

[0077] Because the Bank Marketing data is a classification task, the model outputs [sample number]. Number of categories The algorithm will use softmax to obtain the probability value of each category, and then take the category with the higher probability as the final output label.

[0078] Fifteen random seeds were set up for training and testing, and the average of the 15 test results was taken as the final result. The Adamw optimizer was used for training, with a maximum of 200 training epochs.

[0079] For classification tasks, accuracy is used as the evaluation metric for the model (the higher the better). The specific calculation method is shown below:

[0080]

[0081] Wherein, TP (True Positive) is the number of positive classes correctly predicted as positive, TN (True Negative) is the number of negative classes correctly predicted as negative, FP (False Positive) is the number of negative classes incorrectly predicted as positive, and FN (False Negative) is the number of positive classes incorrectly predicted as negative.

[0082] The model proposed in this invention has an accuracy of 0.9096 on the Bank Marketing dataset.

[0083] In comparison, the FT-Transformer model proposed in reference [1] was used to predict the Bank Marketing dataset. Using the same data preprocessing and training process as described above, and running 15 random seeds and averaging, the final test accuracy of FT-Transformer was 0.9095. It can be seen that the model proposed in this invention outperforms FT-Transformer (0.9096 > 0.9095).

[0084] Example 2: Regression Task Prediction

[0085] This embodiment uses California Housing (Sparse spatial autoregressions) data as an example to illustrate how to use the method proposed in this invention for regression task prediction.

[0086] Data information and preprocessing

[0087] This dataset collects relevant variable information for all neighborhood clusters in California from the 1990 census. In this sample, each neighborhood cluster contains an average of approximately 1425.5 residents living in a geographically compact area. Generally, the geographical area contained is inversely proportional to population density. The dataset aims to predict home prices for all neighborhood clusters in California based on the provided information. The dataset contains 20,640 samples and eight feature columns, all of which are numerical features (median income, housing median age, total rooms, total bedrooms, population, households, latitude, longitude).

[0088] The dataset was divided into a training set (13209 samples), a validation set (3303 samples), and a test set (4128 samples) in a ratio of 0.64:0.16:0.20. The training set was used to train the model, the validation set was used to find the optimal hyperparameters, and the test set was used to test the model's prediction performance. The QuantileTransformer function from the Scikit-learn library was used to normalize the data. For missing values, missing values ​​for numerical features were filled with the mean of the feature column, while missing values ​​for categorical features were filled with null.

[0089] Model training and testing

[0090] Set the model's hyperparameters to , , This consists of three groups of intra-sample attention transformers and inter-sample attention transformers, each with three transformer branches and one multilayer perceptron branch. Within the transformer branches, the multi-head attention mechanism has eight heads, with embeddings representing lengths. In the multilayer perceptron branch, the coefficients for dimensionality increase... The training learning rate was 0.0008, and the weight decay was 1e-5. This embodiment uses the same hyperparameters as Embodiment 1. In practice, customized tuning can be performed to achieve better results.

[0091] Because the California Housing data is a regression task, the model output [sample number] Number of categories This is the final predicted housing price.

[0092] Fifteen random seeds were set up for training and testing, and the average of the 15 test results was taken as the final result. The Adamw optimizer was used for training, with a maximum of 200 training epochs.

[0093] The root mean squared error (RMSE) is used as the evaluation metric for regression tasks (the smaller the better). The specific calculation method is as follows:

[0094]

[0095] in, For the sample size, For the first The true value of each sample For the first The model prediction value for each sample.

[0096] The model proposed in this invention achieved an RMSE of 0.4562 on the California Housing dataset.

[0097] In comparison, the FT-Transformer model proposed in reference [1] was used to predict the California Housing dataset. Using the same data preprocessing and training process as described above, and running 15 random seeds and calculating the mean, the final test RMSE of FT-Transformer was 0.4590. It can be seen that the model proposed in this invention outperforms FT-Transformer (0.4562 < 0.4590).

[0098] References

[0099] 【1】 Gorishniy Y., Rubachev I., Khrulkov V., Babenko A. RevisitingDeep Learning Models for Tabular Data. Advances in Neural Information Processing Systems (NeurIPS), 2021.

[0100] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0101] As used in this invention, the term "comprising" is an open-ended expression, meaning it includes the contents specified in this invention but does not exclude other aspects.

[0102] As used in this invention, the term "and / or" includes any one or more of the related listed items and all combinations thereof.

[0103] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.

Claims

1. A tabular data prediction model based on a multi-branch parallel transformer, characterized in that, include: The system consists of an input layer, a multi-branch feature extraction module, and a fusion output layer; among which, The input layer consists of a feature segmenter, which does not use CLS tokens and is used to segment the input features. Convert to embedded representation , The length of the embedded representation, To input the number of columns in the table data, , The number of columns for numerical features. The number of columns for categorical features; The multi-branch feature extraction module includes a parallel transformer branch and a multilayer perceptron branch; the transformer branch consists of... The multilayer perceptron is composed of alternating stacked attention transformer layers within and between samples; the multilayer perceptron branch consists of three linear layers and an activation function. The fusion output layer is used to combine the outputs of the transformer branch and the multilayer perceptron branch along the column number. The dimensions are joined, and after average pooling to compress the dimensions, the prediction results are output through the prediction head.

2. The tabular data prediction model based on multi-branch parallel transformer according to claim 1, characterized in that, For a given feature Its embedding representation is calculated as follows: in, It is the first Bias terms for each feature, Embedding functions are categorized into numerical feature embedding functions. and categorical feature embedding function : By weight Element-wise multiplication is implemented; By looking up the table Implementation, in which Indicates the first The number of values ​​for each category feature; The embedding representations for numerical features and categorical features are as follows: in, It is the one-hot vector corresponding to the categorical feature. The number of columns for numerical features. The number of columns for categorical features, and . After the feature segmentation unit, the original two-dimensional input [number of samples] , number of columns ] becomes three-dimensional [sample number] , number of columns Embedding length ],in The number of samples for the input table data.

3. The tabular data prediction model based on multi-branch parallel transformer according to claim 1, characterized in that, In the transformer branch: In-sample attention transformer for input [number of samples] , number of columns Embedding length Each sample [number of columns] in ] Embedding length The different columns interact with each other through multi-head attention, enabling attention calculation between different feature columns within the sample; Before entering the transformer, the inter-sample attention transformer first processes the input [number of samples]. , number of columns Embedding length The order is rearranged to [1, number of samples]. , number of columns Embedding length ]; rearrange [1, sample number , number of columns Embedding length The data is input into a multi-head attention system, where different samples interact through this system to calculate attention between samples. After multi-head attention, the data is rearranged into [number of samples]. , number of columns Embedding length The shape of the image is used to enter the feedforward network module; Hybrid attention transformer parallel arrangement These three elements together form the transformer branch; The value range is [1, 10], with an interval of 1; finally A hybrid attention transformer branch passes through the number of columns Connect along the dimension to obtain the output [number of samples]. , number of columns Embedding length ].

4. The tabular data prediction model based on multi-branch parallel transformer according to claim 1, characterized in that, Each branch of the multilayer perceptron consists of three linear layers and an activation function, namely an up-dimensionality layer, a mapping layer, and a down-dimensionality layer. The activation function uses the rectified linear unit function. After passing through a multilayer perceptron branch, the input [number of samples] , number of columns Embedding length The change is: [sample number] , number of columns Embedding length ]→[sample size] , number of columns Embedding length ]→[sample size] , number of columns Embedding length ]; Multilayer perceptrons arranged in parallel These elements together form the multilayer perceptron branch. The value range is [1, 10], with an interval of 1; finally A multilayer perceptron branch passes through the column number Connect along the dimension to obtain the output [number of samples]. , number of columns Embedding length ].

5. The tabular data prediction model based on multi-branch parallel transformer according to claim 1, characterized in that, The processing steps of the fused output layer include: The output of the multi-branch feature extraction module, i.e., the output of the transformer branch [number of samples] , number of columns Embedding length The output of the multilayer perceptron branch [number of samples] , number of columns Embedding length ], along the column number Connect the dimensions to obtain [number of samples]. , number of columns Embedding length ]; After the average pooling layer, along the column number Average pooling is performed on the dimension to obtain [number of samples].

1. Embedding length ], then compress the number of columns. This dimension yields [number of samples] Embedding length At this point, each sample is represented as a string of length […]. ; [Sample Number] Embedding length The input is processed through a prediction head, resulting in the predicted output. The prediction head is a linear layer with an input dimension of [missing information]. The output dimension is the dimension of the prediction result. The output shape is [number of samples] Number of categories ].

6. A method for constructing and training a tabular data prediction model based on a multi-branch parallel transformer, characterized in that, Includes the following steps: Step 1: Collection and labeling of sample data. Obtain the original sample dataset containing feature columns and predicted target labels. The feature columns include numerical features and / or categorical features, and the predicted target labels are categorical labels or continuous numerical labels. Step 2: Data preprocessing, including normalization / standardization, encoding, missing value imputation, and / or outlier handling of the sample dataset; Step 3: Dataset partitioning. The preprocessed dataset is divided into training set, validation set, and test set. Step 4: Model construction and initialization. Construct the model network as described in any one of claims 1-5, and initialize the parameters of feature embedding, attention weights, feedforward network, multilayer perceptron, and fusion output layer. Step 5: Model training. Input the training set samples into the model and train it using supervised learning. Step 6: Model validation and training stoppage. Periodically validate the model using the validation set. When the performance reaches the preset threshold or there is no improvement for several consecutive rounds, stop training and save the model parameters. Step 7: Model testing and application. Apply the saved model to the test set or the actual data to be predicted, and output the classification results or regression prediction results.

7. The construction training method according to claim 6, characterized in that, Step 2 preprocessing specifically includes: Normalize or standardize the numerical features; Encode categorical features; For numerical features, missing values ​​are filled with the mean or median; for categorical features, missing values ​​are filled with a preset default value. Truncate or smooth out outliers.

8. The construction training method according to claim 6, characterized in that, In step 5, the training includes: The input samples are converted into embedded representations using a feature segmenter. The embedded representation is input into the multi-branch feature extraction module for feature extraction; Calculate the loss function based on the difference between the model output and the corresponding true label; Based on the loss function, the model parameters are updated using the backpropagation algorithm; The model training process employs a gradient optimization algorithm for parameter updates.

9. The construction training method according to claim 6, characterized in that, The hyperparameters for model training include: The number of multi-head attention heads is 8, and the embedding length is... The learning rate is 0.0008, the weight decay is 1e-5, and the maximum number of training rounds is 200.

10. An application of a tabular data prediction model based on a multi-branch parallel transformer as described in any one of claims 1-5 in classification or regression tasks.

Images