A tabular data classifier based on Siamese networks and its applications

Abstract

This invention discloses a tabular data classifier based on Siamese networks, comprising: two sub-networks with shared weights; during training, each of the two sub-networks receives one sample from a pair of samples, calculates a classification loss and a contrastive loss, and obtains a final loss function by combining L2 regularization constraints, and trains the network based on the final loss function; during inference, each of the sub-networks processes the input sample and outputs a classification result. This invention constructs paired training datasets and trains the model using two sub-networks with shared weights, enabling the model to fully perceive the similarity information between samples during the learning process, effectively modeling the complex feature relationships in tabular data. The PLR ​​embedding network designed for numerical features can fully mine the potential information of numerical features, and combined with the appropriate processing of categorical features, further improves the model's ability to model tabular data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, in particular to supervised learning, tabular data classification, deep learning, Siamese network and contrastive learning technology, and more particularly to a tabular data classifier based on Siamese network and application thereof. BACKGROUND

[0002] Tabular data refers to a data structure organized in a two-dimensional table form, usually composed of rows and columns, each row representing a sample (or observation instance), and each column representing a feature (or variable). It belongs to structured data, has clear field definition and data type, and is convenient for storage, query and analysis, and is widely used in finance, medicine, marketing, social science and industrial analysis.

[0003] In machine learning tasks, classification based on tabular data is a typical supervised learning scenario. Traditional methods such as logistic regression, decision tree, random forest and gradient boosting machine (such as XGBoost, LightGBM) have been excellent in this field for a long time. In recent years, the development of deep learning has promoted the rise of neural network models designed specifically for tabular data. Literature 【1】 proposes a deep learning model MNCA based on neighborhood component analysis (NCA), which combines the neighborhood component analysis method with the deep learning model and introduces the neighborhood information for feature extraction.

[0004] Siamese network is a special neural network structure first proposed by Bromley et al. in 1993, originally used for handwritten signature verification tasks. Its input consists of pairs of samples, aiming to learn a metric space that maps similar samples to close positions and dissimilar samples to distant positions. Siamese network usually consists of two weight-shared sub-networks, each of which extracts feature vectors from input samples. The similarity of samples is determined by calculating the distance between feature vectors. The core of Siamese network is to learn a generalizable similarity metric rather than directly outputting class labels.

[0005] However, tabular data often contains complex feature relationships and noise, and existing deep learning methods are difficult to fully model them, resulting in classification accuracy that needs to be improved and cannot meet the high requirements of classification accuracy in practical applications. SUMMARY

[0006] In view of the technical defect that the classification accuracy of the table data classification model in the prior art is not high due to the difficulty in fully modeling complex feature relationships and noise, the present invention provides a table data classifier based on a siamese network, which improves the classification accuracy of table data by optimizing the network structure and loss function design.

[0007] The table data classifier based on the siamese network of the present invention includes two sub-networks with shared weights. During training, the two sub-networks respectively receive one sample from a pair of samples, and the final loss function is obtained by calculating the classification loss, contrast loss and combining L2 regularization constraints, and the network is trained based on the final loss function; during inference, only a single sub-network is used to process the input sample to obtain the classification result. Among them, the classification loss uses the cross-entropy loss function, which is applicable to binary classification or multi-classification tasks. The calculation formula in the binary classification scenario is:

[0008]

[0009] Where is the true label of the sample, is the predicted label of the model.

[0010] The calculation formula of the contrast loss is:

[0011]

[0012] Where , indicates whether the pair of samples is similar or matched, represents that the samples are similar or matched, represents that the samples are not similar or not matched; is a preset threshold, which is used to determine the minimum distance by which negative samples are penalized, because only when the distance of the negative sample <m will it be penalized; therefore The size of depends on the distance scale of the negative samples in the training set, and it can be anything from <1 (e.g., 0.5) to >1 (e.g., 2, 5, 10, etc.). When that is, when the sample pair is similar, , which is used to minimize the L2 distance of similar samples; when that is, when the sample pair is not similar, , which is used to maximize the L2 distance of dissimilar samples.

[0013] Adding the calculated classification loss and the contrast loss and applying a certain degree of L2 regularization constraint, the final loss function is obtained:

[0014] Where is the network weight, This represents the strength of the regularization term.

[0015] In this invention, the structure of the sub-network is as shown in the attached figure. Figure 2 As shown, it can be divided into three main parts: the embedding module, the feature extraction module, and the output module. Among them,

[0016] Embedding module: Consists of feature embedding layers, used to process numerical and categorical features in the input table data respectively. The numerical features are embedded through the PLR ​​embedding network proposed in reference [2], and the structure is shown in the appendix. Figure 3 As shown, this is processed using a PLR embedding network based on a periodic activation function. The formula for calculating the periodic activation function is:

[0017]

[0018] in, For trainable parameters, from a normal distribution Mid-sampling initialization, and The parameters are selected (tuned) through the validation set; the numerical features are processed by a periodic activation function, and then the embedding result is obtained through a linear mapping and a linear rectified unit (ReLU); the processed numerical feature embedding result is rearranged and concatenated with the categorical feature along the column dimension to obtain the output of the embedding module.

[0019] Assume the input table data is in the shape of [number of samples] , number of columns ], that is, shared There are 100 samples, and each sample has 100 samples. The features are dimensional, where numerical features (features are continuous numerical values) have a dimension of 1. Categorical features (features are discrete numerical values ​​or text) have a dimension of 1. ,and Then the shape of the embedded numerical features is [number of samples]. , number of columns , ]( (where the embedding length is the numerical feature embedding length), after rearrangement it is [number of samples]. , number of columns ], and categorical features [number of samples] , number of columns After splicing, the embedded module outputs a shape of [number of samples]. , number of columns ].

[0020] Feature extraction module: consists of a linear mapping layer (Linear, with an output dimension of...) It consists of a feature encoder and a batch normalization layer (BatchNormal), wherein the feature encoder contains There are 1 feature encoding layer, each feature encoding layer is batch normalized and linearly mapped (output dimension is ). ), Linear Rectified Unit (ReLU), Dropout, and Linear Mapping (output dimension is...) The sequence of ) is arranged as follows, and the calculation process of the feature coding layer is as follows:

[0021]

[0022] The output dimension of the linear mapping layer is The output dimensions of the linear mapping in the feature encoding layer are respectively and After processing by the feature extraction module, the output shape is [number of samples]. , ].

[0023] Output module: Consists of a classification head, which is a linear layer with an input dimension of [missing information]. The output dimension is the number of categories. The final output shape is [number of samples] Number of categories ].

[0024] This invention also proposes an application of a Siamese network-based tabular data classifier in tabular data classification tasks.

[0025] The tabular data classifier based on Siamese networks of the present invention has the following beneficial effects:

[0026] 1. By constructing paired training datasets and training two sub-networks with shared weights, the model can fully perceive the similarity information between samples during the learning process, effectively modeling the complex feature relationships in tabular data.

[0027] 2. A loss function design combining classification loss and contrastive loss is adopted. Classification loss ensures the model's classification prediction ability, while contrastive loss can maximize the distance between samples of different categories and minimize the distance between samples of the same category, thereby improving the model's ability to distinguish feature differences. At the same time, L2 regularization constraint effectively reduces the risk of model overfitting.

[0028] 3. The PLR ​​embedding network is designed for numerical features, which can fully explore the potential information of numerical features. Combined with the reasonable processing of categorical features, it can further improve the model's ability to model tabular data.

[0029] 4. Experimental results show that the classifier of this invention has a higher accuracy in tabular data classification tasks than existing deep learning methods, and can better adapt to noisy tabular data classification scenarios, thus having broad application value. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the network structure of the tabular data classifier based on Siamese networks according to the present invention;

[0032] Figure 2 This is a schematic diagram of the subnetwork structure in the twin network of the present invention;

[0033] Figure 3 This is a schematic diagram of the numerical feature embedding network in the sub-network of the present invention;

[0034] Figure 4 This is a schematic diagram of the feature encoder structure in the sub-network of the present invention. Detailed Implementation

[0035] The invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the invention are all common knowledge and general knowledge in the art, and the invention does not have any particular limitations.

[0036] Example 1: Customer Behavior Prediction

[0037] This embodiment uses the BankMarketing dataset as an example to illustrate in detail the implementation process of the tabular data classifier based on Siamese networks of the present invention.

[0038] 1. Data Information and Preprocessing

[0039] The Bank Marketing dataset, sourced from https: / / archive.ics.uci.edu / dataset / 222 / bank+marketing, relates to a telemarketing campaign conducted by a Portuguese bank. The campaign involved multiple phone calls to the same customer to determine if they would subscribe to the bank's time deposit products. Therefore, the task is a binary classification task predicting whether a customer will subscribe to the bank's time deposits. The dataset contains 45,211 samples and 16 feature columns, including 7 numerical features (age, balance, day, duration, campaign, pdays, previous) and 9 categorical features (job, marital, education, default, housing, loan, contact, month, poutcome).

[0040] The dataset was divided into a training set (28,934 samples), a validation set (7,234 samples), and a test set (9,043 samples) in a ratio of 0.64:0.16:0.20. The training set was used for model training, the validation set for hyperparameter optimization, and the test set for model performance evaluation. The QuantileTransformer function from the Scikit-learn library was used to normalize the data. For missing values, missing values ​​for numerical features were filled with the mean of the feature column, and missing values ​​for categorical features were filled with null.

[0041] 2. Construction of Paired Training Datasets

[0042] To train a Siamese network, a training set of paired samples needs to be constructed. The specific process is as follows:

[0043] Constructing pairs of similar samples: Bank Marketing data is a binary classification problem, i.e., the class labels are {0,1}. Therefore, samples from class 0 and class 1 are paired to obtain a set of pairs of similar samples. To save memory, the maximum number of sample pairs is set to max_sample=100000. If the number of sample pairs exceeds max_sample, then max_sample samples are randomly selected as the final set of sample pairs of the same type. .

[0044] Heterogeneous sample pair construction: Based on the category with more samples, construct heterogeneous sample pairs. That is, if there are more samples in class 0, randomly match each sample in that class with a sample from the other class. Then, for each sample in class 0, randomly match a sample from class 1, resulting in a set of heterogeneous sample pairs. Similarly, the maximum number of samples in the set of out-of-class samples is limited as described above.

[0045] Will and The dataset is used as the training dataset for the Siamese network. During training, subnetwork 1 receives the first sample from the sample pair, and subnetwork 2 receives the second sample from the sample pair.

[0046] 3. Model parameter settings and training

[0047] Use Optuna (Reference: Optuna:

[0048] A next-generation hyperparameter optimization framework was used. The network's hyperparameters were selected using a validation set and 100 trials were conducted. The following hyperparameter settings were ultimately determined:

[0049] Embedded module: , , The numerical features, after being processed by a periodic activation function, output a shape of [number of samples]. , number of columns 8, ];

[0050] Feature extraction module: (i.e., one feature coding layer) dropout=0.1 ;

[0051] Contrast Loss: Threshold ;

[0052] Final loss function: regularization term strength ;

[0053] Training parameters: learning rate = 0.0001, optimizer is Adamw, maximum number of training epochs = 200, batch size = 256.

[0054] During model training, based on the final loss function Backpropagation updates the network weights, and the model output is processed by the sigmoid function to obtain the probability that the sample is a positive class.

[0055] 4. Model Testing and Result Comparison

[0056] Fifteen random seeds were used for model training and testing, and the average of the 15 test results was taken as the final performance metric. The Adamw optimizer was used for training, with a maximum of 200 epochs and a batch size of 256. Accuracy was used to evaluate the model's classification performance, calculated using the following formula:

[0057]

[0058] Wherein, TP (True Positive) is the number of positive classes correctly predicted as positive, TN (True Negative) is the number of negative classes correctly predicted as negative, FP (False Positive) is the number of negative classes incorrectly predicted as positive, and FN (False Negative) is the number of positive classes incorrectly predicted as negative.

[0059] Test results show that the tabular data classifier based on Siamese networks of this invention achieves an accuracy of 0.9076 on the Bank Marketing dataset. Furthermore, the classification accuracy is as follows for different values ​​of the threshold m: m=1, 0.9053; m=5, 0.9066; m=15, 0.9068; m=20, 0.9068.

[0060] In comparison, experiments were conducted using the MNCA model proposed in reference [1] with the same dataset, data preprocessing, and training process. The model was run with 15 random seeds and the mean was taken, resulting in a test accuracy of 0.9057. Experimental results demonstrate that the classifier of this invention outperforms the existing MNCA model in tabular data classification tasks, verifying the effectiveness and superiority of this invention.

[0061] References

[0062] 【1】REVISITING NEAREST NEIGHBOR FOR TABULAR DATA: A DEEP TABULARBASELINE TWO DECADES LATER

[0063] 【2】 On Embeddings for Numerical Features in Tabular Deep Learning

[0064] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0065] As used in this invention, the term "comprising" is an open-ended expression, meaning it includes the contents specified in this invention but does not exclude other aspects.

[0066] As used in this invention, the term "and / or" includes any one or more of the related listed items and all combinations thereof.

[0067] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.

Claims

1. A tabular data classifier based on Siamese networks, characterized in that, include: Two subnetworks with shared weights; During training, each of the two sub-networks receives one sample from a pair of samples. The final loss function is obtained by calculating the classification loss, the contrastive loss, and combining it with L2 regularization constraints. The network is then trained based on the final loss function. During inference, the input sample is processed by a single sub-network, and the classification result is output.

2. The tabular data classifier based on Siamese networks according to claim 1, characterized in that, The classification loss uses the cross-entropy loss function, and the calculation formula for the binary classification scenario is as follows: in, For the true labels of the samples, Predict labels for the model.

3. The tabular data classifier based on Siamese networks according to claim 1, characterized in that, The formula for calculating the contrast loss is: in, , Indicates whether paired samples are similar or matched. This indicates that the samples are similar or matched. This indicates that the samples are dissimilar or mismatched. This is a preset threshold.

4. The tabular data classifier based on Siamese networks according to claim 1, characterized in that, The final loss function is the sum of classification loss, contrastive loss, and L2 regularization term, and is calculated using the following formula: in, For network weights, This represents the strength of the regularization term.

5. The tabular data classifier based on Siamese networks according to claim 1, characterized in that, Each of the sub-networks includes an embedding module, a feature extraction module, and an output module. The output of the embedding module is connected to the input of the feature extraction module, and the output of the feature extraction module is connected to the input of the output module.

6. The tabular data classifier based on Siamese networks according to claim 5, characterized in that, The embedding module is used to process numerical features and categorical features in the input table data. Specifically, it includes: embedding the numerical features and rearranging them, then concatenating the rearranged numerical feature embedding result with the categorical feature along the column dimension to obtain the output of the embedding module. The format of the input table data is [number of samples] , number of columns ], where the numerical feature dimension is The categorical feature dimension is ,and The embedded module outputs a shape of [number of samples]. , number of columns ], The embedding length of the numerical feature.

7. The tabular data classifier based on Siamese networks according to claim 6, characterized in that, The numerical features are embedded using a PLR embedding network based on a periodic activation function, the formula for which the periodic activation function is calculated is: Among them, among them, For trainable parameters, from a normal distribution Mid-sampling initialization, and Parameters are selected using a validation set; numerical features are processed by a periodic activation function and then sequentially passed through a linear mapping and a linear rectifier unit to obtain the embedding result.

8. The tabular data classifier based on Siamese networks according to claim 5, characterized in that, The feature extraction module includes a linear mapping layer, a feature encoder, and a batch normalization layer. The feature encoder contains L feature encoding layers, each arranged in the order of batch normalization, linear mapping, linear rectifier unit, Dropout, and linear mapping. The calculation process of the feature encoding layer is as follows: The output dimension of the linear mapping layer is The output dimensions of the linear mapping in the feature encoding layer are respectively and After processing by the feature extraction module, the output shape is [number of samples]. , ]; And / or, The output module consists of a classification head, which is a linear layer with an input dimension of [missing information]. The output dimension is the number of categories. The output shape is [number of samples] Number of categories ].

9. The tabular data classifier based on Siamese networks according to claim 1, characterized in that, The pairwise training set used during training is constructed in the following way: Construction of similar sample pairs: Combine samples from each category in the dataset in pairs to obtain a set of similar sample pairs. ,like If the number of sample pairs exceeds the preset maximum value max_sample, then max_sample samples are randomly selected; Construction of dissimilar sample pairs: Taking the category with the larger number of samples as a benchmark, a sample from another category is randomly matched for each sample in that category, resulting in a set of dissimilar sample pairs. ,like If the number of sample pairs exceeds max_sample, then max_sample pairs of samples are randomly selected; and The dataset is combined and used as the training dataset for the Siamese network.

10. An application of the tabular data classifier as described in any one of claims 1-9 in a tabular data classification task.

Images