A tabular data classifier based on deep support vector machines, its classification method, and its application.
By combining deep learning and support vector machines in a tabular data classifier, the classification boundary is maximized, solving the problem of low classification accuracy in existing technologies and achieving higher classification accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI QUSU CHAOWEI TECHNOLOGY CO LTD
- Filing Date
- 2026-01-23
- Publication Date
- 2026-06-02
AI Technical Summary
Existing deep learning-based tabular data classification models have low classification accuracy under complex classification boundaries and cannot maximize the classification boundary.
By combining deep learning models with support vector machines, and through input embedding, feature extraction and output modules, a support vector machine loss function is introduced to maximize the classification boundary and improve classification accuracy.
It effectively solves the problem of classification accuracy under complex classification boundaries and improves the accuracy of binary and multi-class classification tasks for tabular data.
Smart Images
Figure CN122132932A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of machine learning and data analysis technology, and in particular to a tabular data classifier, classification method and application based on deep support vector machines. Background Technology
[0002] Tabular data is the most common data format in machine learning and data analysis. It consists of rows (samples) and columns (features). Each row represents an independent observation sample, and each column represents a well-defined feature (such as age, price, etc.). This structured nature allows tabular data to be processed and analyzed efficiently by computers, making it a core material for training machine learning models (such as classification tasks). Its classic applications include business analytics and intelligence (such as customer relationship management, financial risk control and credit), and scientific research and discovery (drug development, medical diagnosis, etc.).
[0003] Classification based on tabular data is a classic problem in supervised learning. The core objective is to predict the predefined category or label for each sample based on known features (input) in the tabular data. The model learns the mapping relationship between features and labels from the training data, outputs the predicted probability for each category, and uses the one with the highest probability as the final classification result. This technology is key to achieving automated decision-making and intelligent judgment, with applications spanning multiple fields such as spam filtering, financial risk assessment, and assisted medical diagnosis.
[0004] For a long time, traditional machine learning methods have dominated tabular data modeling. With the advancement of deep learning technology, more and more neural network-based models have been tried to process tabular data and have shown promising performance. Reference [1] proposed a deep learning model MNCA based on Neighborhood Component Analysis (NCA), which combines the NCA method with the deep learning model to introduce neighbor information for feature extraction. However, existing deep learning-based tabular data classification models generally have a drawback: for complex classification boundaries, they cannot maximize the classification boundary, resulting in low classification accuracy.
[0005] To address this issue, the present invention combines the classic support vector machine method with a deep learning model, introducing the constraint of the maximum classification boundary into the deep learning model to solve the aforementioned technical problems. Summary of the Invention
[0006] To address the shortcomings of existing deep learning-based tabular data classification models, such as their inability to maximize complex classification boundaries and low classification accuracy, this invention aims to provide a tabular data classifier based on deep support vector machines. By integrating the advantages of deep learning models and support vector machines, this invention maximizes the classification boundary and improves the accuracy of tabular data classification.
[0007] The tabular data classifier based on deep support vector machines proposed in this invention includes: an input embedding module, a feature extraction module, and an output module connected in sequence; wherein,
[0008] Suppose the input table data has the shape of [number of samples n, number of columns k], meaning there are n samples in total, and each sample has k-dimensional features, including numerical features (features are continuous numerical values). Dimensionality and categorical features (features are discrete numerical values or text) Wei, that is .
[0009] 1. Input embedded module:
[0010] The input embedding module mainly consists of a feature embedding layer, which processes numerical features and categorical features separately.
[0011] For numerical features, the feature embedding layer adopts the PLR embedding network (an embedding network based on a periodic activation function) proposed in reference [2], where the calculation formula of the periodic activation function is:
[0012] ;
[0013] In the formula, It is a set of trainable parameters derived from a normal distribution. Sampling initialization is performed in the process, and f and σ are obtained by parameter selection (tuning) through the validation set; Input features The frequency mapping vector obtained after linear transformation and scaling; The input is a numerical feature.
[0014] After numerical features are processed by a periodic activation function, the original numerical feature input [number of samples n, number of columns] is... ] will become [number of samples n, number of columns] After passing through a linear mapping and linear rectifier unit, the embedded features [number of samples n, number of columns] are finally obtained. , ],in is the embedding length of the numerical feature.
[0015] Embed the processed numerical features into [number of samples n, number of columns]. , Rearranged into [number of samples n, number of columns] × ], and categorical features [number of samples n, number of columns] The input embedding module's output is obtained by concatenating along the column dimension [number of samples n, number of columns]. × + ].
[0016] 2. Feature Extraction Module:
[0017] The feature extraction module is used to extract key features from the output of the input embedding module. It mainly consists of a linear mapping (with an output dimension of d), L feature encoding layers, and a batch normalization layer, where L ranges from [1, 10].
[0018] Each feature encoding layer consists of batch normalization, the first linear mapping (output dimension is...) in sequence. The feature encoding layer consists of a Rectified Linear Unit (ReLU), Dropout, and a second linear mapping (Linear, with an output dimension of d). For the input x of the feature encoding layer, the calculation process is as follows:
[0019] ,
[0020] After processing by the feature extraction module, the output is [number of samples n, d]. The range of values for d is [32, 64, 128, 256, 512, 1024].
[0021] 3. Output module:
[0022] The output module consists of a classification head and a support vector machine loss function (SVMLoss).
[0023] The classification head is a linear layer with an input dimension of . The output dimension is the number of categories. Therefore, the output shape is [number of samples n, number of categories]. ].
[0024] During training, the output is [number of samples n, number of classes]. After processing with the Support Vector Machine (SVM) loss function, a boundary maximization constraint is introduced to maximize the classification boundary and improve classification accuracy. The hyperparameter of the SVM loss function is C, with values ranging from [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000]. C is obtained through hyperparameter tuning on the validation set and is used to control the strength of the regularization term. The SVM loss function is:
[0025]
[0026] Where N is the total number of training samples, that is, the number of samples in the training set; Let be the true label corresponding to the i-th training sample; f( ) is the model's response to input samples The predicted output; Let be the input feature vector of the i-th training sample.
[0027] Based on the above, this invention also proposes a tabular data classification method based on deep support vector machines, comprising the following steps:
[0028] Step 1, Data Preprocessing: Determine the structure of the input table data as [number of samples n, number of columns k], where, , The dimension of the numerical feature. The dimensions of categorical features are defined; the dataset is divided into training, validation, and test sets; numerical features are normalized and missing values are filled.
[0029] Step 2, Input Embedding Processing: The numerical features and categorical features are processed separately by the input embedding module and then concatenated for output;
[0030] Step 3, Feature Extraction: Extract key features from the output of the input embedding module using the feature extraction module;
[0031] Step 4, Classification Output: The classification result is output through the output module, which includes a classification head and a support vector machine loss function. The support vector machine loss function is used to introduce boundary maximization constraints, which integrate the feature extraction capability of deep learning with the boundary optimization capability of support vector machines to achieve binary or multi-class classification of tabular data.
[0032] Based on the above, this invention also proposes an application of a table data classifier or table data classification method based on deep support vector machine in binary or multi-class classification tasks.
[0033] The binary or multi-classification tasks include business analysis, financial risk control, credit assessment, drug development, medical diagnosis, or pulsar candidate identification.
[0034] The tabular data classifier based on deep support vector machines disclosed in this invention has the following beneficial effects:
[0035] By introducing the support vector machine loss function, the deep classification network is given the constraint of the maximum classification boundary. Combined with the powerful feature extraction capability of the deep network, the classification accuracy problem under complex classification boundaries is effectively solved. It is suitable for binary and multi-class classification tasks of various tabular data, with wide application scenarios and strong practicality. Attached Figure Description
[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0037] Figure 1 This is a schematic diagram of the network structure of the tabular data classifier based on deep support vector machines according to the present invention;
[0038] Figure 2 This is a schematic diagram of the structure of the numerical feature embedding network (PLR embedding network) in this invention. Detailed Implementation
[0039] The invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the invention are all common knowledge and general knowledge in the art, and the invention does not have any particular limitations.
[0040] Example 1: Determining whether a candidate is a real pulsar
[0041] This embodiment uses the HTRU dataset (https: / / archive.ics.uci.edu / dataset / 372 / htru2) as an example to illustrate the specific application process of the tabular data classifier based on deep support vector machine of this invention.
[0042] 1.1 Data Information and Preprocessing:
[0043] The HTRU dataset describes a set of pulsar candidate samples collected in the High Time Resolution Universe Survey (South), with the goal of determining whether a candidate is a real pulsar. The dataset contains 17,898 samples and 8 feature columns, all of which are numerical: Profile_mean, Profile_stdev, Profile_skewness, Profile_kurtosis, DM_mean, DM_stdev, DM_skewness, and DM_kurtosis.
[0044] The dataset was divided into a training set (11454 samples), a validation set (2864 samples), and a test set (3580 samples) in a ratio of 0.64:0.16:0.20. The training set was used to train the model, the validation set was used to find the optimal hyperparameters, and the test set was used to test the model's prediction performance. The QuantileTransformer function from the Scikit-learn library was used to normalize the data; for missing values of numerical features, the mean of the feature column was used for imputation.
[0045] 1.2 Model Training and Testing:
[0046] The model's hyperparameters are set as follows: In the feature embedding layer, f=77, σ=0.0443136. That is, the output shape of the periodic activation function is [number of samples n, number of columns 8, 2×34]; in the feature extraction module, L=1, The model has an embedding length of 512, dropout of 0.1, and d of 128, meaning it contains one feature encoding layer with a dimensionality of 512. A 10% dropout is applied to the dimension, resulting in a final feature length of 128. In the output layer, the hyperparameter C of the support vector machine loss function is 0.1. The training rate is 0.001, and the weight decay is 0.0002. Since the HTRU data corresponds to a binary classification task, the model output is processed by the sigmoid function to obtain the probability that a sample is a positive class.
[0047] Fifteen random seeds were used for model training and testing, and the average of the 15 test results was taken as the final result. The Adamw optimizer was used for training, with a maximum of 200 epochs and a batch size of 256. Accuracy was used as the evaluation metric for the model; higher accuracy indicates better classification performance. The specific calculation method is as follows:
[0048]
[0049] Wherein, TP (True Positive) is the number of positive classes correctly predicted as positive, TN (True Negative) is the number of negative classes correctly predicted as negative, FP (False Positive) is the number of negative classes incorrectly predicted as positive, and FN (False Negative) is the number of positive classes incorrectly predicted as negative.
[0050] The model of this invention achieved an accuracy of 0.9801 on the HTRU dataset.
[0051] As a comparative experiment, the MNCA model proposed in reference [1] was used to classify the HTRU dataset. The same data preprocessing and training process as in this embodiment was used, and 15 random seeds were run and the mean was calculated. The final test accuracy of the MNCA model was 0.9791. It can be seen that the classification result of the model proposed in this invention is better than that of the MNCA model (0.9801 > 0.9791).
[0052] References
[0053] [1] Revisiting Nearest Neighbor for Tabular Data: A Deep TabularBaseline Two Decades Later, Han-Jia Ye, Huai-Hong Yin, De-Chuan Zhan, Wei-LunChao, 2025, arXiv:2407.03257 (https: / / arxiv.org / abs / 2407.03257)
[0054] 【2】Gorishniy, Y., Rubachev, I., & Babenko, A. (2022). On Embeddingsfor Numerical Features in Tabular Deep Learning. In Proceedings of the 36thConference on Neural Information Processing Systems (NeurIPS 2022)( https: / / arxiv.org / abs / 2203.05556)
[0055] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
[0056] As used in this invention, the term "comprising" is an open-ended expression, meaning it includes the contents specified in this invention but does not exclude other aspects.
[0057] As used in this invention, the term "and / or" includes any one or more of the related listed items and all combinations thereof.
[0058] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of the inventive concept are included in this invention and are protected by the appended claims.
Claims
1. A tabular data classifier based on deep support vector machines, characterized in that, include: The input embedding module, feature extraction module, and output module are connected sequentially; among them, The data shape of the input table is [number of samples n, number of columns k]. , The dimension of the numerical feature. Dimensions for categorical features; The input embedding module is used to process numerical features and categorical features separately and then concatenate the output; The feature extraction module is used to extract key features from the output of the input embedding module; The output module includes a classification head and a support vector machine loss function. The support vector machine loss function is used to introduce boundary maximization constraints. The classifier achieves binary or multi-class classification of tabular data by integrating the feature extraction capability of deep learning with the boundary optimization capability of support vector machines.
2. The tabular data classifier based on deep support vector machine according to claim 1, characterized in that, The input embedding module consists of a feature embedding layer. The feature embedding layer processes numerical features using a PLR embedding network. The periodic activation function in the PLR embedding network is calculated using the following formula: ； In the formula, It is a set of trainable parameters; Input features The frequency mapping vector obtained after linear transformation and scaling; For numerical features input; After numerical features are processed by a periodic activation function, they are represented by [number of samples n, number of columns]. ] becomes [number of samples n, number of columns] [2f], then processed by a linear mapping and linear rectifier unit to obtain the embedded features [number of samples n, number of columns]. , ]; The embedding length of the numerical feature; Embed the processed numerical features into [number of samples n, number of columns]. , Rearranged as [number of samples n, number of columns] × After that, with categorical features [number of samples n, number of columns] The input embedding module's output is obtained by concatenating along the column dimension [number of samples n, number of columns]. × + ].
3. The tabular data classifier based on deep support vector machine according to claim 2, characterized in that, The parameters From normal distribution During sampling initialization, f and σ are obtained by parameter selection using the validation set.
4. The tabular data classifier based on deep support vector machine according to claim 1, characterized in that, The feature extraction module consists of a linear mapping, L feature encoding layers and a batch normalization layer; where L ranges from [1, 10] and the output dimension of the linear mapping is d.
5. The tabular data classifier based on deep support vector machine according to claim 4, characterized in that, The feature encoding layer consists of batch normalization, a first linear mapping, a linear rectified unit, Dropout, and a second linear mapping in sequence; the output dimension of the first linear mapping is... The output dimension of the second linear mapping is The calculation process of the feature coding layer is as follows: After processing by the feature extraction module, the output is [number of samples n, d].
6. The tabular data classifier based on deep support vector machine according to claim 1, characterized in that, The classification head is a linear layer with an input dimension of [dimensionality]. The output dimension is the number of categories. The output shape is [number of samples n, number of categories]. ].
7. The tabular data classifier based on deep support vector machine according to claim 1, characterized in that, The formula for calculating the support vector machine loss function is as follows: ， Where C is a hyperparameter with a value range of [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000], used to control the strength of the regularization term; N is the total number of training samples, i.e. the number of samples in the training set; Let be the true label corresponding to the i-th training sample; ) is the model's response to input samples The predicted output; Let be the input feature vector of the i-th training sample.
8. A tabular data classification method based on deep support vector machines, characterized in that, Includes the following steps: Step 1, Data Preprocessing: Determine the structure of the input table data as [number of samples n, number of columns k], where , The dimension of the numerical feature. Dimensions for categorical features; The dataset is divided into training, validation and test sets. Numerical features are normalized and missing values are filled. Step 2, Input Embedding Processing: The numerical features and categorical features are processed separately by the input embedding module and then concatenated for output; Step 3, Feature Extraction: Extract key features from the output of the input embedding module using the feature extraction module; Step 4, Classification Output: The classification result is output through the output module, which includes a classification head and a support vector machine loss function. The support vector machine loss function is used to introduce boundary maximization constraints, which integrate the feature extraction capability of deep learning with the boundary optimization capability of support vector machines to achieve binary or multi-class classification of tabular data.
9. A tabular data classifier based on deep support vector machines as described in any one of claims 1-7, or the application of the method as described in claim 8 in binary or multi-class classification tasks.
10. The application as described in claim 9, characterized in that, The binary or multi-classification tasks include business analysis, financial risk control, credit assessment, drug development, medical diagnosis, or pulsar candidate identification.