Form model cloning method based on generated data

Abstract

The invention discloses a table model cloning method based on generated data. Firstly, a table generator is used for synthesizing table samples based on numerical value and category features, then inverse preprocessing operation is carried out on the synthesized features, and the synthesized numerical value features are converted into a proper range to lead out response of a victim model. Using a contrast table converter (CTT) as a trunk of a clone model, training the model by using the acquired query-response pair, and establishing a clear decision boundary of the CTT by using a data enhancement method and an attention mechanism in combination with contrast loss and cross entropy loss, so that the prediction of the CTT is consistent with the prediction of a victim model; and finally, further optimizing a table generator by taking the output of the CTT as guidance and combining numerical loss, classification loss and joint constraint. According to the method, the form model with an agnostic framework can be cloned in an environment without real data.

Description

Technical Field

[0001] This application belongs to the technical field of model cloning, and particularly relates to a table model cloning method based on generated data. Background Art

[0002] Table data includes various features, such as numerical features and categorical features, and has become a common format for organizing information. To effectively manage this data, many table models are applied to real-world scenarios, such as credit assessment, network intrusion detection, and energy management.

[0003] Despite these remarkable achievements, table models still face significant security threats. For example, there is evidence that adversaries can extract deployed table models ("victim" models) to develop functionally equivalent cloned models. These behaviors, known as table model cloning, may endanger the intellectual property rights of model owners. In addition, adversaries can easily use cloned models to launch further attacks, such as creating adversarial applications for loan fraud, synthesizing adversarial network traffic to attack network facilities, and conducting membership attacks to collect sensitive data, etc.

[0004] Therefore, enhancing the security of table models is crucial. Studying potential cloning methods is an effective strategy to counter these threats. Therefore, some scholars have proposed a table model cloning method based on real data, which queries a specific victim model by collecting real alternative data, and then analyzes the response-query pairs to infer model parameters. However, in most real-world scenarios, it is difficult to collect such alternative data due to limited public information about the victim.

[0005] To reduce this limitation, some scholars have proposed a table model cloning method based on generated data. These methods use handcrafted table samples to induce responses from victim models, and then estimate parameters based on their architectural features, without relying on obtaining real data. However, due to the heterogeneity of table samples, the process of making query samples is both laborious and time-consuming. In addition, most of these methods still require knowledge of the details of the victim architecture, which is difficult to obtain in real-world scenarios. In summary, existing table model cloning methods may not be able to overcome practical constraints and are difficult to be applied in practice. Summary of the Invention

[0006] To solve the above problems, the present invention discloses a table model cloning method based on generated data to overcome the limitations that table model extraction attacks require detailed knowledge of the architecture of the victim model and real training data in real-world applications.

[0007] A table model cloning method based on generated data includes a table generator, a query process, and a contrast table transformer (CTT). The table model cloning method based on generated data includes:

[0008] Synthetic numerical and categorical features are generated from Gaussian noise using a table generator. A customized inverse preprocessing operation is applied to the synthetic features. This operation involves converting the synthetic numerical features to an appropriate range and reordering the features to match the input requirements of the victim model. The transformed synthetic features are combined into query samples to induce responses from the victim model, and these responses are then used to train the CTT.

[0009] During the training process, contrastive loss and cross-entropy loss are used to enable the CTT to establish clear decision boundaries and align its predictions with those of the victim model. The output of the CTT is used as a guide, and numerical loss, classification loss, and joint constraints are combined to optimize the synthesis of table samples.

[0010] Advantages of the present invention:

[0011] 1. The present invention proposes a new table generator for synthesizing feature-related and informative table samples, and designs three loss functions to enhance its generation ability. A new CTT architecture is proposed for table model cloning and effectively handling boundary heterogeneity problems encountered during the cloning process. Experiments have proven that the proposed model is highly effective and can train a cloned model with an accuracy of 92 - 100% of the victim model.

[0012] 2. At the same time, the present invention aims to generate effective data relying on the output of the model without any data, thereby completing model extraction and improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the workflow of the table model cloning method based on generated data of the present invention.

[0014] Figure 2 It is a schematic diagram of the structure of the table generator described in the present invention.

[0015] Figure 3 It is a schematic diagram of the structure of the contrastive table transformer described in the present invention.

[0016] Figure 4 It is a flowchart of the logic of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present invention will be further clarified below in conjunction with the drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. It should be noted that the terms "front", "rear", "left", "right", "up" and "down" used in the following description refer to the directions in the drawings, and the terms "inner" and "outer" refer to the directions towards or away from the geometric center of a specific component, respectively.

[0018] Figure 1 Shows a method for cloning a tabular model based on generated data proposed in this application, including a table generator, a query process, and a contrast table transformer (CTT).

[0019] Among them, the table generator is used to synthesize numerical and categorical features from Gaussian noise. A customized inverse preprocessing operation is applied to the synthesized features. This operation involves converting the synthesized numerical features to an appropriate range and reordering the features to match the input requirements of the victim model. The transformed synthesized features are combined into query samples to induce responses from the victim model, and then these responses are used to train the CTT.

[0020] During the training process, the contrast loss and cross-entropy loss are used to enable the CTT to establish clear decision boundaries and align its predictions with those of the victim model. The output of the CTT is used as a guide, and the numerical loss, classification loss, and joint constraints are combined to optimize the synthesis of tabular samples.

[0021] In one embodiment, a method for cloning a tabular model based on generated data is proposed, including:

[0022] Table generator: The goal of the table generator is to synthesize information-rich and feature-related table samples from Gaussian noise to train the CTT; use J to represent the number of samples and j to represent the sample index.

[0023] Each noise vector z j ∈R F contains F elements, each of which comes from the standard normal distribution. Each table sample contains two parts: a numerical feature vector and a categorical feature vector where N and M represent the number of numerical features and categorical features respectively, and F = N + M. The table generator contains a base encoder, a numerical decoder, and multiple classification decoders; to illustrate the synthesis process in detail, an example of synthesizing a table sample x j from the noise vector z j is provided as follows.

[0024] Base encoder: Table samples in the real world often exhibit feature correlations. To ensure that the synthesized table samples exhibit similar features, a base encoder is designed as the first component of the table generator. The encoder consists of multiple fully connected layers for mapping the input noise vector z j to a semantically rich latent space and uses skip connections to obtain high-level base embeddings. The process of the base encoder can be described as follows:

[0025] Let FC enc(l) represent the first fully - connected layer in the encoder, where \(l\in[1,\ldots,L]\) and \(L\) is the total number of fully - connected layers. The output of each fully - connected layer is given by the following formula:

[0026]

[0027] where is the initial input noise vector, is the base embedding of the output, which is subsequently shared by the numerical and classification decoders to synthesize reasonable and relevant features.

[0028] Numerical decoder: The numerical decoder receives the output of the base encoder as input and outputs numerical features This component consists of multiple fully - connected layers. First, the base embedding is added to the input of each layer to ensure that the knowledge of the base encoder can be effectively propagated to all layers of the numerical decoder. Then, batch normalization is applied after each fully - connected layer to stabilize the training process and improve the quality of the synthesized samples. In addition, the output layer uses the sigmoid function as the activation function to map the output values to the range \([0,1]\).

[0029] Formally described as follows:

[0030]

[0031] Here is the base embedding of the input, and \(BN(\cdot)\) represents the batch normalization operation. \(FC num(l) represents the first layer in the numerical decoder, and \(FC num(L) is the output layer, where the number of neurons is equal to the number \(N\) of numerical features.

[0032] Classification decoder: Considering that each classification feature has a different number of classes, different classification decoders are designed to synthesize different classification features The architecture and input of each classification decoder are similar to those of the numerical decoder, except for the output layer. The output layer of the classification decoder uses the Argmax operation to determine the classification feature values.

[0033] It should be noted that the number of neurons in the output layer of each classification decoder is equal to the number of possible classes.

[0034] Formally described as follows, where represents the output pair values of the \(m\) - th classification decoder.

[0035]

[0036]

[0037] Once all the categorical features are synthesized by the corresponding decoders, the overall categorical feature of the j-th sample can be obtained by concatenating them.

[0038] Figure 2 It is a schematic structural diagram of the table generator.

[0039] Query process: After the table generator synthesizes the numerical features and the categorical features a query process is performed to induce the response of the victim model . The query process includes the following steps:

[0040] Inverse preprocessing: As described above, the output of the numerical decoder is scaled to the range [0,1] using the sigmoid function. However, these values may not be consistent with the input range expected by the victim model. Therefore, inverse min-max normalization is applied to adjust the range of the synthesized numerical features:

[0041]

[0042] Here, represents the n-th adjusted numerical feature of the j-th sample, is the n-th synthesized numerical feature output by the table generator, and represent the potential maximum and minimum values of this feature respectively. These values can be easily inferred according to the input requirements of the victim model. Applying this normalization to all numerical features is denoted as the adjusted feature vector of the j-th sample

[0043] Query the victim model: Then the adjusted numerical features and the synthesized categorical features are used to query the victim model V. In this application, assuming that the victim model is a classification model that outputs hard labels, the query process can be expressed as:

[0044]

[0045] where y j is the response of the victim model to . The query process is performed on all synthesized features, and the synthesized numerical features x num , categorical features x cat and the returned labels y j are collected to form a dataset This dataset will be used to train the cloned model in the subsequent steps.

[0046] Contrastive Table Transformer (CTT): The potential architectural differences between the victim and the cloned model lead to the "boundary heterogeneity" problem. This problem may prevent the cloned model with a simple architecture from establishing a clear decision boundary through the collected dataset and aligning its predictions with those of the victim model, thus reducing the performance of the cloned model.

[0047] To address this challenge, this application proposes using CTT as the backbone architecture of the cloned model. This architecture utilizes the powerful expressive ability of the attention mechanism to extract potential decision boundary knowledge from the collected dataset to facilitate its learning process. Meanwhile, during the training process, it adopts a customized joint constraint to establish a clear decision boundary and optimize its predictions. As Figure 3 shown, CTT consists of three key steps: data augmentation, feature encoding, and boundary knowledge extraction. Each step will be described in detail below.

[0048] Data augmentation: Collecting labeled queries is a costly process because each additional query increases the risk of being detected. Therefore, to effectively utilize the limited labeled queries, a new data augmentation method has been developed that can synthesize new samples from the collected dataset without additional queries.

[0049] First, for each sample in the dataset , calculate its normalized distance to other samples:

[0050]

[0051] Here, represent the Euclidean distance function and Hamming distance function for numerical features and categorical features respectively. In addition, represent the minimum and maximum distances among all numerical and categorical features. These values are used to normalize the contribution of each feature to the total distance. Based on the calculated distances, a similarity matrix is constructed, where represents the similarity between samples and. This process can be effectively executed through tensor parallel computing.

[0052] Next, based on the similarity matrix, for each sample and its nearest neighbor from the same class, insert a new sample between them through the following process:

[0053]

[0054] where and represent the nth numerical feature and the mth categorical feature of the jth inserted sample respectively. In addition, α∈(0,0.5) is a random interpolation coefficient, β∈(0,1) is a predefined probability threshold, 1(·) represents the indicator function, and u represents a random variable sampled from the uniform distribution U(0,1). The inserted sample Be assigned the same label as s j Process each sample in this way to obtain a new augmented dataset as follows:

[0055]

[0056] Feature encoding: Using the augmented data CTT encodes numerical and categorical features into a high-dimensional space to extract rich feature semantics. Specifically, for the nth numerical feature value of the jth sample in map it to a numerical embedding vector using a numerical tokenizer as follows:

[0057]

[0058] where is the numerical embedding vector and D is the dimension of the embedding. In addition, W num is a learnable parameter and b num is the corresponding bias. Process all numerical features in the same way to obtain a series of numerical embeddings

[0059] For the mth categorical feature of the jth sample in , first convert it to a one-hot vector . Then, apply a categorical tokenizer to convert it to a categorical embedding as follows:

[0060]

[0061] where is the embedding vector of the mth categorical feature of the jth sample. Process all categorical features in the same way to obtain a series of categorical embeddings

[0062] After encoding all samples, the resulting numerical and categorical embedding sequences are then concatenated to form an embedding tensor X ∈ R T ×F×d , which serves as the input for subsequent boundary knowledge learning. Here T is the total number of samples in the augmented dataset, and F = N + M represents the total number of numerical and categorical features in each sample.

[0063] Boundary knowledge extraction: In this step, a multi-head attention mechanism is used to capture feature interactions and extract boundary knowledge from the embedding tensor X ∈ R T×P×d as follows.

[0064] First, use the embedding tensor X ∈ R T×P×dAs inputs for the key (K), query (Q), and value (V) tensors. Then, project K and V linearly into two different spaces to obtain key tensor K ∈ R T×P×d and value tensor V ∈ R T×P×d :

[0065] K = XW K

[0066] V = XW V

[0067] where W K , W V ∈ R d×d are learnable parameter matrices. Then, apply the multi - head attention mechanism and combine it with an average pooling operation over the feature dimension P to obtain a fixed - length sample representation as follows:

[0068] M = Meanpooling(MultiHead(Q, K, V))

[0069] where M ∈ R T×d represents the final sample representation. Finally, the sample representation passes through a fully - connected layer to produce a classification output:

[0070]

[0071] Here represents the probability matrix, C is the number of output classes, and W out and b out are the weights and biases of the output layer respectively.

[0072] Training method: Adopt an iterative alternating training strategy to optimize the table generator and CTT. This strategy ensures that they can benefit from each other and improve the overall model cloning performance through multiple rounds of refinement. Each iteration includes training the CTT and the table generator in turn. A detailed example of a single iteration is provided below to illustrate this process.

[0073] Training CTT: As mentioned before, the boundary heterogeneity problem may limit the effectiveness of model cloning. Therefore, first use the contrastive loss to prompt the CTT to establish a clear decision boundary.

[0074] The contrastive loss is widely used in representation learning and aims to pull semantically similar samples (positive sample pairs) closer in the representation space while pushing dissimilar samples (negative sample pairs) farther away. This method can promote the construction of the decision boundary.

[0075] To this end, first label the positive and negative sample pairs according to the augmented dataset Here, each inserted sample is paired with its neighbor sj Labeled as positive sample pairs . These samples are expected to be on the same side of the decision boundary and have similar representations. Conversely, each s j and its nearest neighbor s from a different class k According to the similarity matrix Are selected as negative sample pairs s - =(s k , s j ). These samples may be close to the decision boundary but on the opposite sides, and their representations should be pushed apart to better establish the decision boundary. Then the contrastive loss Is calculated as follows:

[0076]

[0077] Where M k , M g And M j Represent the sample representations of s from multi-head attention and average pooling respectively k , And s j , sim(·) represents the cosine similarity function, and τ is the temperature hyperparameter.

[0078] In addition, the cross-entropy loss Is used to align the predictions of CTT with those of the victim model as follows:

[0079]

[0080] Where y j Represents the label of the sample s j Predicted by the victim model, Represents the predicted probability of CTT for the input sample s j For class c, and I represents the indicator function. Finally, these two losses are combined into a joint constraint For training CTT:

[0081]

[0082] Where λ1 is the hyperparameter that balances the two loss terms. By jointly optimizing And , CTT can effectively establish a clear decision boundary and make the classification results consistent with the victim model.

[0083] Training Table Generator: The goal of the table generator is to synthesize informative table samples to facilitate the learning of CTT. To this end, the table generator is trained to synthesize samples that can guide CTT to produce high-entropy predictions. The rationale behind this is that the high prediction entropy of the synthesized samples not only indicates that CTT has high uncertainty about them but also indicates that these samples are close to its decision boundary. By encouraging the table generator to synthesize such samples, it can help CTT distinguish them, thereby improving its classification performance and facilitating the construction of the decision boundary. Due to the different responsibilities of the numerical decoder, classification decoder, and base encoder, the following three different loss functions are designed:

[0084] Training Numerical Decoder: For the fully differentiable numerical decoder synthesis process, it is directly trained by gradient descent to minimize the entropy loss . Let be the predicted probability vector of CTT for the synthesized sample x j , and the loss function of the numerical decoder can be defined as follows:

[0085]

[0086] Here, H(·) represents the entropy function, C represents the total number of possible classes of the samples, and J represents the total number of synthesized samples. By minimizing , the numerical decoder can synthesize numerical features that maximize the prediction entropy of CTT.

[0087] Training Classification Decoder: The synthesis process of the classification decoder involves a non-differentiable argmax operation, which hinders the backpropagation of gradients and poses a significant challenge to gradient-based optimization. Therefore, policy gradient is introduced to handle this challenge.

[0088] Policy gradient is a reinforcement learning algorithm that estimates the gradient of the expected reward with respect to the policy parameters and uses this gradient to update the parameters. This method can effectively handle non-differentiable operations in the network, such as the Argmax operation. The optimization process of policy gradient can be expressed as follows:

[0089]

[0090] where π θ (act j |st j ) is the probability of taking action act j in state st j under the policy parameters θ, R j is the reward received after taking action act j , and α is the learning rate.

[0091] Applying this method to the classification decoder is as follows:

[0092] - Treat each classification decoder as a policy.

[0093] - Actions correspond to the decoder output, i.e., the classification feature values

[0094] - The state is represented by the input of each classification decoder, i.e.,

[0095] - The parameters of the m-th classification decoder are denoted as θ m .

[0096] The optimization can be derived as follows:

[0097]

[0098] In addition, the following classification loss can be derived , and all classification decoders are optimized by gradient descent:

[0099]

[0100] Here, denotes the probability that the m-th classification decoder synthesizes given the input .

[0101] Train the base encoder: The base encoder output is the shared base embedding for the numerical decoder and the classification decoder. To ensure that the base embedding can meet the requirements of all decoders simultaneously, a joint constraint is introduced, combining the classification loss and the numerical loss to optimize the base encoder:

[0102]

[0103] where lambda2 is the hyperparameter that balances the two loss terms, and z j is the Gaussian noise input of the base encoder.

[0104] In this application, to comprehensively evaluate the model cloning ability of the method, experiments were conducted on four standard tabular datasets, and the information of the experimental datasets is shown in Table 1:

[0105] Table 1: Experimental Results of Standard Tabular Datasets

[0106] Dataset Number of samples Category Classification feature Numerical feature Adult 48,842 2 8 6 Default 30,000 2 10 13 German 1,000 2 14 6 Shop 3,900 4 12 4

[0107] Table 1 shows the statistical information of the datasets, which is described in detail as follows:

[0108] Adult: This dataset is used to predict whether an individual's annual income exceeds $50,000. It contains 48,842 samples, and each sample includes eight categorical features and six numerical features.

[0109] Default: This dataset is related to bank customer defaults and contains 30,000 samples. Each instance includes ten categorical features and six numerical features, and the dataset is divided into two different classes.

[0110] German: This dataset is used for credit scoring and includes 1,000 samples. Each sample contains 14 categorical features and six numerical features, and the dataset is divided into two classes.

[0111] Shop: This dataset records customers' shopping experiences and contains 3,900 samples. Each sample contains 12 categorical features and four numerical features, and is divided into four classes.

[0112] This embodiment uses multiple victim model architectures to evaluate the proposed method, including:

[0113] RF: An ensemble learning method that constructs multiple decision trees and outputs the majority voting results of each tree. RF is known for its ability to resist overfitting and handle high-dimensional data.

[0114] XGBoost: An ensemble learning method that constructs multiple decision trees through a boosting strategy and outputs. XGBoost shows excellent performance in various tabular data tasks.

[0115] LR: A linear model that estimates the probability of the result based on a set of independent variables. LR is widely used in classification tasks and provides interpretable results.

[0116] MLP: A simple artificial neural network that includes an input layer, one or more hidden layers, and an output layer. MLP can learn complex non-linear relationships between features and labels.

[0117] TabNet: A deep learning architecture designed specifically for tabular data. TabNet uses an attention mechanism to learn interpretable feature representations and shows competitive performance on various tabular datasets.

[0118] FTT: A Transformer-based architecture suitable for tabular data. FTT uses self-attention mechanism to capture feature interactions and demonstrates strong performance on tabular tasks.

[0119] All victim models are trained on the training set until they achieve optimal performance on the corresponding test set, ensuring that they are well optimized.

[0120] This application explores methods for cloning tabular models in a challenging environment without real data and architecture-agnostic. Currently, research on cloning tabular models under these conditions is limited. Only DivT can be executed in such a setting, and this method focuses on cloning ensemble-based tabular models. Therefore, this paper uses DivT as a strong baseline and compares its performance with the method in cloning RF and XGBoost. In addition, StealML has demonstrated the ability to clone LR in a similar setting, so it is also used to compare the cloning of LR. For MLP, few existing studies have successfully cloned it under the settings of this application. In addition, there are few studies on cloning Tabnet and FTT models, making the comparison infeasible. Therefore, for the victim models of these architectures, only the performance of the method is presented.

[0121] The implementations of DivT and StealML are based on the official codebases, with only the number of queries and the parameters of the victim models modified. Specifically, a fixed query budget of 9,600 is set for both methods in all experiments, and the parameters of our locally trained models are used as the victim model parameters. Here, each query represents obtaining the output of the victim model for a query sample. In the method, the SGD optimizer with a momentum of 0.9 and a learning rate of 0.001 is used to train CTT. For the table generator, the SGD optimizer with a learning rate of 0.01 is used. The method is configured to run for 300 iterations with a batch size of 32, keeping the total query budget at 9,600.

[0122] The experimental results are shown in Table 2:

[0123] Table 2: Experimental Results

[0124]

[0125]

[0126] Table 2 provides a comprehensive comparison across different datasets and model architectures, demonstrating the performance of the method (TabExtractor) against StealML, DivT, and the victim models. The results show the effectiveness and versatility of TabExtractor in cloning different tabular model architectures in a no-real-data setting.

[0127] For RF, the performance of TabExtractor is comparable to that of DivT. Although DivT has a slight edge over TabExtractor on the Adult and Default datasets, the latter achieved better results on the German and Shop datasets. Notably, on the Shop dataset, the accuracy of the model cloned by TabExtractor (95.90%) is closer to that of the victim model (96.85%) than DivT (92.12%).

[0128] In the case of the XGBoost model, TabExtractor consistently outperforms DivT on all datasets. The performance gap is particularly significant on the Adult dataset (81.03% vs. 76.45%) and the Shop dataset, where the model cloned by TabExtractor achieved a perfect accuracy (100%) matching that of the victim.

[0129] For LR, TabExtractor exhibits comparable or slightly better performance than StealML. On the Adult and German datasets, the accuracy of the model cloned by TabExtractor is almost the same as that of the victim model, slightly outperforming StealML. For the Shop dataset, although the accuracy of both cloning methods is lower than that of the victim model, TabExtractor (59.23%) performs better than StealML (57.02%), demonstrating the effectiveness of TabExtractor in cloning linear models. Notably, this significant performance drop may be due to the limited performance of the victim model.

[0130] TabExtractor also shows promising results for model architectures without comparable baselines. For MLP, the accuracy of the model cloned by TabExtractor is close to that of the victim model on all datasets, with a performance gap between 2% and 6%. Similarly, for the Tabnet and FTT models, TabExtractor demonstrates strong cloning capabilities, usually achieving an accuracy within 2 - 5% of the victim model.

[0131] Notably, TabExtractor performs particularly well on the FTT model on the Shop dataset. This demonstrates TabExtractor's ability to effectively clone complex model architectures even in challenging situations.

[0132] Overall, these results highlight the effectiveness and versatility of TabExtractor in cloning various table model architectures under the conditions of no real data and hard labels. The performance of TabExtractor has always been comparable to or better than existing benchmark methods (if available), and it has demonstrated strong cloning capabilities on model architectures without previous cloning methods.

[0133] In addition, this application studied the impact of the parameter scale of the victim model on the effect of TabExtractor. RF, XGBoost, MLP, TabNet, and FTT models were trained on three datasets (Adult, Default, and German) with different parameter settings. Then, TabExtractor was applied to perform a cloning attack, and the performance of the cloned CTT on the test set was evaluated.

[0134] By adjusting the hyperparameters of the victim model, the number of learnable parameters was controlled. Table 3 shows the hyperparameter settings of the victim model, where version 1 represents the default hyperparameter configuration, and versions 2 and 3 are extended parameter versions. The details are as follows:

[0135] For RF and XGBoost, the number of base classifiers was increased by adjusting their n_estimators hyperparameters.

[0136] For MLP, we enlarged the parameter scale by increasing the dimension of the hidden layer.

[0137] For TabNet, we adjusted the n_attention hyperparameter to increase the output dimension of the attention layer, thus increasing the number of parameters.

[0138] For FTT, we increased the number of attention blocks by adjusting the n_blocks hyperparameter to expand the parameter scale.

[0139] Table 3: Hyperparameter Settings

[0140] Model Version Parameter RF 1 n_estimators = 100 RF 2 n_estimators = 200 RF 3 n_estimators = 300 XGBoost 1 n_estimators = 100 XGBoost 2 n_estimators = 200 XGBoost 3 n_estimators = 300 MLP 1 hidden_layer_sizes = 100 MLP 2 hidden_layer_sizes = 200 MLP 3 hidden_layer_sizes = 300 TabNet 1 n_attention = 8 TabNet 2 n_attention = 16 TabNet 3 n_attention = 32 FTT 1 n_blocks = 2 FTT 2 n_blocks = 4 FTT 3 n_blocks = 6

[0141] The experimental results are shown in Table 4. Analyzing this table yields the following observations. 1) Although increasing the parameters of the victim model enhanced their resistance to TabExtractor to some extent, TabExtractor still demonstrated excellent performance on all datasets and model types. 2) Increasing only the model parameters without correspondingly increasing the dataset size may not significantly improve the attack resistance. For example, despite the increased parameters, the German dataset showed more result fluctuations and inconsistent protection against attacks due to its small sample size.

[0142] In summary, although increasing the model parameters can, to some extent, defend against cloning and attacks, this strategy alone may not be sufficient to provide adequate protection, especially without a corresponding increase in the dataset size. TabExtractor has demonstrated excellent resilience and effectiveness across different architectures, parameter settings, and datasets, which not only shows its superiority as a cloning method but also indicates that defense strategies need to be more comprehensive rather than simply expanding the parameters.

[0143] Table 4: Experimental Results under Different Parameter Sizes of Victim Models

[0144]

[0145]

[0146] This application proposes a table model cloning method TabExtractor based on generated data. First, a new table generator is proposed to synthesize table samples based on numerical and categorical features. In addition, a new contrastive table transformer (CTT) is proposed as the backbone of the cloning model. The model is trained using the obtained query-response pairs, and data augmentation methods and attention mechanisms are used to establish a clear decision boundary for CTT by combining contrastive loss and cross-entropy loss, making its predictions consistent with those of the victim model. Finally, guided by the output of CTT, the table generator is further optimized by combining numerical loss, classification loss, and joint constraints.

[0147] A table model cloning method based on generated data adopted in this embodiment realizes the advantage of effectively implementing model extraction without the need to know the target model architecture and training data.

[0148] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A table model cloning method based on generated data, characterized in that: The table model cloning method includes a table generator, a comparison table transformer CTT, and a query process; specifically includes the following steps: S1 First, the table generator synthesizes Gaussian noise into numerical and categorical features; and applies customized inverse preprocessing operations to the synthesized features to scale them to the range that the victim model can respond to, and reorders the features to match the input requirements of the victim model; S2: The converted synthetic features are combined into query samples to induce the response of the victim model, which is then used to train the clone model with the contrast table transformer CTT as the backbone; S3: During training, data augmentation methods and attention mechanisms are used in combination with contrastive loss and cross entropy loss to establish a clear decision boundary for CTT and keep its predictions consistent with those of the victim model. S4: And use the output of CTT as guidance and combine numerical loss, classification loss and joint constraints to further optimize the table generator.

2. The table model cloning method based on generated data according to claim 1, characterized in that: The table generator in S1 includes a basic encoder, a numerical decoder, and multiple classification decoders; The steps are as follows: S11: The base encoder consisting of multiple fully connected layers maps the input Gaussian noise vector to a semantically rich latent space and uses skip connections to obtain high-level base embeddings; S12: A numerical decoder composed of multiple fully connected layers receives the output of the basic encoder as input and outputs numerical features; For different categories, different classification decoders are used to synthesize different category features. The output layer of the classification decoder uses the Argmax operation to determine the classification feature value, and the rest of the structure is similar to the numerical decoder.

3. The table model cloning method based on generated data according to claim 1, characterized in that: The backbone of the clone model in S2, namely the comparison table transformer CTT, includes: applying a data augmentation method, constructing a similarity matrix based on the normalized distance of each sample, inserting new samples according to the matrix, and obtaining a new enhanced data set; performing feature encoding, encoding numerical and categorical features into a high-dimensional space to extract rich feature semantics; performing victim model decision boundary knowledge parsing, using a multi-head attention mechanism to capture feature interactions and extract boundary knowledge from query feedback to the victim model.

4. The table model cloning method based on generated data according to claim 1, characterized in that: The S3 trains a contrastive table transformer CTT, including using contrastive loss to encourage CTT to establish clear decision boundaries; and using cross entropy loss to align the predictions of CTT with those of the victim model.

5. The table model cloning method based on generated data according to claim 1, characterized in that: The S4 training table generator comprises: S41: Direct training of a fully differentiable numerical decoder synthesis process via gradient descent; S42: Introduce policy gradient to handle non-differentiable operations in the network and use it to train the synthesis process of the category decoder; S43: Introduce a joint constraint to combine classification loss and numerical loss to optimize the base encoder.

Images