A method for generating tabular data based on local context retrieval and fine-tuning
By employing local contextual retrieval and fine-tuning methods, and utilizing the TabPFN model and SGLD rules to generate tabular data, this approach addresses the challenge of balancing data diversity and realism in existing technologies, thereby improving the accuracy of data generation and the generalization ability of downstream models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PLA DALIAN NAVAL ACADEMY
- Filing Date
- 2026-04-03
- Publication Date
- 2026-06-30
AI Technical Summary
Existing technologies struggle to balance the diversity and authenticity of generated tabular data, especially in scenarios where data is scarce or imbalanced. Existing methods suffer from unstable training and inaccurate distribution matching.
A local context-based retrieval and fine-tuning method is adopted. The local calibration parameters are fine-tuned through the TabPFN model, and pseudo-data samples are generated by iterative sampling of dynamic local context. The SGLD rule is used for updating, and finally a tabular dataset is constructed.
It significantly improves the accuracy and robustness of tabular data generation, and the generated data is more closely aligned with the local distribution characteristics of the original data, enhancing the authenticity and consistency of the data. It is suitable for data augmentation and privacy protection, and improves the generalization ability of downstream machine learning models.
Smart Images
Figure CN122310122A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of machine learning and tabular data generation technology, and in particular to a tabular data generation method based on local context retrieval and fine-tuning. Background Technology
[0002] Tabular data generation is a challenging task in machine learning, especially in scenarios where data is scarce or class imbalanced. The generation of high-quality synthetic data is crucial for improving the performance of downstream tasks. In recent years, methods based on generative adversarial networks (GANs) and variational autoencoders (VAEs) have achieved remarkable results in image and sequence data generation, but they still face problems such as pattern collapse, training instability, and inaccurate distribution matching in tabular data generation. To address this, researchers have proposed a series of table generation methods based on energy models, normalized flow, and diffusion models, such as CTGAN, TVAE, and TabDDPM. However, these methods often struggle to balance the diversity and realism of the generated data. Summary of the Invention
[0003] This invention provides a method for generating tabular data based on local context retrieval and fine-tuning, in order to overcome the problem that existing methods cannot balance the diversity and authenticity of the generated data.
[0004] To achieve the above objectives, the technical solution of the present invention is as follows: A method for generating tabular data based on local context retrieval and fine-tuning includes: S1. Obtain raw tabular data samples for training any machine learning from a public benchmark database, perform preprocessing and data partitioning to obtain feature columns and label columns, and construct a training set based on the feature columns and label columns; S2. Based on the training set, perform local calibration parameter fine-tuning on the TabPFN model to obtain the parameter-fine-tuned TabPFN model; S3. Based on the training set, pseudo-data samples are obtained by manually setting the number of target samples and the target category label; S4. Based on the pseudo-data samples and the TabPFN model with fine-tuned parameters, iterative updates are performed using an iterative sampling method based on dynamic local context to finally obtain the tabular dataset.
[0005] Furthermore, the step of fine-tuning the local calibration parameters includes: S21. Based on the training set, select a batch of query samples from the training set; S22. Using the K-nearest neighbor algorithm, obtain the Euclidean distance between the query sample and samples outside the query sample in the training set, and find the sample with the closest Euclidean distance to the query sample. A real sample, and Each real sample is used as a local context neighborhood set; S23. Input the query sample and the local context neighborhood set into the TabPFN model to obtain the predicted logical value of the query sample; S24. Based on the query sample and the predicted logical value of the query sample, update the parameters of the TabPFN model by minimizing the classification cross-entropy loss to obtain the parameter-fine-tuned TabPFN model.
[0006] Furthermore, the specific steps of the iterative sampling method based on dynamic local context include: S41. Using the pseudo-data samples as the query objects, calculate the Euclidean distance between the samples in the training set and the query objects, and select the sample with the closest Euclidean distance to the query object. 1 neighbor sample, and The feature columns and label columns of each neighbor sample are combined to construct the local context of the pseudo data sample; S42. Based on the pseudo-data samples, the local context of the pseudo-data samples, and the preset target category labels, the TabPFN model with fine-tuned parameters is used to make predictions and obtain the predicted logical value of the target category. S43. Based on the predicted logical value of the target category, the class conditional energy is calculated using the class conditional energy formula to obtain the class conditional energy. S44. Based on the class conditional energy, the pseudo-data samples are updated using the SGLD rule to finally obtain the tabular dataset.
[0007] Furthermore, the formula for the class conditional energy is:
[0008] In the formula, Conditional energy for class; The logical value for the target category; This is a sample of pseudo-data; The preset target label.
[0009] Furthermore, the expression for the SGLD rule is:
[0010] In the formula, This is the updated data sample; Step size; The gradient of the energy function; Noise figure; The expected value is 0, and the covariance matrix is... The standard multivariate Gaussian distribution.
[0011] Beneficial effects: The present invention provides a table data generation method based on local context retrieval and fine-tuning. By fine-tuning the local calibration parameters of the TabPFN basic model, it can effectively explore the potential correlation between feature columns and label columns when the number of training samples is limited, and significantly improve the accuracy and robustness of the model in small sample table data generation tasks. By introducing an iterative sampling mechanism based on dynamic local context, the pseudo data sample most relevant to the current context can be dynamically retrieved during the generation process, making the generated table data more consistent with the local distribution characteristics of the original data, thus enhancing the authenticity and consistency of the data. By artificially setting pseudo-data samples as initial seeds and combining them with an iterative sampling strategy, this invention can gradually expand the coverage of generated data, avoid generating samples that are singular or repetitive, and improve the diversity of generated data. It is suitable for scenarios such as data augmentation and privacy-preserving generation. The generated tabular data samples can be used to expand the training set and improve the generalization ability of downstream machine learning models, especially in cases of scarce data or class imbalance, where they have a significant data augmentation effect. They still have strong versatility without relying on specific table structures or domain knowledge and can be widely applied to structured data generation tasks in finance, healthcare, industry, and other fields. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a flowchart illustrating the table data generation method of the present invention; Figure 2 The graphs show the probability density distribution curves of real data and data generated by the present invention in the embodiments of the present invention. Figure 3 The Pearson correlation coefficients between features in the real dataset in this embodiment of the invention; Figure 4 This invention generates Pearson correlation coefficients between features in the dataset for use in embodiments of the invention. Detailed Implementation
[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0015] This embodiment provides a method for generating table data based on local context retrieval and fine-tuning, such as... Figure 1 As shown, it includes: S1. Obtain raw tabular data samples for training any machine learning from a public benchmark database, perform preprocessing and data partitioning to obtain feature columns and label columns, and construct a training set based on the feature columns and label columns; Specifically, the preprocessing involves data cleaning and removing missing or abnormal records. The preprocessed raw table data samples are then manually divided into feature columns based on the machine learning application scenario and training task. and label column The feature columns are Z-score standardized to obtain standardized feature columns. The data cleaning, removal of missing or abnormal records, data sample division and Z-score annotation are existing technologies and will not be described in detail here.
[0016] S2. Based on the training set, perform local calibration parameter fine-tuning on the TabPFN model to obtain the parameter-fine-tuned TabPFN model; Preferably, the step of fine-tuning the local calibration parameters includes: S21. Based on the training set, select a batch of query samples from the training set; S22. Using the K-nearest neighbor algorithm, obtain the Euclidean distance between the query sample and samples outside the query sample in the training set, and find the sample with the closest Euclidean distance to the query sample. A real sample, and Each real sample is used as a local context neighborhood set; S23. Input the query sample and the local context neighborhood set into the TabPFN model to obtain the predicted logical value of the query sample; S24. Based on the query sample and the predicted logical value of the query sample, update the parameters of the TabPFN model by minimizing the classification cross-entropy loss to obtain the parameter-fine-tuned TabPFN model.
[0017] Specifically, based on the training set, select a batch of query samples from the training set. ; For query samples The K-nearest neighbor algorithm is used to calculate the Euclidean distance between the query sample and samples outside the query sample in the training set, thus obtaining the closest Euclidean distance. Using real samples, a local context neighborhood set is constructed, expressed as:
[0018] In the formula, For local context neighborhood set; For query samples; For the retrieved first Features of each context sample; For the retrieved first The labels of each context sample; Input the query sample and the local context neighborhood set into the parameter. In the TabPFN model, the predicted logistic value of the query sample is obtained; by minimizing the classification cross-entropy loss, the parameters are adjusted. After updating and optimizing, the TabPFN model with fine-tuned parameters is obtained. The expression for minimizing the classification cross-entropy loss is:
[0019] In the formula, To minimize the classification cross-entropy loss; These are the parameters of the TabPFN model; Batch size; Total number of categories; For the first The true labels of each query sample; For the first The true labels of each query sample; For category traversal variables; The index for the queried sample; The TabPFN model; For the first The local context neighborhood set of each query sample.
[0020] S3. Based on the training set, pseudo-data samples are obtained by manually setting the number of target samples and the target category label; Specifically, the steps to obtain pseudo-data samples are as follows: S31. Based on the training set, obtain the target category labels in the training set as follows: Real samples; S32. Randomly sample the real sample to obtain the sampled sample; S33. Add a small Gaussian noise to the sampled data to obtain pseudo data samples. The Gaussian noise mentioned is existing technology and will not be described in detail here.
[0021] S4. Based on the pseudo-data samples and the TabPFN model with fine-tuned parameters, iterative updates are performed using an iterative sampling method based on dynamic local context to finally obtain the tabular dataset.
[0022] Preferably, the specific steps of the iterative sampling method based on dynamic local context include: S41. Using the pseudo-data samples as the query objects, calculate the Euclidean distance between the samples in the training set and the query objects, and select the sample with the closest Euclidean distance to the query object. 1 neighbor sample, and The feature columns and label columns of each neighbor sample are combined to construct the local context of the pseudo data sample; S42. Based on the pseudo-data samples, the local context of the pseudo-data samples, and the preset target category labels, the TabPFN model with fine-tuned parameters is used to make predictions and obtain the predicted logical value of the target category. S43. Based on the predicted logical value of the target category, the class conditional energy is calculated using the class conditional energy formula to obtain the class conditional energy. S44. Based on the class conditional energy, the pseudo-data samples are updated using the SGLD rule to finally obtain the tabular dataset.
[0023] Preferably, the conditional energy formula is:
[0024] In the formula, Conditional energy for class; The logical value for the target category; This is a sample of pseudo-data; The preset target label.
[0025] Specifically, the steps for deriving the conditional energy formula are as follows: S431. Define the class-unknown energy function as:
[0026] In the formula, For the sample Class-agnostic energy; The feature vector of the input tabular data; For any one of the set of all possible category labels; The TabPFN model, tuned for parameters, uses the feature vectors of the input tabular data. Predicted as category The output logical value; For any category in the set of all possible category labels The output logical value is first obtained by taking the exponent, summing, and then performing the natural logarithm operation; S432. Based on the class-agnostic energy function, make the feature vector of the input table data... The joint probability follows the Boltzmann distribution:
[0027] In the formula, Feature vectors of the input tabular data The joint probability density; Meanwhile, the feature vector of the input tabular data With real category labels It also satisfies the following Bayesian formula:
[0028] In the formula, Class conditional probability density, i.e., the true class label Under the given conditions, the feature vector of the input tabular data The probability density of occurrence; This is the posterior probability; S433. Based on the class-agnostic energy function, the posterior probability is rewritten using logarithmic operations, and the expression is:
[0029] In the formula, To provide a given input feature sample Under the conditions, Belongs to the real category tag The posterior probability; S434. Based on the class-agnostic energy function and the rewritten posterior probability, the class-conditional energy function is obtained through Bayes' theorem and the Boltzmann distribution, and its expression is:
[0030] In the formula, For a given category label Under the condition of input feature samples Class-conditional energy function; Substituting the pseudo-data samples and the preset target labels into it, we get .
[0031] Preferably, the expression for the SGLD rule is:
[0032] In the formula, This is the updated data sample; Step size; The gradient of the energy function; Noise figure; The expected value is 0, and the covariance matrix is... The standard multivariate Gaussian distribution.
[0033] In a specific embodiment, the Pima Indians Diabetes dataset was selected as the experimental subject. This dataset is a common diabetes prediction dataset used for machine learning related tasks. It contains 768 records, each of which contains 8 feature attributes: number of pregnancies, glucose concentration, blood pressure, subcutaneous fat thickness, insulin level, BMI index, diabetes genetic function, age, and one target label, namely whether the person has diabetes. In this embodiment, 80% of the data is selected as the training set. The remaining 20% will be used as the test set. For subsequent evaluation; The training set is cleaned to ensure that there are no missing values; continuous numerical features are standardized using Z-Score to make their mean 0 and variance 1, so as to meet the input distribution requirements of the pre-trained TabPFN model. Load the pre-trained TabPFN base model and perform local calibration fine-tuning on that model; Regarding specific parameters, the number of neighbors to be retrieved is set to l=10. For each sample in the training set, its l nearest neighbors in the feature space are retrieved as context and the input sequence is constructed. The encoder of TabPFN is fine-tuned by minimizing the classification cross-entropy loss. The number of iterations (Epoch) is set to 5 and the learning rate is 0.0001. This process allows the model to fully learn the local manifold structure specific to the current dataset. Initialize iterative sampling based on dynamic local context; set the number of generated samples according to requirements. From the training set Randomly select samples from each category and add Gaussian noise. As initial pseudo-samples ; Set the number of iterations. Iterative sampling based on dynamic local context is performed to obtain 30 updated samples; the 30 updated samples are then inversely standardized to obtain the final synthetic tabular dataset.
[0034] In this embodiment, in order to verify the effectiveness of the data generated by the present invention, it is evaluated from three aspects: univariate distribution, multivariate correlation, and performance of downstream machine learning tasks. In this example, the univariate distribution selects BMI (Body Mass Index) as a feature, and plots probability density distribution curves for real data and data generated in this embodiment; for example... Figure 2 As shown, the distribution curve of the generated data highly coincides with the real data, and the peaks and troughs are basically the same, indicating that the table data generation method in this embodiment can capture the edge distribution of a single feature very well. The multivariate correlations were assessed by plotting a heatmap using the Pearson correlation coefficient to evaluate the dependencies between features; such as Figure 3 and Figure 4 As shown, where Figure 3 It generates data. Figure 4 The data is real data; as can be seen from the comparison, the generated data accurately reproduces the key feature relationships in the original data, such as the positive correlation between "Age" and "Number of Pregnancy" and the strong correlation between "Skin Thickness" and "Insulin". This proves that the local context retrieval mechanism of the table data generation method in this embodiment effectively captures the complex high-dimensional joint distribution structure in the table data. The downstream machine learning task performance is evaluated using XGBoost as the downstream classifier, assessing the utility of the generated data. Two sets of experiments are set up: training on the original data and training on the generated data. The original data training involves training XGBoost using the original training set. The generated data training involves training XGBoost using only the synthetic dataset (excluding the original dataset) generated by the tabular data generation method of this embodiment. Both models were evaluated on the real test set using AUC as the metric, and five independent replicate experiments were conducted to eliminate the influence of randomness. The experimental results are shown in Table 1: Table 1
[0035] The results are shown in Table 1. The model trained using only the data generated by the table data generation method of this embodiment has an average AUC of 0.8334, which is significantly better than the average AUC of 0.7750 of the benchmark model trained using the original data. This result fully demonstrates that the table data generated by the table data generation method of this embodiment not only retains the discriminative information of the original data, but also introduces benign perturbations and smoothing through energy-based SGLD sampling, which achieves excellent data augmentation and regularization effects and can effectively improve the generalization ability of downstream tasks.
[0036] The present invention has the following beneficial effects: This invention provides a table data generation method based on local context retrieval and fine-tuning. By fine-tuning the local calibration parameters of the TabPFN basic model, it can effectively mine the potential correlation between feature columns and label columns when the number of training samples is limited, and significantly improve the accuracy and robustness of the model in small sample table data generation tasks. By introducing an iterative sampling mechanism based on dynamic local context, the pseudo data sample most relevant to the current context can be dynamically retrieved during the generation process, making the generated table data more consistent with the local distribution characteristics of the original data, thus enhancing the authenticity and consistency of the data. By artificially setting pseudo-data samples as initial seeds and combining them with an iterative sampling strategy, this invention can gradually expand the coverage of generated data, avoid generating samples that are singular or repetitive, and improve the diversity of generated data. It is suitable for scenarios such as data augmentation and privacy-preserving generation. The generated tabular data samples can be used to expand the training set and improve the generalization ability of downstream machine learning models, especially in cases of scarce data or class imbalance, where they have a significant data augmentation effect. They still have strong versatility without relying on specific table structures or domain knowledge and can be widely applied to structured data generation tasks in finance, healthcare, industry, and other fields.
[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A table data generation method based on local context retrieval and fine-tuning, characterized in that, include: S1. Obtain raw tabular data samples for training any machine learning from a public benchmark database, perform preprocessing and data partitioning to obtain feature columns and label columns, and construct a training set based on the feature columns and label columns; S2. Based on the training set, perform local calibration parameter fine-tuning on the TabPFN model to obtain the parameter-fine-tuned TabPFN model; S3. Based on the training set, pseudo-data samples are obtained by manually setting the number of target samples and the target category label; S4. Based on the pseudo-data samples and the TabPFN model with fine-tuned parameters, iterative updates are performed using an iterative sampling method based on dynamic local context to finally obtain the tabular dataset.
2. The table data generation method based on local context retrieval and fine-tuning according to claim 1, characterized in that, The steps for fine-tuning the local calibration parameters include: S21. Based on the training set, select a batch of query samples from the training set; S22, obtain the Euclidean distance between the query sample and the samples in the training set except the query sample by K-neighbor algorithm, obtain the nearest real sample to the query sample, and take the real sample as the local context neighborhood set; S23. Input the query sample and the local context neighborhood set into the TabPFN model to obtain the predicted logical value of the query sample; S24. Based on the query sample and the predicted logical value of the query sample, update the parameters of the TabPFN model by minimizing the classification cross-entropy loss to obtain the parameter-fine-tuned TabPFN model.
3. The method for generating table data based on local context retrieval and fine-tuning according to claim 1, characterized in that, The specific steps of the iterative sampling method based on dynamic local context include: S41. Using the pseudo-data samples as the query objects, calculate the Euclidean distance between the samples in the training set and the query objects, and select the sample with the closest Euclidean distance to the query object. 1 neighbor sample, and The feature columns and label columns of each neighbor sample are combined to construct the local context of the pseudo data sample; S42. Based on the pseudo-data samples, the local context of the pseudo-data samples, and the preset target category labels, the TabPFN model with fine-tuned parameters is used to make predictions and obtain the predicted logical value of the target category. S43. Based on the predicted logical value of the target category, the class conditional energy is calculated using the class conditional energy formula to obtain the class conditional energy. S44. Based on the class conditional energy, the pseudo-data samples are updated using the SGLD rule to finally obtain the tabular dataset.
4. The method for generating table data based on local context retrieval and fine-tuning according to claim 3, characterized in that, The formula for the conditional energy class is: In the formula, Conditional energy for class; The logical value for the target category; This is a sample of pseudo-data; The preset target label.
5. The method for generating table data based on local context retrieval and fine-tuning according to claim 4, characterized in that, The expression for the SGLD rule is: In the formula, This is the updated data sample; Step size; The gradient of the energy function; Noise figure; The expected value is 0, and the covariance matrix is... The standard multivariate Gaussian distribution.