A table data synthesis method, system, device and storage medium

Abstract

The application discloses a table data synthesis method, system, device and storage medium. The table data synthesis method comprises the following steps: removing duplicate semantic dimension label values of all semantic dimensions of a first table training sample set; combining each semantic dimension of each first table training sample and each semantic dimension label value of each semantic dimension; removing duplicate first combined label values in a first combined label value set; determining the number of times that each second combined label value in a second combined label value set appears in the first combined label value set; screening the first table training sample set based on the total number of times of each second combined label value, a first probability value and a preset number of training samples; training a first table data synthesis model based on a second table training sample set to obtain a trained table data synthesis model, thereby improving the efficiency and accuracy of table data synthesis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of table data synthesis, and in particular to a table data synthesis method, system, device and storage medium. BACKGROUND

[0002] In recent years, the demand for high-quality synthetic data has surged due to the need for privacy-preserving data analysis, compliance data sharing, and personalized machine learning applications. Unlike images or text, where elements (e.g., pixels or tokens) are typically homogeneous and spatially related, table data is composed of different fields, such as categorical and numerical, each with different distribution properties and semantic meanings. This structural diversity presents unique challenges for generating synthetic table data.

[0003] In the prior art, table data is mainly synthesized through artificially defined rule templates, but different data sources require the development of adaptive modules, resulting in low processing efficiency. SUMMARY

[0004] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application proposes a table data synthesis method, system, device and storage medium, which can process different types of data through a trained table data synthesis model, improving the efficiency and accuracy of table data synthesis.

[0005] In a first aspect of the present application, a table data synthesis method is provided, comprising the following steps:

[0006] A first table training sample set is constructed, wherein the first table training sample set includes a plurality of semantic dimensions of each first table training sample and a plurality of semantic dimension label values of each semantic dimension;

[0007] All semantic dimension label values of the first table training sample set are de-duplicated to obtain a first semantic dimension label value set;

[0008] Each semantic dimension of each first table training sample and each semantic dimension label value of each semantic dimension are combined to obtain a first combined label value set;

[0009] The first combined label value in the first combined label value set is de-duplicated to obtain a second combined label value set;

[0010] The number of occurrences of each second combined label value in the first combined label value set in the second combined label value set is determined as the total number of each second combined label value;

[0011] determine a first probability value of each of the first semantic dimension label values based on a total number of each of the second combination label values;

[0012] filter the first table training sample set based on the first probability value and a preset training sample number to obtain a second table training sample set;

[0013] construct a first table data synthesis model, and train the first table data synthesis model based on the second table training sample set to obtain a trained table data synthesis model.

[0014] According to the table data synthesis method, the following beneficial effects are achieved:

[0015] The method first constructs a first table training sample set, and removes all semantic dimension label values of the first table training sample set to obtain a first semantic dimension label value set. The core category of the semantic dimension can be extracted, and the interference of redundant information is eliminated to provide an accurate and efficient label basis for subsequent model training. Then, a first probability value of each first semantic dimension label value is determined. The first table training sample set is filtered based on the first probability value and a preset training sample number to obtain a second table training sample set. The first table training sample set is filtered based on the first probability value, which can reduce the interference of extreme samples on subsequent model training and control the sample size to balance the training efficiency and accuracy of the model. Finally, a first table data synthesis model is constructed, and the first table data synthesis model is trained based on the second table training sample set to obtain a trained table data synthesis model. Different types of data are processed by using the trained table data synthesis model, and the efficiency and accuracy of table data synthesis are improved.

[0016] According to some embodiments of the present application, before the determination of the first probability value of each of the first semantic dimension label values based on the total number of each of the second combination label values, the following steps are included:

[0017] construct a first semantic constraint table based on the first semantic dimension label value set, wherein the first semantic constraint table is a hash structure;

[0018] The determination of the first probability value of each of the first semantic dimension label values based on the total number of each of the second combination label values includes:

[0019] determine a second probability value of each of the second combination label values based on the total number of all the second combination label values;

[0020] determine the first probability value of each of the first semantic dimension label values based on the second probability value and the first semantic constraint table.

[0021] According to some embodiments of the present application, the first table training sample set is filtered based on the first probability value and a preset number of training samples to obtain a second table training sample set, including:

[0022] The first semantic constraint table is updated based on the first probability value to obtain a second semantic constraint table, wherein the key of the second semantic constraint table is a first semantic dimension label value in the first semantic dimension label value set, and the value of the second semantic constraint table is the first probability value of each first semantic dimension label value.

[0023] The second semantic constraint table is randomly sampled based on the preset number of training samples by a weighted random sampling method to obtain a second semantic dimension label value.

[0024] A first table training sample is randomly extracted from the first table training sample set based on the second semantic dimension label value as a second table training sample set, wherein the second semantic dimension label value corresponds to a second table training sample in the second table training sample set.

[0025] According to some embodiments of the present application, the first table data synthesis model includes a first generator, a first discriminator and a first classifier, and the first table data synthesis model is trained based on the second table training sample set to obtain a trained table data synthesis model, including:

[0026] The second semantic dimension label value and a first random noise vector are input into the first generator to obtain a first table synthesis data set output by the first generator, wherein a first table synthesis data in the first table synthesis data set corresponds to the second semantic dimension label value, and the first random noise vector is a random noise vector randomly generated and having a total number same as that of the second semantic dimension label value.

[0027] The second table training sample set is input into the first discriminator to obtain a first score value output by the first discriminator, and the first table synthesis data set is input into the first discriminator to obtain a second score value output by the first discriminator.

[0028] A first loss value is determined by a Wasserstein distance loss function with gradient penalty based on the first score value and the second score value.

[0029] The second table training sample set is input into the first classifier to obtain a first semantic dimension probability value output by the first classifier, and the first table synthesis data set is input into the first classifier to obtain a second semantic dimension probability value output by the first classifier.

[0030] determining a second loss value based on the first semantic dimension probability value and the second semantic dimension probability value by a weighted multi-class cross-entropy loss function and an entropy regularization term method;

[0031] determining a third loss value based on the second score value, the first semantic dimension probability value and the second semantic dimension probability value;

[0032] updating the first table data synthesis model based on the first loss value, the second loss value and the third loss value to obtain the trained table data synthesis model.

[0033] According to some embodiments of the present application, the updating of the first table data synthesis model based on the first loss value, the second loss value and the third loss value to obtain the trained table data synthesis model comprises:

[0034] in the case that the first loss value is less than a first preset loss threshold, the second loss value is less than a second preset loss threshold and the third loss value is less than a third preset loss threshold, taking the first table data synthesis model as the trained table data synthesis model;

[0035] in the case that the first loss value is greater than or equal to a first preset loss threshold, the second loss value is greater than or equal to a second preset loss threshold or the third loss value is greater than or equal to a third preset loss threshold, iteratively updating the first discriminator based on the first loss value, iteratively updating the first classifier based on the second loss value and iteratively updating the first generator based on the third loss value until a preset iteration number is reached to obtain the trained table data synthesis model.

[0036] According to some embodiments of the present application, the determination of the third loss value based on the second score value, the first semantic dimension probability value and the second semantic dimension probability value comprises:

[0037] determining a cross-entropy loss value based on the first semantic dimension probability value and the second semantic dimension probability value by a weighted multi-class cross-entropy loss function;

[0038] calculating an expected value of the second score value;

[0039] determining the third loss value based on the cross-entropy loss value and the expected value.

[0040] According to some embodiments of the present application, the trained table data synthesis model comprises a trained generator, and the method further comprises:

[0041] In the case of obtaining the table data synthesis requirement of the user, based on the table data synthesis requirement, corresponding semantic dimension label values are screened from the second semantic dimension label values as third semantic dimension label values;

[0042] The third semantic dimension label values and second random noise vectors are input into the trained generator to obtain a second table synthesis data set output by the trained generator, wherein the second random noise vectors are random noise vectors randomly generated and having the same total number as the total number of the third semantic dimension label values;

[0043] The second table synthesis data set is sent to the user.

[0044] In a second aspect of the present application, a table data synthesis system is provided, which comprises:

[0045] A data construction module is configured to construct a first table training sample set, wherein the first table training sample set comprises a plurality of semantic dimensions of each first table training sample and a plurality of semantic dimension label values of each semantic dimension;

[0046] A first deduplication module is configured to deduplicate all the semantic dimension label values of the first table training sample set to obtain a first semantic dimension label value set;

[0047] A combination module is configured to combine each semantic dimension of each first table training sample and each semantic dimension label value of each semantic dimension to obtain a first combined label value set;

[0048] A second deduplication module is configured to deduplicate first combined label values in the first combined label value set to obtain a second combined label value set;

[0049] A total number determination module is configured to determine the number of occurrences of each second combined label value in the first combined label value set as the total number of each second combined label value in the second combined label value set;

[0050] A first probability value determination module is configured to determine a first probability value of each first semantic dimension label value based on the total number of each second combined label value;

[0051] A data screening module is configured to screen the first table training sample set based on the first probability value and a preset training sample number to obtain a second table training sample set;

[0052] The model training module is configured to construct a first table data synthesis model, train the first table data synthesis model based on the second table training sample set, and obtain a trained table data synthesis model.

[0053] The system first constructs a first table training sample set, removes duplicate semantic dimension label values of the first table training sample set, and obtains a first semantic dimension label value set. The core category of the semantic dimension can be extracted, and the interference of redundant information is eliminated to provide an accurate and efficient label basis for subsequent model training. Then, a first probability value of each first semantic dimension label value is determined. The first table training sample set is filtered based on the first probability value and a preset training sample number to obtain a second table training sample set. The first table training sample set is filtered based on the first probability value, which can reduce the interference of extreme samples on subsequent model training and control the sample size to balance the training efficiency and accuracy of the model. Finally, the first table data synthesis model is constructed, the first table data synthesis model is trained based on the second table training sample set, and the trained table data synthesis model is obtained. Different types of data are processed by using one trained table data synthesis model, and the efficiency and accuracy of table data synthesis are improved.

[0054] In a third aspect, the present application provides a table data synthesis electronic device, which comprises at least one control processor and a memory connected in communication with the at least one control processor. The memory stores instructions executable by the at least one control processor. The instructions are executed by the at least one control processor to enable the at least one control processor to perform the table data synthesis method described above.

[0055] In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the table data synthesis method described above.

[0056] It should be noted that the beneficial effects of the second to fourth aspects of the present application and the prior art are the same as those of the above-mentioned table data synthesis system and the prior art, which will not be described here.

[0057] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0058] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, including the appended drawings, wherein:

[0059] Figure 1is a flowchart of an embodiment of the table data synthesis method provided by the present application;

[0060] Figure 2 is a structural diagram of an embodiment of the table data synthesis system provided by the present application;

[0061] Figure 3 is a structural diagram of an embodiment of the electronic device provided by the present application. DETAILED DESCRIPTION

[0062] The embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0063] In the description of the present application, if the first, second, etc. are described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implying the number of the indicated technical features or the order of the indicated technical features.

[0064] In the description of the present application, it is to be understood that the orientation description, such as the orientation or position relationship indicated by up, down, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as indicating or implying that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0065] In the description of the present application, it is to be understood that, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and the person skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0066] In recent years, the demand for high-quality synthetic data has surged due to the demand for privacy-preserving data analysis, compliance data sharing, and personalized machine learning applications. Unlike images or text, where elements (e.g., pixels or tokens) are typically homogeneous and spatially related, table data is composed of different fields, such as categorical and numerical, each with different distribution properties and semantic meanings. This structural diversity presents unique challenges for the generation of synthetic table data.

[0067] In the prior art, table data is mainly synthesized through artificially defined rule templates, but different data sources require the development of adaptive modules, resulting in low processing efficiency.

[0068] To address the aforementioned technical deficiencies, embodiments of this application provide a method, system, device, and storage medium for synthesizing tabular data.

[0069] Please see Figure 1 This is a flowchart illustrating a method for synthesizing tabular data according to an embodiment of this application. This method is applied to an electronic device, which may be a server, etc. Figure 1 As shown, the method for synthesizing the table data includes:

[0070] Step S101: Construct a first table training sample set, wherein the first table training sample set includes several semantic dimensions of each first table training sample and several semantic dimension label values ​​of each semantic dimension;

[0071] The aforementioned semantic dimension label values ​​can be specific numerical values ​​or symbols used to identify or classify an object or entity under a specific semantic dimension.

[0072] Before step S101 described above, steps S1011 to S1017 may be included:

[0073] Step S1011: Collect the first historical table data;

[0074] Step S1012: Clean the first historical table data to remove data containing null values ​​and other erroneous records, and obtain the second historical table data; the first historical table data can be cleaned manually.

[0075] Step S1013: Manually divide the data in the second historical table into categorical fields and numeric fields;

[0076] Step S1014: Convert the numeric field using the following formula to obtain the preprocessed numeric field:

[0077] ;

[0078] in, For the second historical table data, the first Line number The preprocessed numeric field of the column, This is a minimum boundary value pre-set according to actual needs. These are constant values ​​pre-set according to actual needs. For the second historical table data, the first Line number Numeric fields in a column.

[0079] Step S1015: Encode the preprocessed numerical field using a variational Gaussian mixture model (VGM) to obtain the encoded numerical field;

[0080] Step S1016, encode the discrete type field by a one-hot encoding method to obtain an encoded discrete type field;

[0081] Step S1017, splice the encoded numerical type field and the encoded discrete type field of each historical table data to obtain a spliced field;

[0082] The step S101 can include a step S1019:

[0083] Step S1019, construct a first table training sample set based on the spliced field, wherein the first table training sample set includes the spliced field.

[0084] Step S102, de-duplicate all semantic dimension label values of the first table training sample set to obtain a first semantic dimension label value set;

[0085] Step S103, combine each semantic dimension of each first table training sample and each semantic dimension label value of each semantic dimension to obtain a first combined label value set;

[0086] The first combined label value in the first combined label value set can be “gender (semantic dimension) female (semantic dimension label value)”, can also be “gender (semantic dimension) male (semantic dimension label value)”, and can also be “age (semantic dimension) youth (semantic dimension label value)”.

[0087] Step S104, de-duplicate the first combined label value in the first combined label value set to obtain a second combined label value set;

[0088] Step S105, determine the number of times each second combined label value in the second combined label value set appears in the first combined label value set as the total number of times of each second combined label value;

[0089] In step S105, the determination of the number of times each second combined label value in the second combined label value set appears in the first combined label value set as the total number of times of each second combined label value can be that the number of times each second combined label value in the second combined label value set appears in the first combined label value set is counted by a counting function as the total number of times of each second combined label value.

[0090] Step S106, determine a first probability value of each first semantic dimension label value based on the total number of times of each second combined label value;

[0091] Step S107, filter the first table training sample set based on the first probability value and a preset training sample number to obtain a second table training sample set;

[0092] The preset number of training samples can be a value set in advance according to actual needs.

[0093] In step S108, a first table data synthesis model is constructed, the first table data synthesis model is trained based on the second table training sample set, and a trained table data synthesis model is obtained.

[0094] The first table data synthesis model can include a first generator, a first discriminator, and a first classifier.

[0095] The trained table data synthesis model can include a trained generator, a trained discriminator, and a trained classifier.

[0096] The method can extract the core categories of semantic dimensions, eliminate redundant information interference, provide accurate and efficient label basis for subsequent model training, and then determine the first probability value of each first semantic dimension label value. Based on the first probability value and the preset number of training samples, the first table training sample set is filtered to obtain the second table training sample set. Thus, the first table training sample set is filtered based on the first probability value, which can reduce the interference of extreme samples on subsequent model training, control the sample size, balance the training efficiency and accuracy of the model, construct the first table data synthesis model, train the first table data synthesis model based on the second table training sample set, obtain the trained table data synthesis model, and process different types of data through the trained table data synthesis model, thereby improving the efficiency and accuracy of table data synthesis.

[0097] In some embodiments, before step S106, step S201 can be included.

[0098] In step S201, a first semantic constraint table is constructed based on the first semantic dimension label value set, and the first semantic constraint table is a hash structure.

[0099] In step S201, the first semantic constraint table is constructed based on the first semantic dimension label value set, which can be the first semantic constraint table as the key.

[0100] Step S106 can include steps S202 to S203.

[0101] In step S202, a second probability value of each second combined label value is determined based on the total number of all second combined label values.

[0102] In step S202, the second probability value of each second combined label value can be determined based on the total number of all second combined label values by the following formula:

[0103] ;

[0104] wherein, is the first probability value of the second combined label value with semantic dimension being and semantic dimension label value being , is the total number of second combined label values with semantic dimension being and semantic dimension label value being , is the total number of second combined label values with semantic dimension being and semantic dimension label value being , is the total number of second combined label values with semantic dimension being .

[0105] In step S203, the first probability value of each first semantic dimension label value can be determined based on the second probability value and the first semantic constraint table by the following formula:

[0106]

[0107] ;

[0108] ;

[0109] wherein, is the constraint indicator function value of the second combined label value with semantic dimension label value being , is the first semantic constraint table, is the first probability value of the first semantic dimension label value with semantic dimension label value being , is the total number of second combined label values with semantic dimension label value being .

[0110] The present application quantifies the occurrence rule of label combination, and combines the rule constraint in the semantic level, so that the first probability value of the first semantic dimension label value is more accurate, and more accurate data basis is provided for the subsequent steps.

[0111] ​In some embodiments, step S107 can include steps S301-S303:

[0112] Step S301: updating the first semantic constraint table based on the first probability value to obtain a second semantic constraint table, wherein a key of the second semantic constraint table is a first semantic dimension label value in the first semantic dimension label value set, and a value of the second semantic constraint table is the first probability value of each first semantic dimension label value.

[0113] In step S301, updating the first semantic constraint table based on the first probability value to obtain a second semantic constraint table can be updating the value of the first semantic constraint table with the first probability value, and taking the updated first semantic constraint table as the second semantic constraint table.

[0114] Step S302: randomly sampling the second semantic constraint table based on a preset number of training samples by a weighted random sampling method to obtain a second semantic dimension label value.

[0115] In step S302, randomly sampling the second semantic constraint table based on the preset number of training samples by the weighted random sampling method to obtain the second semantic dimension label value can be randomly sampling the second semantic dimension label values from the second semantic constraint table by the weighted random sampling method, with a total number of the second semantic dimension label values being the preset number of training samples.

[0116] Step S303: randomly selecting a first table training sample from the first table training sample set based on the second semantic dimension label value to obtain a second table training sample set, wherein the second semantic dimension label value and the second table training sample in the second table training sample set correspond to each other.

[0117] In step S303, randomly selecting a first table training sample from the first table training sample set based on the second semantic dimension label value to obtain a second table training sample set can be randomly selecting a first table training sample whose semantic dimension label value belongs to the second semantic dimension label value from the first table training sample set to obtain the second table training sample set.

[0118] The present application can reduce the interference of extreme samples on subsequent model training by screening the first table training sample set based on the first probability value, and can control the sample size to balance the training efficiency and accuracy of the model.

[0119] In some embodiments, step S108 can include steps S401-S407:

[0120] In step S401, the second semantic dimension label value and the first random noise vector are input into the first generator to obtain a first table synthetic dataset output by the first generator, wherein the first table synthetic data in the first table synthetic dataset correspond one-to-one to the second semantic dimension label value, and the first random noise vector is a random noise vector randomly generated and having a total number same as that of the second semantic dimension label value.

[0121] In step S402, the second table training sample set is input into the first discriminator to obtain a first score value output by the first discriminator, and the first table synthetic dataset is input into the first discriminator to obtain a second score value output by the first discriminator.

[0122] In step S403, the first loss value is determined based on the first score value and the second score value by using a Wasserstein distance loss function with gradient penalty.

[0123] In step S403, the first loss value can be determined by calculating the first score value and the second score value by using the Wasserstein distance loss function with gradient penalty.

[0124] In step S404, the second table training sample set is input into the first classifier to obtain a first semantic dimension probability value output by the first classifier, and the first table synthetic dataset is input into the first classifier to obtain a second semantic dimension probability value output by the first classifier.

[0125] In step S405, the second loss value is determined based on the first semantic dimension probability value and the second semantic dimension probability value by using a weighted multi-class cross-entropy loss function and an entropy regularization term method.

[0126] In step S405, the second loss value can be determined based on the first semantic dimension probability value and the second semantic dimension probability value by using the weighted multi-class cross-entropy loss function and the entropy regularization term method, and the calculation formula can be:

[0127] ;

[0128] ;

[0129] ;

[0130] wherein, is a cross-entropy loss value, is an entropy regularization term loss value, is the second loss value, The training batch size is the training sample set for the second table or the synthetic dataset for the first table. This represents the total number of semantic dimensions in the training sample set for the second table. The first one is pre-set according to actual needs The weight values ​​of each semantic dimension For the first The training batch of the first The semantic dimension probability value of each semantic dimension. For the first The training batch of the first The true semantic dimension of a semantic dimension The above-mentioned values ​​are pre-set according to actual needs. The training batch of the first Each semantic dimension can belong to the training sample set of the second table and / or the synthetic dataset of the first table.

[0131] Step S406: Determine the third loss value based on the second score, the first semantic dimension probability value, and the second semantic dimension probability value;

[0132] Step S407: Update the first table data synthesis model based on the first loss value, the second loss value, and the third loss value to obtain the trained table data synthesis model.

[0133] In some embodiments, step S407 may include steps S501 to S502:

[0134] Step S501: When the first loss value is less than the first preset loss threshold, the second loss value is less than the second preset loss threshold, and the third loss value is less than the third preset loss threshold, the first table data synthesis model is used as the first table data synthesis model.

[0135] Step S502: When the first loss value is greater than or equal to the first preset loss threshold, the second loss value is greater than or equal to the second preset loss threshold, or the third loss value is greater than or equal to the third preset loss threshold, the first discriminator is iteratively updated based on the first loss value, the first classifier is iteratively updated based on the second loss value, and the first generator is iteratively updated based on the third loss value, until the number of iterations reaches the preset number of iterations, and the trained tabular data synthesis model is obtained.

[0136] This application improves the accuracy of model training by updating the first table data synthesis model by combining the first loss value, the second loss value and the third loss value.

[0137] In some embodiments, step S406 may include steps S601 to S603:

[0138] Step S601, determining a cross-entropy loss value based on the first semantic dimension probability value and the second semantic dimension probability value through a weighted multi-class cross-entropy loss function;

[0139] In step S601, determining the cross-entropy loss value based on the first semantic dimension probability value and the second semantic dimension probability value through the weighted multi-class cross-entropy loss function can be determining the cross-entropy loss value based on the first semantic dimension probability value and the second semantic dimension probability value through the weighted multi-class cross-entropy loss function by the following formula:

[0140] .

[0141] Step S602, calculating an expected value of the second score value;

[0142] Step S603, determining a third loss value based on the cross-entropy loss value and the expected value.

[0143] In step S603, determining the third loss value based on the cross-entropy loss value and the expected value can be determining the third loss value based on the cross-entropy loss value and the expected value by the following formula:

[0144] ;

[0145] wherein, the third loss value is L3, the expected value of the second score value is E, is a value set in advance according to actual needs.

[0146] The present application determines the third loss value based on the cross-entropy loss value and the expected value, provides data basis for subsequent model updating, and improves the generalization ability and accuracy of the model.

[0147] In some embodiments, the method can further include steps S701 to S703:

[0148] Step S701, in the case of obtaining a table data synthesis requirement of a user, filtering a corresponding semantic dimension label value from the second semantic dimension label value as a third semantic dimension label value based on the table data synthesis requirement;

[0149] Step S702, inputting the third semantic dimension label value and a second random noise vector into the trained generator to obtain a second table synthesis data set output by the trained generator, wherein the second random noise vector is a random noise vector randomly generated and having a total number same as a total number of the third semantic dimension label value;

[0150] Step S703, sending the second table synthesis data set to the user.

[0151] The application processes different types of data through a trained table data synthesis model, and improves the efficiency and accuracy of table data synthesis.

[0152] The following is a specific table data generation case. Specifically, first, real table data sets in the fields of operators, e-commerce, medical treatment and finance are selected. These data sets cover various statistical distribution types and semantic distributions, and have high representativeness. The features and labels of each data set are organized in table form, facilitating subsequent processing.

[0153] Some experimental data of the application are provided below:

[0154] The application selects four generation models as a control group to generate the data sets in the above four fields, and then compares the performance of each model. The four models include: CTGAN: a widely used table data generation model based on a generative adversarial network; TVAE: a table data generation model based on a variational autoencoder; GReaT: a table data generation model based on a large language model; and CoDi: a table data generation model based on a diffusion model.

[0155] Statistical distribution consistency: two indicators are calculated: average Wasserstein distance (WD) for measuring the distance between numerical feature distributions; and average Jensen-Shannon divergence (JSD) for evaluating the similarity of categorical feature distributions, to quantify to what extent the synthesized data reflects the statistical distribution characteristics of the real data. As shown in Table 1, the embodiment of the application is always better than the other four comparative schemes on all data sets.

[0156] Table 1

[0157]

[0158] Semantic consistency: taking the operator field as a core case, the statistical distribution consistency of the synthesized data under different semantic condition supervision is compared. As shown in Table 2, the embodiment of the application is always better than the GReaT comparative scheme on all data sets.

[0159] Table 2

[0160]

[0161] Downstream utility: For each dataset, the real data is split into 80% training set and 20% test set. The generative model is trained on the real data training set and generates the same size synthetic data. The classifier is trained using synthetic data and evaluated on the real data test set. Accuracy (Acc.) and F1-Score are used as evaluation metrics. Five widely used classifiers (Random Forest, Multi-Layer Perceptron (MLP), XGBoost, Logistic Regression, and Naive Bayes) are tested on four application domains (Telecom customer value prediction, payment method prediction in e-commerce, heart disease diagnosis in medicine, and loan default prediction in finance). The results of all classifiers are averaged to provide a comprehensive evaluation. As shown in the table, the embodiments of the present application are always superior to the other four comparative schemes on all datasets. In particular, Real represents the direct use of real data without any processing by the generative model.

[0162] Table 3

[0163]

[0164] The present application can efficiently process heterogeneous distribution, significantly improve the fidelity of data statistical distribution, multi-dimensional semantic conditional constraint, ensure the semantic consistency of generated data, and support data generation under arbitrary semantic conditions. Cross-domain experiments prove that the method has scalability.

[0165] In addition, with reference to Figure 2 An embodiment of the present application provides a table data synthesis system, which comprises a data construction module 1100, a first deduplication module 1200, a combination module 1300, a second deduplication module 1400, a total number determination module 1500, a first probability value determination module 1600, a data screening module 1700 and a model training module 1800, wherein:

[0166] The data construction module 1100 is used for constructing a first table training sample set, wherein the first table training sample set comprises a plurality of semantic dimensions of each first table training sample and a plurality of semantic dimension label values of each semantic dimension;

[0167] The first deduplication module 1200 is used for deduplicating all semantic dimension label values of the first table training sample set to obtain a first semantic dimension label value set;

[0168] The combination module 1300 is used for combining each semantic dimension of each first table training sample and each semantic dimension label value of each semantic dimension to obtain a first combined label value set;

[0169] The second deduplication module 1400 is used for deduplicating the first combined label value in the first combined label value set to obtain a second combined label value set;

[0170] The total number determination module 1500 is configured to determine the number of occurrences of each second combined label value in the first combined label value set as the total number of each second combined label value.

[0171] The first probability value determination module 1600 is configured to determine the first probability value of each first semantic dimension label value based on the total number of each second combined label value.

[0172] The data screening module 1700 is configured to screen the first table training sample set based on the first probability value and the preset number of training samples to obtain a second table training sample set.

[0173] The model training module 1800 is configured to construct a first table data synthesis model, train the first table data synthesis model based on the second table training sample set, and obtain a trained table data synthesis model.

[0174] The system can extract the core categories of semantic dimensions, eliminate redundant information interference, provide accurate and efficient label basis for subsequent model training, and then determine the first probability value of each first semantic dimension label value. The first table training sample set is screened based on the first probability value and the preset number of training samples to obtain a second table training sample set. The first table training sample set is screened based on the first probability value, which can reduce the interference of extreme samples on subsequent model training and control the sample size to balance the training efficiency and accuracy of the model. Finally, the first table data synthesis model is constructed, the first table data synthesis model is trained based on the second table training sample set, and a trained table data synthesis model is obtained, which realizes processing of different types of data through a trained table data synthesis model and improves the efficiency and accuracy of table data synthesis.

[0175] It should be noted that the system embodiment and the method embodiment described above are based on the same inventive concept, and therefore the related content of the method embodiment described above is also applicable to the system embodiment, which will not be described here.

[0176] Figure 3 A hardware structure schematic diagram of table data synthesis provided by an embodiment of the present application is shown.

[0177] The table data synthesis device can include a processor 301 and a memory 302 having computer program instructions stored therein.

[0178] In particular, the processor 301 can include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to perform the operations of the embodiments of the application.

[0179] The memory 302 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 302 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a solid-state drive (SSD), a USB drive, or a combination of two or more of these. Where appropriate, the memory 302 can include removable or non-removable (or fixed) media, where appropriate. Where appropriate, the memory 302 can be internal or external to the integrated gateway disaster recovery device. In some embodiments, the memory 302 is non-volatile, solid-state memory.

[0180] In some embodiments, the memory 302 can include read-only memory (ROM), random-access memory (RAM), a magnetic disk storage medium, an optical storage medium, a flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to perform operations described with reference to the methods according to an aspect of the present disclosure.

[0181] The processor 301 implements any of the table data synthesis methods in the above embodiments by reading and executing computer program instructions stored in the memory 302.

[0182] In one example, the table data synthesis device can further include a communication interface 303 and a bus 310. As shown, the processor 301, the memory 302, and the communication interface 303 are connected through the bus 310 and complete communication with each other. Figure 3

[0183] The communication interface 303 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the application.

[0184] ​Bus 310 includes a hardware, software, or both that couples components of the table data synthesis device to each other. As an example and not by way of limitation, the bus can include an accelerated graphics port (AGP) or other graphics bus, an enhanced industry standard architecture (EISA) bus, a front-side bus (FSB), a HyperTransport (HT) interconnect, an industry standard architecture (ISA) bus, an infiniband (IB) interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standards board (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 310 can include one or more buses. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0185] The table data synthesis device can perform the table data synthesis method in the embodiments of the application based on a three-dimensional design model, thereby realizing the table data synthesis method and system described in the embodiments of the application. Figure 1 and Figure 2 The table data synthesis method and system described in the embodiments of the application.

[0186] In addition, in combination with the table data synthesis method in the above embodiments, the embodiments of the application can provide a computer storage medium to realize. The computer storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to realize any one of the table data synthesis methods in the above embodiments.

[0187] It needs to be clear that the application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the application.

[0188] The functional blocks shown in the structural block diagrams above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium, or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. A "machine-readable medium" includes any medium that can store or transport information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROMs, flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments can be downloaded via computer networks such as the Internet, intranets, and the like.

[0189] It is also important to note that the examples mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the steps mentioned above, that is, the steps can be performed in the order mentioned in the examples, or in an order different from the examples, or several steps can be performed simultaneously.

[0190] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing devices to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0191] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A method of tabular data synthesis, characterized by, The table data synthesis method comprises: constructing a first table training sample set, wherein the first table training sample set comprises a plurality of semantic dimensions of each first table training sample and a plurality of semantic dimension label values of each semantic dimension; de-duplicating all the semantic dimension label values of the first table training sample set to obtain a first semantic dimension label value set; combining each semantic dimension of each first table training sample and each semantic dimension label value of each semantic dimension to obtain a first combined label value set; de-duplicating first combined label values in the first combined label value set to obtain a second combined label value set; determining a total number of occurrences of each second combined label value in the first combined label value set as a total number of each second combined label value; determining a first probability value of each first semantic dimension label value based on the total number of each second combined label value; screening the first table training sample set based on the first probability value and a preset training sample number to obtain a second table training sample set; constructing a first table data synthesis model, training the first table data synthesis model based on the second table training sample set, and obtaining a trained table data synthesis model.

2. The method of claim 1, wherein, Before the step of determining a first probability value of each first semantic dimension label value based on the total number of each second combined label value, the method comprises: constructing a first semantic constraint table based on the first semantic dimension label value set, wherein the first semantic constraint table is a hash structure; the step of determining a first probability value of each first semantic dimension label value based on the total number of each second combined label value comprises: determining a second probability value of each second combined label value based on the total number of all second combined label values; determining a first probability value of each first semantic dimension label value based on the second probability value and the first semantic constraint table.

3. The method of claim 2, wherein, The step of screening the first table training sample set based on the first probability value and a preset training sample number to obtain a second table training sample set comprises: updating the first semantic constraint table based on the first probability value to obtain a second semantic constraint table, wherein the key of the second semantic constraint table is a first semantic dimension label value in the first semantic dimension label value set, and the value of the second semantic constraint table is the first probability value of each first semantic dimension label value; randomly sampling the second semantic constraint table by a weighted random sampling method based on the preset training sample number to obtain a second semantic dimension label value; randomly extracting a first table training sample from the first table training sample set based on the second semantic dimension label value to obtain a second table training sample set, wherein the second semantic dimension label value corresponds to a second table training sample in the second table training sample set in a one-to-one manner.

4. The method of claim 3, wherein, The first table data synthesis model comprises a first generator, a first discriminator and a first classifier, the first table data synthesis model is trained based on the second table training sample set, and a trained table data synthesis model is obtained, comprising: inputting the second semantic dimension label value and a first random noise vector into the first generator to obtain first table synthesis data set output by the first generator, wherein first table synthesis data in the first table synthesis data set corresponds to the second semantic dimension label value one by one, and the first random noise vector is a random noise vector randomly generated and having a total number same as that of the second semantic dimension label value; inputting the second table training sample set into the first discriminator to obtain first score value output by the first discriminator; and inputting the first table synthesis data set into the first discriminator to obtain second score value output by the first discriminator; determining first loss value based on the first score value and the second score value through a Wasserstein distance loss function with gradient penalty; inputting the second table training sample set into the first classifier to obtain first semantic dimension probability value output by the first classifier; and inputting the first table synthesis data set into the first classifier to obtain second semantic dimension probability value output by the first classifier; determining second loss value based on the first semantic dimension probability value and the second semantic dimension probability value through a weighted multi-class cross-entropy loss function and an entropy regularization term method; determining third loss value based on the second score value, the first semantic dimension probability value and the second semantic dimension probability value; updating the first table data synthesis model based on the first loss value, the second loss value and the third loss value to obtain the trained table data synthesis model.

5. The method of claim 4, wherein, The updating of the first table data synthesis model based on the first loss value, the second loss value and the third loss value to obtain the trained table data synthesis model comprises: in a case where the first loss value is less than a first preset loss threshold, the second loss value is less than a second preset loss threshold and the third loss value is less than a third preset loss threshold, taking the first table data synthesis model as the first table data synthesis model; in a case where the first loss value is greater than or equal to the first preset loss threshold, the second loss value is greater than or equal to the second preset loss threshold or the third loss value is greater than or equal to the third preset loss threshold, iteratively updating the first discriminator based on the first loss value, iteratively updating the first classifier based on the second loss value and iteratively updating the first generator based on the third loss value until a preset iteration number is reached, and obtaining the trained table data synthesis model.

6. The method of claim 4, wherein, The determining of the third loss value based on the second score value, the first semantic dimension probability value and the second semantic dimension probability value comprises: determine a cross-entropy loss value through a weighted multi-class cross-entropy loss function based on the first semantic dimension probability value and the second semantic dimension probability value; calculate an expected value of the second score value; determine the third loss value based on the cross-entropy loss value and the expected value.

7. The method of claim 4, wherein, The trained table data synthesis model comprises a trained generator, and the method further comprises: In the case that a table data synthesis requirement of a user is acquired, filtering a corresponding semantic dimension label value from the second semantic dimension label value as a third semantic dimension label value based on the table data synthesis requirement; inputting the third semantic dimension label value and a second random noise vector into the trained generator to obtain a second table synthesis dataset output by the trained generator, wherein the second random noise vector is a random noise vector randomly generated and having a same total number as a total number of the third semantic dimension label value; sending the second table synthesis dataset to the user.

8. A tabular data synthesis system characterized by, The table data synthesis system comprises: a data construction module configured to construct a first table training sample set, wherein the first table training sample set comprises a plurality of semantic dimensions of each first table training sample and a plurality of semantic dimension label values of each semantic dimension; a first deduplication module configured to deduplicate all semantic dimension label values of the first table training sample set to obtain a first semantic dimension label value set; a combination module configured to combine each semantic dimension of each first table training sample and each semantic dimension label value of each semantic dimension to obtain a first combined label value set; a second deduplication module configured to deduplicate first combined label values in the first combined label value set to obtain a second combined label value set; a total number determination module configured to determine a number of times each second combined label value in the second combined label value set appears in the first combined label value set as a total number of each second combined label value; a first probability value determination module configured to determine a first probability value of each first semantic dimension label value based on the total number of each second combined label value; a data filtering module configured to filter the first table training sample set based on the first probability value and a preset training sample number to obtain a second table training sample set; a model training module configured to construct a first table data synthesis model, train the first table data synthesis model based on the second table training sample set, and obtain a trained table data synthesis model.

9. A table data synthesizing apparatus characterized by comprising: The at least one control processor and the memory connected in communication with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform a table data synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer executable instructions for causing a computer to perform a table data synthesis method according to any one of claims 1 to 7.

Images