Structured data synthesis method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a structured data synthesis method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of field data in a plurality of pieces of obtained structured data, and obtaining a field feature vector corresponding to each piece of field data; for each clustering level, based on the field feature vectors of all the corresponding field data, selecting a target data synthesis model corresponding to the clustering level from a plurality of data synthesis models; marginal distribution of the field type corresponding to each clustering hierarchy in the multiple pieces of structured data is obtained, for each clustering hierarchy, data processing is conducted on the input data through the corresponding target data synthesis model to obtain synthesized output data, the input data of the first layer is the corresponding marginal distribution, and the input data of the second layer is the corresponding marginal distribution; the input data of other layers are the synthetic output data of the previous layer and the marginal distribution of the current layer; structured synthetic data is obtained based on synthetic output data corresponding to all clustering hierarchies, and the universality and adaptability of the whole synthetic method are enhanced.

Description

Technical Field

[0001] This application relates to the field of structured data processing technology, and in particular to structured data synthesis methods, apparatus, electronic devices and storage media. Background Technology

[0002] In the field of structured data synthesis, the core objective is to generate artificial data that is highly similar to real data in statistical properties for use in scenarios such as data augmentation, privacy protection, or system testing. In related technologies, structured data synthesis primarily relies on a single synthesis model to accomplish this task. For example, the Gaussian Copula method excels at simulating linear dependencies between continuous variables, while methods based on Generative Adversarial Networks (GANs) such as CTGAN and TVAE demonstrate advantages in modeling complex data distributions.

[0003] However, solutions relying on a single model struggle to maintain overall data quality across datasets with diverse distribution characteristics (such as normal, long-tailed, multimodal discrete, etc.), often resulting in compromises and poor versatility of existing structured data synthesis methods. Summary of the Invention

[0004] This application provides a structured data synthesis method, apparatus, electronic device, and storage medium, which can improve the versatility of the structured data synthesis method.

[0005] To achieve the above objectives, a first aspect of this application proposes a structured data synthesis method, the method comprising: Feature extraction is performed on the field data in the acquired multiple structured data to obtain a field feature vector corresponding to each field data. The field feature vector includes a clustering level index used to indicate the clustering level of the field data. For each clustering level, based on the field feature vectors of all the corresponding field data, a target data synthesis model corresponding to the clustering level is selected from multiple data synthesis models; Obtain the marginal distribution of the field type corresponding to each clustering level in multiple pieces of structured data. For each clustering level, use the corresponding target data synthesis model to process the input data to obtain synthesized output data. The input data of the first layer is the corresponding marginal distribution, and the input data of other layers is the synthesized output data of the previous layer and the marginal distribution of the current layer. Based on the synthetic output data corresponding to all the clustering levels, structured synthetic data is obtained.

[0006] In some embodiments, the process of acquiring the multiple pieces of structured data includes: Obtain multiple initial structured data sets, identify missing structured data sets with missing data from the multiple initial structured data sets, and determine the marginal distribution of each field type based on the complete observation data from the multiple initial structured data sets; The cumulative probability value of each complete observation data is obtained by combining the complete observation data and the marginal distribution corresponding to each of the field types; Based on the cumulative probability values, a correlation matrix between the field types is obtained, and the correlation matrix is ​​used as a connection parameter to construct a multivariate Gaussian connection model. The observed data in the missing structured data is input into the marginal distribution corresponding to the field type to obtain the cumulative probability value of the observation, and the cumulative probability value is input into the multivariate Gaussian connection model to obtain the missing cumulative probability value corresponding to the missing data in the missing structured data. The cumulative probability value of missing data is input into the inverse distribution of the marginal distribution corresponding to the field type of the missing data to obtain the filling data, and the filling data is used to update the missing structured data to obtain the structured data.

[0007] In some embodiments, the step of extracting features from the field data in the acquired multiple pieces of structured data to obtain a field feature vector corresponding to each field data includes: When the field data is a standard type field, the statistical characteristics of each field data are calculated one by one. Based on the statistical characteristics and statistical test methods, the distribution type of each field data is determined, and based on the distribution type, the corresponding feature type code of the field data is obtained. When the field data is a non-standard type field, calculate the empirical quantile vector of the field data; Calculate the correlation coefficient between each of the field data and the other field data one by one, and construct a correlation matrix among all the field data based on all the correlation coefficients; Based on the correlation matrix, the correlation distance between all the field data is determined, and hierarchical clustering is performed on all the field data based on the correlation distance to obtain multiple clustering levels and the clustering level index corresponding to each field data. Based on the feature type encoding, the clustering level index, and / or the empirical quantile vector, the field feature vector corresponding to each field data is obtained.

[0008] In some embodiments, the statistical features include field skewness and field kurtosis, and the statistical tests include normality tests and multimodal tests. Determining the distribution type of each field data based on the statistical features and statistical tests includes: The field data is tested and calculated based on the normality test method to obtain a normal distribution confidence value. Based on the numerical relationship between the normal distribution confidence value and the preset distribution threshold, the normal distribution type result is obtained. The field skewness is compared with a preset skewness threshold, and the field kurtosis is compared with a preset kurtosis threshold to obtain the long-tail distribution type result; The field data is tested and calculated using statistical testing methods to obtain a multi-peak distribution confidence value. Based on the numerical relationship between the multi-peak distribution confidence value and a preset distribution threshold, the multi-peak distribution type result is obtained. Based on the normal distribution type result, the long-tailed distribution type result, and the multimodal distribution type result, the distribution type corresponding to the field data is obtained.

[0009] In some embodiments, obtaining the field feature vector corresponding to each field data based on the feature type encoding, the clustering level index, and / or the empirical quantile vector includes: Based on the field type corresponding to each of the field data, obtain the corresponding sensitivity value; Obtain sample values ​​from multiple pieces of the structured data; Based on the feature type encoding, the clustering level index, the sensitivity value, the sample value, and / or the empirical quantile vector, the field feature vector corresponding to each field data is obtained.

[0010] In some embodiments, for each clustering level, selecting a target data synthesis model corresponding to the clustering level from multiple data synthesis models based on the field feature vectors of all the corresponding field data includes: Based on the field feature vectors of all the field data in the clustering level, quantized weights corresponding to multiple feature types are obtained; Based on the quantization weight and scoring function corresponding to each feature type, a score is calculated for each data synthesis model to obtain a type performance score for each data synthesis model corresponding to each feature type. The performance scores of the multiple types are weighted to obtain the performance score of each data synthesis model, and the data synthesis model with the highest performance score is selected from the multiple data synthesis models as the target data synthesis model corresponding to the clustering level.

[0011] In some embodiments, the feature types include distribution features, correlation features, sensitivity features, and scale features, and the quantization weights include distribution quantization weights, correlation quantization weights, sensitivity quantization weights, and scale quantization weights. The quantization weights corresponding to multiple feature types are obtained based on the field feature vectors of all field data at the clustering level, including: Euclidean norm calculation is performed on the feature type encoding of all field feature vectors at the clustering level to obtain the distribution quantization weights corresponding to the distribution features; Extract the largest correlation coefficient from all the field feature vectors of the clustering level, and use it as the correlation quantification weight corresponding to the correlation feature; The sensitivity values ​​are extracted from all the field feature vectors of the clustering level and used as the sensitivity quantization weights corresponding to the sensitivity features. Extract sample values ​​from all the field feature vectors of the clustering level, and perform logarithmic processing on the sample values ​​to obtain the scale quantization weights corresponding to the scale features.

[0012] In some embodiments, obtaining structured synthetic data based on the synthetic output data corresponding to all the clustering levels includes: Based on the synthetic output data corresponding to all the clustering levels, the initial structured synthetic data is obtained.

[0013] The initial structured synthetic data is input into a residual mapping network to obtain residual adjustment values. Based on the residual adjustment values, the initial structured synthetic data is adjusted to obtain multiple pieces of structured synthetic data corresponding to the structured data.

[0014] In some embodiments, the training process of the residual mapping network includes: Acquire multiple structured sample data; Each of the structured sample data is synthesized to obtain synthesized sample data, and the statistical deviation characteristics between each of the synthesized sample data and the corresponding structured sample data are calculated. Missing data imputation is performed on each of the structured sample data to obtain sample imputation data. The sample imputation data, the statistical bias features, and the context information of the structured sample data are then input into the initial residual mapping network for data processing to obtain the training residual adjustment value. The initial residual mapping network is trained and updated multiple times based on the loss function, and the residual mapping network is obtained based on the initial residual mapping network after multiple training and updates. The loss function is obtained based on the difference between the training residual adjustment value and the preset correction increment.

[0015] In some embodiments, there are multiple residual mapping networks, and the step of inputting the initial structured synthesis data into the residual mapping network to obtain residual adjustment values ​​includes: The initial structured synthesis data is input into each of the residual mapping networks one by one to obtain the initial residual adjustment value corresponding to each residual mapping network; Based on the residual error and stability index of each residual mapping network, the overall network performance of each residual mapping network is calculated, and the network weight of each residual mapping network is obtained based on the overall network performance. The initial residual adjustment values ​​are weighted and fused based on the network weights of the multiple residual mapping networks to obtain the residual adjustment values.

[0016] In some embodiments, the method further includes: Based on the sensitivity value in the feature vector of each field data, the encryption budget allocation weight corresponding to each field data is obtained; Based on the feature type encoding in the feature vector corresponding to each field data, the noise addition mechanism corresponding to each field data is obtained; The corresponding field data is encrypted based on the encryption budget allocation weight and the noise-adding mechanism.

[0017] To achieve the above objectives, a second aspect of this application provides a structured data synthesis apparatus, the apparatus comprising: The feature extraction module is used to extract features from the field data in the acquired multiple structured data to obtain a field feature vector corresponding to each field data. The field feature vector includes a clustering level index for indicating the clustering level of the field data. The synthesis model selection module is used to select the target data synthesis model corresponding to the clustering level from multiple data synthesis models based on the field feature vector of all the corresponding field data for each clustering level. The hierarchical synthesis processing module is used to obtain the marginal distribution of the field type corresponding to each clustering level in multiple pieces of structured data. For each clustering level, the input data is processed using the corresponding target data synthesis model to obtain synthesized output data. The input data of the first layer is the corresponding marginal distribution, and the input data of other layers is the synthesized output data of the previous layer and the marginal distribution of the current layer. The overall synthesis module is used to obtain structured synthesized data based on the synthesis output data corresponding to all the clustering levels.

[0018] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the structured data synthesis method as described in the first aspect.

[0019] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the structured data synthesis method as described in the first aspect.

[0020] The structured data synthesis method, apparatus, electronic device, and storage medium proposed in this application include the following steps: First, feature extraction is performed on field data from multiple pieces of acquired structured data to obtain a field feature vector corresponding to each field data. The field feature vector includes a clustering level index indicating the clustering level of the field data. Then, for each clustering level, a target data synthesis model corresponding to the clustering level is selected from multiple data synthesis models based on the field feature vectors of all corresponding field data. Next, the marginal distribution of field types corresponding to each clustering level in the multiple pieces of structured data is obtained. For each clustering level, the input data is processed using the corresponding target data synthesis model to obtain synthesized output data. The input data of the first layer is the corresponding marginal distribution, and the input data of other layers are the synthesized output data of the previous layer and the marginal distribution of the current layer. Finally, structured synthesized data is obtained based on the synthesized output data corresponding to all clustering levels. This application embodiment deeply analyzes the inherent statistical characteristics and complex hierarchical dependencies of each data field through feature extraction and hierarchical clustering. Then, for each field level with different characteristics, the most suitable target data synthesis model is dynamically and intelligently selected from multiple candidate models. By performing optimal model matching and generating conditions layer by layer according to hierarchy and features, it ensures that fields in the dataset, whether long-tailed, multimodal, or normally distributed, can be processed most appropriately. This significantly improves the fidelity of the synthesized data in terms of marginal distribution and field correlation, ultimately obtaining structured synthesized data of higher quality with statistical characteristics closer to real data, and effectively enhancing the versatility and adaptability of the entire synthesis method.

[0021] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0022] Figure 1This is a flowchart of a structured data synthesis method provided in an embodiment of this application.

[0023] Figure 2 This is a flowchart of a structured data acquisition process provided in another embodiment of this application.

[0024] Figure 3 yes Figure 1 The flowchart for step 101.

[0025] Figure 4 yes Figure 3 The flowchart for step 301.

[0026] Figure 5 yes Figure 3 The flowchart for step 305.

[0027] Figure 6 This is a schematic diagram of a field feature vector generation process provided in another embodiment of this application.

[0028] Figure 7 yes Figure 1 The flowchart for step 102.

[0029] Figure 8 yes Figure 7 The flowchart for step 701.

[0030] Figure 9 This is a schematic diagram of a process for selecting the optimal synthesis model and performing structured data synthesis, provided in another embodiment of this application.

[0031] Figure 10 yes Figure 1 The flowchart for step 104.

[0032] Figure 11 This is a flowchart of the training process of a residual mapping network provided in another embodiment of this application.

[0033] Figure 12 yes Figure 10 The flowchart for step 1002.

[0034] Figure 13 This is a schematic diagram of a process for adjusting residuals in initial structured synthetic data according to an embodiment of this application.

[0035] Figure 14 This is a flowchart illustrating the security processing in a structured data synthesis process, provided in one embodiment of this application.

[0036] Figure 15 This is a schematic diagram of a security and privacy protection process in structured data processing provided in an embodiment of this application.

[0037] Figure 16 This is a schematic diagram of the framework of a structured data synthesis system provided in one embodiment of this application.

[0038] Figure 17 This is a schematic diagram of the structure of a structured data synthesis apparatus provided in an embodiment of this application.

[0039] Figure 18 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0041] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0043] In the field of structured data synthesis, the core objective is to generate artificial data that is highly similar to real data in statistical properties for use in scenarios such as data augmentation, privacy protection, or system testing. Related technologies primarily rely on a single synthesis model or a fixed multi-model fusion scheme to accomplish this task. For example, the Gaussian Copula method excels at simulating linear dependencies between continuous variables, while methods based on Generative Adversarial Networks (GANs) such as CTGAN and TVAE demonstrate advantages in modeling complex data distributions. Another common strategy is to run multiple different synthesis models in parallel, and then weight and fuse the results generated by each model according to empirically or manually configured fixed weights, hoping to improve the diversity and quality of the synthesized data.

[0044] However, the above methods have significant drawbacks. First, schemes relying on a single model have poor generality and struggle to maintain data quality across datasets with diverse distribution characteristics (such as normal, long-tailed, multimodal discrete, etc.), often resulting in compromises. Second, multi-model strategies employing fixed-weight fusion typically derive their weights from empirical values ​​and cannot adaptively adjust them based on the data's inherent characteristics. When faced with complex, high-dimensional mixed distributions, this static fusion approach easily leads to biases in the marginal distribution of the generated data or distortions in the correlations between fields. Therefore, existing technologies generally lack a mechanism that can dynamically and intelligently select and apply the most suitable generative model based on the characteristics of different fields within the data and their hierarchical dependencies, resulting in poor fidelity and usability of the synthesized data.

[0045] To improve the versatility of structured data synthesis methods, this application embodiment uses feature extraction and hierarchical clustering to deeply analyze the inherent statistical characteristics and complex hierarchical dependencies of each data field. Then, for each field level with different characteristics, the most suitable target data synthesis model is dynamically and intelligently selected from multiple candidate models. By performing optimal model matching and generating conditions layer by layer according to hierarchy and features, it is ensured that fields in the dataset, whether long-tailed, multimodal, or normally distributed, can be processed most appropriately. This significantly improves the fidelity of the synthesized data in terms of marginal distribution and field correlation, ultimately obtaining higher-quality structured synthesized data with statistical characteristics closer to real data, and effectively enhancing the versatility and adaptability of the entire synthesis method.

[0046] The structured data synthesis method, apparatus, electronic device, and storage medium in the embodiments of this application will be described in detail below. First, the structured data synthesis method in the embodiments of this application will be described. (Refer to...) Figure 1 This is an optional flowchart of the structured data synthesis method provided in the embodiments of this application. Figure 1 The method described may include, but is not limited to, steps 101 to 104. It is also understood that this embodiment... Figure 1 The order of steps 101 to 104 is not specifically limited; the order of steps can be adjusted or certain steps can be added or removed according to actual needs. The structured data synthesis method provided in this application can be applied to any control system with computing resources (such as servers, computers, etc.).

[0047] Step 101: Extract features from the field data of the acquired multiple structured data to obtain the field feature vector corresponding to each field data. The field feature vector includes a clustering level index used to indicate the clustering level of the field data.

[0048] Step 101 will be described in detail below.

[0049] In some embodiments, feature extraction is first performed on the acquired multiple pieces of structured data that require structured synthesis. Here, structured data refers to tabular data composed of multiple fields (i.e., data columns). The data within each field is analyzed to generate a standardized field feature vector. This field feature vector is a multi-dimensional vector containing a comprehensive description of the statistical characteristics of the field's data, such as distribution type, empirical quantiles, sensitivity level, and data size. Furthermore, the vector includes a clustering hierarchy index. This index is derived through hierarchical clustering analysis of the correlations between all fields, clarifying the hierarchy of each field in the data dependency relationship and laying the foundation for subsequent data generation according to dependency order.

[0050] In addition, since there may be missing data in the multiple structured data sets obtained, it is necessary to perform corresponding correction processing on the multiple structured data sets in advance, as described below.

[0051] Reference Figure 2 The process of acquiring multiple pieces of structured data includes the following steps 201 to 205.

[0052] Step 201: Obtain multiple initial structured data sets, identify missing structured data sets with missing data from the multiple initial structured data sets, and determine the marginal distribution of each field type based on the complete observation data from the multiple initial structured data sets.

[0053] Step 202: Combine the complete observation data and marginal distribution corresponding to each field type to obtain the cumulative probability value of each complete observation data.

[0054] Step 203: Based on the cumulative probability values, obtain the correlation matrix between field types, and use the correlation matrix as the connection parameter to construct a multivariate Gaussian connection model.

[0055] Step 204: Input the observed data from the missing structured data into the marginal distribution corresponding to the field type to obtain the cumulative probability value of the observations, and input the cumulative probability value into the multivariate Gaussian connection model to obtain the missing cumulative probability value corresponding to the missing data in the missing structured data.

[0056] Step 205: Input the cumulative probability value of missing data into the inverse distribution of the marginal distribution corresponding to the field type of the missing data to obtain the filling data, and update the missing structured data with the filling data to obtain the structured data.

[0057] Steps 201 to 205 are described in detail below.

[0058] In some embodiments, the acquired initial structured data is first preprocessed. The primary task is to identify records in the dataset with missing data in one or more fields; these records are identified as missing structured data. Simultaneously, the system utilizes complete observation data in the dataset that does not contain any missing values ​​to determine the marginal distribution for each field type individually. Here, "marginal distribution" refers to a mathematical model describing the probability distribution of individual field data, such as a normal distribution or a log-normal distribution. Determining this model is the foundation for all subsequent calculations, accurately characterizing the inherent statistical properties of each data field.

[0059] In one example, given the initial structured data containing a set of fields {transaction log, credit score} with missing values ​​(i.e., missing structured data), the system first fits the marginal distribution of each field type using complete observation data. By comparing the fitting effects of candidate distributions such as normal, log-normal, and gamma distributions, the best distribution type is selected and the corresponding parameters (i.e., the marginal distribution) are estimated. For fields that do not conform to the standard distribution, nonparametric fitting is performed using kernel density estimation or an empirical distribution function.

[0060] Then, the system will use the marginal distribution of each field type determined in the previous step to transform the complete observation data. Specifically, for each field type, the system will transform its corresponding complete observation data. Substitute each value into the cumulative distribution function (CDF) corresponding to the marginal distribution of the field (i.e., The calculation is performed within this range. The result of this calculation is the cumulative probability value corresponding to each complete set of observations. This cumulative probability value is a number between 0 and 1, representing the relative position of the original data point within the distribution of this field. Through this step, heterogeneous data from different distributions are uniformly transformed into the same probability space, providing a standardized basis for the next step of accurately measuring their correlation.

[0061] Furthermore, based on the cumulative probability values ​​obtained after the standardization process in the previous step, the system will further calculate and construct a correlation matrix, which quantifies the strength and direction of dependencies between different field types. Subsequently, this correlation matrix will serve as a core join parameter to construct a multivariate Gaussian join model. The "multivariate Gaussian join model," or multivariate Gaussian Copula model, is a powerful statistical tool that can separate the marginal distributions of multiple variables from their correlation structure (i.e., join functions), thus flexibly describing the joint distribution of multiple variables.

[0062] In one example, the observations for each field type are first converted into the cumulative probability values ​​of the corresponding marginal distribution. These probability values ​​are then transformed into standard normal variables through inverse standard normal transformation. Finally, the correlation matrix between the z variables is calculated as the Copula parameters. Then, the multivariate Gaussian Copula function (i.e., the multivariate Gaussian Copula model) is established: Where C represents the multivariate Gaussian Copula function, arrive It is the cumulative probability value of the marginal distribution of each field. It is the inverse function of the standard normal distribution. With a mean of 0 and a covariance matrix The multivariate normal distribution function.

[0063] In one embodiment, a pre-built multivariate Gaussian join model is used to predict missing data. The system first converts existing observations from a missing structured dataset into cumulative probability values ​​using the marginal distribution of their corresponding field types. These cumulative probability values ​​are then input into the multivariate Gaussian join model as conditions. The model utilizes the established correlation structure to calculate the most probable probability distribution of the missing data under those conditions and samples the corresponding cumulative probability value for the missing data. Essentially, this process infers unknown information within a probability space based on existing information.

[0064] In one example, for each sample containing missing values, the observed field values ​​are first converted into cumulative probabilities. Then, based on the conditional distribution properties of the multivariate normal distribution, the conditional Copula distribution of the missing field under the given observed field conditions is calculated. The conditional distribution remains a normal distribution, and its mean and variance can be directly calculated through block matrix operations.

[0065] Finally, the system performs the inverse process of data transformation to obtain the final imputation values. The cumulative probability value of missing data obtained in the previous step is input into the inverse distribution function (i.e., the quantile function) of the marginal distribution of the field type corresponding to the missing data. Through this inverse calculation, the standardized cumulative probability value is restored to a specific value with the scale of the original data; this value is the imputation data. Finally, the system updates the corresponding position in the missing structured data with the imputation data, completing the repair of a single record. This process is repeated until all missing data is filled, resulting in a complete set of structured data that can be used for subsequent analysis.

[0066] In one example, from the conditional distribution The cumulative probability value of the missing field is obtained by random sampling. (i.e., the cumulative probability value of missing values), and then through the inverse function of the corresponding marginal distribution. The probability values ​​are transformed back to the original data scale to obtain the imputed values ​​(i.e., the imputed data) for the missing locations. For fields that cannot be fitted by a standard distribution, an empirical quantile function or the inverse transformation of kernel density estimation is used for transformation. Then, the imputed quality is verified and iteratively optimized: the imputed result is verified to ensure that the original correlation structure is maintained, i.e., the correlation matrix of the imputed data is similar to that of the original correlation matrix. near.

[0067] Through steps 201 to 205 above, compared to using simple methods such as mean and median, this method adopts a multivariate Gaussian connection model. Its core advantage lies in its ability to accurately preserve the complex, non-linear correlation structure between fields in the original data. By separating the marginal distribution of each field from its joint correlation structure, it can not only generate reasonable filling values ​​for fields with different distribution types, but more importantly, it ensures that the filled data does not destroy the inherent statistical dependencies of the entire dataset, thereby greatly improving the quality and authenticity of the filled data and providing a more reliable data foundation for subsequent data synthesis or other analysis tasks.

[0068] The following describes how to extract features from field data in structured data.

[0069] Reference Figure 3 The process involves extracting features from the field data of the acquired multiple structured data sets to obtain the field feature vector corresponding to each field data set, including the following steps 301 to 305.

[0070] Step 301: When the field data is a standard type field, calculate the statistical characteristics of each field data one by one. Based on the statistical characteristics and statistical test methods, determine the distribution type of each field data, and based on the distribution type, obtain the feature type code of the corresponding field data.

[0071] Step 301 will be described in detail below.

[0072] First, the system processes the field data that can be categorized into standard type fields. It calculates the statistical characteristics of each field, where "statistical characteristics" refers to indicators that quantify the shape of the data distribution, such as skewness (measuring data symmetry) and kurtosis (measuring the thickness of the data tails). Then, based on these statistical characteristics and combined with various statistical tests, such as the Shapiro-Wilk test for normality and Hartigan's dip test for multimodality, the system determines the distribution type of each field, such as normal, long-tailed, or multimodal discrete distribution. Finally, this determined distribution type is converted into a standardized feature type encoding, such as one-hot encoding, for subsequent machine processing. This will be described in more detail below.

[0073] In some embodiments, statistical feature calculation includes calculating field skewness. and field kurtosis The skewness and tail thickness of the data distribution are quantified, where μ is the mean parameter and σ is the variance parameter.

[0074] Reference Figure 4 The statistical test methods include normality test methods and multimodal test methods. Based on statistical characteristics and statistical test methods, the distribution type of each field data is determined, including the following steps 401 to 404.

[0075] Step 401: Perform a test calculation on the field data based on the normality test method to obtain the normality distribution confidence value. Based on the numerical relationship between the normality distribution confidence value and the preset distribution threshold, obtain the normality distribution type result.

[0076] Step 402: Compare the field skewness with the preset skewness threshold and the field kurtosis with the preset kurtosis threshold to obtain the long-tail distribution type result.

[0077] Step 403: Perform statistical testing on the field data to obtain the confidence value of the multimodal distribution. Based on the numerical relationship between the confidence value of the multimodal distribution and the preset distribution threshold, obtain the result of the multimodal distribution type.

[0078] Step 404: Based on the results of the normal distribution type, the long-tailed distribution type, and the multimodal distribution type, obtain the distribution type corresponding to the field data.

[0079] Steps 401 to 404 are described in detail below.

[0080] In some embodiments, this method first performs a normality test on the field data to determine whether it conforms to a normal distribution. The system uses a preset normality test method, such as the Shapiro-Wilk test, to perform the test calculation on the field data. This calculation generates a normality confidence value, where "confidence value" typically refers to the p-value in statistical tests. The system then compares this obtained normality confidence value with a preset distribution threshold (e.g., a significance level of 0.05). Based on this numerical relationship, for example, when the confidence value is greater than the threshold, the system determines the normality type of the field data, that is, it is determined that it approximately follows a normal distribution. It is judged to be approximately normally distributed.

[0081] Next, this method determines the long-tail distribution characteristics of the field data. This process does not rely on specific hypothesis testing, but is based on two key statistical features of the field data: field skewness and field kurtosis. The system compares the calculated field skewness (a measure of distribution asymmetry) with a preset skewness threshold (e.g., absolute value greater than 1.0), and simultaneously compares the field kurtosis (a measure of the thickness of the distribution tails) with a preset kurtosis threshold (e.g., greater than 3.0). Only when both field skewness and field kurtosis exceed their respective preset thresholds will the system determine the long-tail distribution type of the field data, i.e., determine that it has significant long-tail characteristics. and It was determined to be a long-tailed distribution.

[0082] To identify the presence of multiple concentrated data clusters, this method performs multimodality detection on the field data. The system employs a specialized statistical test, such as Hartigan's dip test, to evaluate the degree to which the field data deviates from a unimodal distribution. This test outputs a multimodal distribution confidence value (usually a p-value), reflecting the probability that the data is unimodal. The system then compares this confidence value to a preset distribution threshold (e.g., 0.05). When the confidence value is less than this threshold, it indicates that the data exhibits statistically significant multimodal characteristics. Based on this, the system determines the multimodal distribution type of the field data, i.e., using Hartigan's dip test to detect multimodality. This indicates the presence of multi-peak characteristics.

[0083] Finally, this method integrates the results of all the preceding steps to determine a final, definite distribution type for each field. The system integrates the obtained normal distribution type results, long-tail distribution type results, and multimodal distribution type results. Through a preset decision logic, such as setting a decision priority, the system selects from multiple possible results to obtain the final distribution type corresponding to the field data. For example, a field that passes the long-tail distribution determination will be classified as a long-tail distribution, even if it fails the normality test. This step ensures that each field is assigned a classification label that best reflects its data characteristics.

[0084] In one example, one-hot encoding is used: normal distribution: [0,1,0,0], long-tailed distribution: [1,0,0,0], multimodal discrete: [0,0,1,0], unclassified: [0,0,0,1].

[0085] Through steps 401 to 404 above, a single and one-sided detection method is abandoned. Instead, a multi-dimensional and systematic identification framework consisting of various statistical tests and feature comparisons is constructed. By conducting specific tests and comprehensive judgments on key features such as the normality, long tail, and multimodality of the data, the true distribution patterns hidden behind the field data can be identified more accurately and robustly. This provides a more reliable and accurate basis for subsequent data synthesis processes (such as model selection and parameter configuration), which is a key prerequisite for improving the quality and fidelity of the final synthesized data.

[0086] Step 302: When the field data is a non-standard type field, calculate the empirical quantile vector of the field data.

[0087] Step 303: Calculate the correlation coefficient between each field of data and the other field of data one by one, and construct a correlation matrix between all field data based on all correlation coefficients.

[0088] Step 304: Based on the correlation matrix, determine the correlation distance between all field data, and perform hierarchical clustering on all field data based on the correlation distance to obtain multiple clustering levels and the clustering level index corresponding to each field data.

[0089] Step 305: Based on feature type encoding, clustering level index and / or empirical quantile vector, obtain the field feature vector corresponding to each field data.

[0090] Steps 302 to 305 are described in detail below.

[0091] In some embodiments, for non-standard data fields that cannot be categorized into any standard distribution type, this method employs a non-parametric approach to describe their data characteristics. The system calculates an empirical quantile vector for these fields. Here, the "empirical quantile vector" is an ordered set containing values ​​at multiple specific quantiles (e.g., 10%, 25%, 50%, 75%, 90%, etc.). This vector effectively outlines the overall profile, central tendency, and dispersion of the field's data distribution without pre-assuming it conforms to any specific mathematical distribution model, thus providing a robust and informative feature descriptor for such complex fields.

[0092] In one example, empirical quantile feature extraction for unclassified fields involves calculating empirical quantile vectors for fields that cannot be classified into standard distribution types. This serves as a distribution characteristic descriptor for the data in this field. Quantiles are calculated using linear interpolation. ,in where n is the sample size. This indicates a rounding down operation on the np parameter. These quantile features can effectively capture the distribution shape and variability of the data.

[0093] To comprehensively understand the relationships between all fields in the dataset, this application calculates the correlation coefficient between any two fields. Here, "correlation coefficient" is a statistical indicator that measures the degree of association between variables. To capture different types of relationships, the system can calculate multiple correlation coefficients simultaneously, such as the Pearson correlation coefficient (measuring linear relationships) and the Spearman rank correlation coefficient (measuring monotonic relationships). Based on all calculated correlation coefficients, the system constructs a correlation matrix for all fields, which comprehensively and systematically displays the pairwise dependencies between all fields in the dataset.

[0094] Based on the correlation matrix constructed in the previous step, the system first converts the correlation metric into a correlation distance, typically defined as 1 minus the absolute value of the correlation coefficient (i.e., distance = ...). This matrix identifies strong correlations and their corresponding short distances. Then, based on this correlation distance matrix, the system employs a hierarchical clustering algorithm (such as the Ward method) to perform aggregation analysis on all field data. This algorithm generates a tree-like dependency structure, naturally dividing the data into multiple clustering levels and avoiding the subjectivity of manual specification. Finally, each field data is assigned a corresponding clustering level index based on its position within this tree structure, which clarifies its order in subsequent generation processes.

[0095] In one example, the hierarchical mapping and generation order are represented as follows: hierarchical ordering, the depth of the clustering tree corresponds to the generation level, the root node is the first level (strongest correlation field), and the leaf node is the last level; generation order, the generation order is determined by tree traversal, ensuring that the parent level fields are generated before the child level fields. For example, the correlation matrix is ​​as follows, and the clustering result is: first level {transaction volume, credit score}, second level {number of transactions}.

[0096]

[0097] Next, this application enters the feature integration stage, assembling the discrete information extracted in all the preceding steps into a unified, standardized field feature vector. This field feature vector selectively includes previously generated feature information based on the type of field data: for standard type fields, it includes the obtained feature type encoding; for non-standard type fields, it includes the calculated empirical quantile vector. Furthermore, all field vectors contain a defined clustering level index and can be further integrated with other metadata such as sensitivity and data size, ultimately forming a feature carrier that comprehensively describes the field, as described below.

[0098] Reference Figure 5 Based on feature type encoding, clustering level index and / or empirical quantile vector, the field feature vector corresponding to each field data is obtained, including the following steps 501 to 503.

[0099] Step 501: Based on the field type corresponding to each field of data, obtain the corresponding sensitivity value.

[0100] Step 502: Obtain sample values ​​from multiple structured data sets.

[0101] Step 503: Based on feature type encoding, clustering level index, sensitivity value, sample value and / or empirical quantile vector, obtain the field feature vector corresponding to each field data.

[0102] Steps 501 to 503 are described in detail below.

[0103] In some embodiments, this method also requires assigning a business-level attribute to each field, i.e., obtaining its corresponding sensitivity value. Here, "field type" refers to the business meaning of the field, such as "transaction history" or "credit rating." The system assigns a preset sensitivity level, such as low, medium, or high, to the field type based on the information it carries and relevant business regulations. This level is then quantified into a specific sensitivity value, for example, encoded using {0, 1, 2}. This value reflects the degree of privacy protection required for the field data and is an important basis for subsequent operations such as differential privacy budget allocation.

[0104] Then, this method obtains a sample value that describes the size of the entire dataset. Here, "sample value" specifically refers to the total number of records or samples in multiple structured data sets, such as 500,000 samples. This value macroscopically reflects the volume and sufficiency of the training data. Since different data synthesis models (especially deep learning models and traditional statistical models) have different dependencies and adaptability to sample size, obtaining this sample value and using it as a feature helps in making more reasonable decisions during the subsequent model selection stage.

[0105] Finally, this method performs a final feature integration step, unifying the feature information analyzed and obtained in the preceding steps into a field feature vector corresponding to each field's data. This field feature vector is a structured multidimensional array, constructed based on the aforementioned technical features: it includes feature type encoding describing the distribution type, cluster hierarchy index representing data dependencies, sensitivity values, and scale feature values ​​obtained after processing sample values ​​(such as values ​​after logarithmic transformation). Furthermore, depending on whether the field is a standard or non-standard type, the vector may selectively include empirical quantile vectors. Through this step, all discrete information about a field is integrated into a standardized data structure, achieving a unified representation and efficient utilization of feature information.

[0106] In one example, all feature information is encoded into a unified feature vector for use by subsequent modules. The feature vector is designed with dimensions such that it is used for both intelligent model matching and as input features for the residual mapping network, achieving a unified representation and efficient utilization of feature information. The field feature information includes the following:

[0107]

[0108] Through steps 501 to 503 above, the inherent distribution characteristics, external association structure, business-level sensitivity attributes, and macro-scale information of the dataset of the field are systematically integrated, so that the final generated field feature vector constitutes a comprehensive and standardized "digital profile" of the field. This multi-dimensional and highly condensed feature representation provides high-quality and standardized input for subsequent advanced processing processes such as intelligent model matching, hierarchical condition generation, and privacy compliance management. It is the core technical prerequisite for realizing the adaptability, accuracy, and security of the entire data synthesis method.

[0109] Reference Figure 6 This is a schematic diagram illustrating the generation process of a field feature vector provided in an embodiment of this application. For example... Figure 6The diagram illustrates the complete workflow of the data preprocessing and feature analysis module in this application. The initial stage of the workflow focuses on handling missing values ​​in the original data, using a multivariate Gaussian Copula method for imputation. This process first fits the marginal distributions of each field to the complete data and estimates the correlation structure. Then, a Copula model is constructed, and missing values ​​are generated through conditional distribution sampling for imputation. To ensure the quality of the imputation, a verification step is included in the workflow to check whether the correlation matrix of the imputed data is consistent with the original matrix.

[0110] After data population, the process moves to a stage of in-depth feature analysis for each field. The system first determines the specific distribution type of each field by calculating statistical characteristics such as skewness and kurtosis, combined with methods like normality and multimodality tests. Based on the determination results, fields are categorized as normal, long-tailed, multimodal, or unclassified, and corresponding one-hot codes or empirical quantile vectors are generated as quantitative descriptions of their distribution characteristics. Next, the system calculates Pearson and Spearman correlation coefficient matrices and uses the Ward hierarchical clustering algorithm to analyze the dependencies between fields, thereby determining the hierarchical order of field generation. Finally, the process integrates all analyzed features—including distribution codes, quantiles, sensitivity, data size, and hierarchical indexes—into a unified feature vector, which serves as standardized input for subsequent processing modules, completing the entire preprocessing and feature analysis process.

[0111] Through steps 301 to 305 above, not only are the inherent distribution characteristics of individual fields identified and quantified in a refined manner, but also the hidden dependency structure and hierarchical relationship between fields are revealed profoundly through correlation analysis and hierarchical clustering. Then, these heterogeneous and multi-dimensional analysis results are systematically integrated into a standardized field feature vector. This process transforms the raw, difficult-to-use structured data into standardized input that is rich in information, has a clear structure, and is easy for machines to understand. This provides a solid and comprehensive data foundation for subsequent advanced functions such as intelligent model selection and on-demand condition generation, greatly improving the accuracy and effectiveness of the entire data processing flow.

[0112] Step 102: For each clustering level, based on the field feature vectors of all corresponding field data, select the target data synthesis model corresponding to the clustering level from multiple data synthesis models.

[0113] Step 102 is described in detail below.

[0114] In some embodiments, after feature extraction is completed, the system dynamically selects a target data synthesis model corresponding to the current level from a preset model library containing multiple candidate data synthesis models for each clustering level.

[0115] In one example, taking financial risk control data as an example, assume that the system identifies the following field dependency structure: First layer: fields {transaction history, credit score}, which have high linear correlation and marginal distributions of long tail and normal, respectively; Second layer: field {number of transactions}, which is weakly correlated with the fields in the first layer and marginally distributed as a multimodal discrete distribution.

[0116] The "data synthesis model" here refers to algorithmic models such as Gaussian_Copula, CTGAN, and TVAE that can generate artificial data. The selection is based on the field feature vector [v_dist, v_quant, v_sens, v_scale, v_layer], which the system uses to calculate a multi-dimensional fitness score. This scoring function comprehensively evaluates each candidate model's ability to handle the distribution characteristics, correlation structure, privacy requirements, and data scale of the current level's fields. Ultimately, the model with the highest score is determined as the target data synthesis model for that level, thus achieving intelligent and adaptive model selection, as described below.

[0117] Reference Figure 7 For each clustering level, based on the field feature vectors of all corresponding field data, the target data synthesis model corresponding to the clustering level is selected from multiple data synthesis models, including the following steps 701 to 703.

[0118] Step 701: Based on the field feature vectors of all field data at the clustering level, obtain the quantized weights corresponding to multiple feature types.

[0119] Step 701 will be described in detail below.

[0120] In some embodiments, a set of quantized weights is first extracted and generated based on the field feature vectors corresponding to all field data in the current clustering level to be processed. Here, "feature type" refers to different dimensions describing data characteristics, such as distribution characteristics, correlation characteristics, sensitivity characteristics, and scale characteristics. For each feature type, the system calculates a corresponding quantized weight from the field feature vectors. For example, the distribution quantized weight is obtained by calculating the Euclidean norm of the feature type encoding, or the maximum correlation coefficient within the cluster is extracted as the correlation quantized weight. These quantized weights concretize the characteristics of the data in various dimensions into numerical values, providing an objective basis for subsequent scoring calculations, as described below.

[0121] Reference Figure 8 The quantization weights include distribution quantization weights, correlation quantization weights, sensitivity quantization weights, and scale quantization weights. Based on the field feature vectors of all field data at the clustering level, the quantization weights corresponding to multiple feature types are obtained, including the following steps 801 to 804.

[0122] Step 801: Calculate the Euclidean norm for the feature type encoding of all field feature vectors in the clustering level to obtain the distribution quantization weights corresponding to the distribution features.

[0123] Step 802: Extract the largest correlation coefficient from all field feature vectors at the cluster level, and use it as the correlation quantification weight corresponding to the correlation feature.

[0124] Step 803: Extract sensitivity values ​​from all field feature vectors at the clustering level, and use them as the sensitivity quantification weights corresponding to the sensitivity features.

[0125] Step 804: Extract sample values ​​from all field feature vectors at the clustering level, and perform logarithmic processing on the sample values ​​to obtain the scale quantization weights corresponding to the scale features.

[0126] Steps 801 to 804 are described in detail below.

[0127] In some embodiments, the distribution features are first quantized to obtain their corresponding distribution quantization weights. The system extracts the feature type encoding from the feature vectors of all fields in the current clustering level. Here, the "feature type encoding" is typically a one-hot encoding vector representing a specific distribution type (such as normal, long-tailed, etc.). Next, the system calculates the Euclidean norm of this feature type encoding, which is the distribution quantization weight. The result of this calculation is a scalar value, which is used as the distribution quantization weight corresponding to the distribution feature. This weight reflects the type and complexity of the data distribution of the field at that level in numerical form.

[0128] Next, the correlation features are quantified to obtain the correlation quantification weight. The system analyzes the correlation information between fields implicit in or related to the feature vectors of all fields in the current clustering level. This information comes from the correlation matrix constructed in the previous steps. From this information, the system extracts the maximum absolute value of the correlation coefficient between any two fields within this level, which is the correlation quantification weight. The highest extracted correlation coefficient is directly used as the correlation quantification weight corresponding to the correlation feature, which concisely and effectively represents the most important dependency strength within the current field cluster.

[0129] Next, the sensitivity features will be quantized to obtain sensitivity quantization weights. The system will directly extract predefined sensitivity values ​​from the feature vectors of all fields at the current clustering level. Here, the "sensitivity value" is an encoded value representing the privacy sensitivity level of the field data (e.g., low / medium / high correspond to 0 / 1 / 2 respectively). This directly extracted sensitivity value, without further calculation, can be used as the sensitivity quantization weight corresponding to the sensitivity feature, i.e., the sensitivity quantization weight. This is used to subsequently evaluate the adaptability of the synthetic model in terms of privacy protection.

[0130] Simultaneously, the size feature is quantized to obtain the size quantization weight. The system first extracts sample values ​​describing the size of the entire dataset, i.e., the total number of records, from the field feature vector. Since the absolute value of the sample number can span multiple orders of magnitude, to ensure its more stable and effective role in the scoring model, the system performs logarithmic processing on this sample value, for example, calculating its base-10 logarithm. The result after logarithmic processing is the size quantization weight corresponding to the size feature, i.e., the size quantization weight. It reflects the size of the training data and is used to evaluate the performance of different models on this data scale.

[0131] In one example, for the subset {transaction volume, credit score}, v_dist = [1,0,0,0] (long tail) and [0,1,0,0] (normal). w_dist = 0.8; Extract from the correlation matrix w_corr = 0.75; v_sens = 1 (medium sensitivity) w_sens = 1; v_scale = 5.7 w_scale = 5.7; This gives the weight vector [0.8, 0.75, 1, 5.7].

[0132] Through steps 801 to 804 above, a clear and explicit set of rules is established to systematically extract and convert the rich and multi-dimensional information in the field feature vector into a set of concise and standardized quantitative weights. Each weight precisely corresponds to a core feature type, transforming abstract data characteristics (such as distribution complexity, correlation strength, privacy requirements, and data volume) into scalar values ​​that can be directly used for mathematical operations. This process greatly simplifies the input of the subsequent scoring function, making each dimension of model evaluation have an objective and quantifiable basis, thereby ensuring the automation, transparency, and reliability of the entire model selection process.

[0133] Step 702: Calculate the score for each data synthesis model based on the quantization weight and scoring function corresponding to each feature type, and obtain the type performance score of each data synthesis model corresponding to each feature type.

[0134] Step 703: Perform weighted processing on the performance scores of multiple types to obtain the performance score of each data synthesis model, and select the data synthesis model with the highest performance score from the multiple data synthesis models as the target data synthesis model corresponding to the clustering level.

[0135] Steps 701 to 703 are described in detail below.

[0136] In some embodiments, the system utilizes the quantized weights obtained in the previous step, combined with a series of preset scoring functions, to perform item-by-item scoring calculations for each data synthesis model in the model library. For each feature type, there is a dedicated scoring function (such as the distribution fitness function f_dist, the correlation handling capability function f_corr, the privacy protection function f_priv, and the sample size adaptation function f_scale). This scoring function receives the quantized weights corresponding to that feature type as input and outputs a score, which is the type performance score of the data synthesis model on that specific feature type. For example, the performance of a model in processing long-tailed distributed data will be given a specific score by the distribution fitness function based on the quantized weights. This step decomposes the overall capability of a model into performance across multiple different dimensions and quantifies them.

[0137] In one example, the specific implementation of each scoring function is shown below.

[0138] (1) Distribution fitness function : Find a predefined rating table based on the distribution type:

[0139] (2) Correlation processing capability function Piecewise function based on correlation strength:

[0140] (3) Privacy protection capability function Score based on sensitivity level:

[0141] (4) Sample size adaptation function Adaptability based on sample size:

[0142] Then, the final model decision-making process is executed. The system will weight the multiple performance scores obtained in the previous step, that is, use a set of preset weight coefficients (i.e., Determined through cross-validation of historical data, for example: (Importance of distribution adaptation) (Importance of correlation processing) (Importance of privacy protection) (Scale-adaptive importance) The scores of each model on different feature types are weighted and summed to calculate the final performance score of the data synthesis model, as shown in the following formula.

[0143]

[0144] This comprehensive performance score fully reflects the overall matching degree between the model and the data features of the current cluster level. Finally, the system compares the performance scores of all candidate data synthesis models and selects the model with the highest performance score as the unique target data synthesis model corresponding to that cluster level, which will then be used to perform subsequent data generation tasks.

[0145] In one example, for the field subset {transaction history, credit score}, with features [long tail + normal, high correlation 0.75, medium sensitivity 1, large sample 5.7], its Gaussian_Copula score is: f_dist = 0.65 (mean of long tail 0.4 + normal 0.9). f_priv = 0.5 (medium performance under medium sensitivity), f_scale = 0.82 (slightly decreases under large samples), therefore The corresponding CTGAN score is: f_dist = 0.75 (mean of long tail 0.8 + normal 0.7). f_priv = 0.4 (privacy protection not supported), f_scale = 0.87 (good performance with large samples), therefore Therefore, Gaussian_Copula was chosen as the generative model for this subset of data.

[0146] Through steps 701 to 703 above, a complete, objective, and data-driven dynamic model selection framework is established, replacing the traditional approach that relies on human experience or uses fixed models. By transforming abstract data features into specific quantitative weights and using multi-dimensional scoring functions for refined evaluation and weighted decision-making, it can adaptively and optimally match the most suitable generation tool from multiple candidate models based on the unique characteristics of each field cluster. This intelligent "tailored" strategy ensures that different parts of the dataset can be processed most effectively, thereby greatly improving the quality of each step in the hierarchical generation process. It is the core mechanism to ensure the high fidelity of the final synthesized data.

[0147] Step 103: Obtain the marginal distribution of the field type corresponding to each cluster level in multiple structured data. For each cluster level, use the corresponding target data synthesis model to process the input data and obtain the synthesized output data. The input data of the first layer is the corresponding marginal distribution, and the input data of other layers is the synthesized output data of the previous layer and the marginal distribution of the current layer.

[0148] Step 103 will be described in detail below.

[0149] In some embodiments, after a model is selected, hierarchical condition generation begins. First, the system needs to obtain the marginal distribution of each field type in the original structured data for each cluster level. Here, "marginal distribution" refers to the data distribution pattern of a single field, which has been identified or fitted during the feature extraction stage. The generation process strictly follows the order determined by the cluster level index. For the first layer, which serves as the root node, its input data is only the marginal distribution corresponding to that layer, and the target data synthesis model generates unconditionally based on this. For all other subsequent layers, the input data is a combination, including both the synthesized output data generated by the previous layer and the marginal distribution of the current layer itself. By using the output of the previous layer as the condition for the current layer, it ensures that the dependencies between fields are accurately transmitted and maintained during the generation process.

[0150] In one example, unconditional generation is performed for the root layer (i.e., the first layer): for the field set {transaction flow, credit score} with v_layer = 1, linked samples are generated using the Gaussian Copula model selected based on its feature vectors. The generation process preserves the marginal distribution characteristics of each field (determined by v_dist and v_quant).

[0151] For multi-level (i.e., from the second level onwards) condition generation: fields of each level are generated sequentially according to the hierarchical order. Each level depends only on the directly above level. That is, the field of the k-th level uses the data of the (k-1)-th level as the condition input. The corresponding condition generation model is adopted to ensure that the parent-child layer dependency relationship is satisfied. The generation quality and consistency constraints are verified layer by layer.

[0152] The following is a pseudocode example.

[0153]

[0154] In addition, sample synthesis and quality control are required. This involves merging the data generated from each layer into a joint sample and verifying whether the statistical properties described by the feature vectors are maintained. Verification includes: marginal distribution fidelity (verifying whether it conforms to the distribution type indicated by v_dist); correlation fidelity (verifying whether it maintains the correlation structure identified by cluster analysis); and sensitivity compliance (ensuring that the privacy protection requirements indicated by v_sens are met). Then, a comprehensive quality score is calculated. When the score is lower than a threshold, a backtracking mechanism is triggered to update the model selection strategy.

[0155] Reference Figure 9 This is a schematic diagram illustrating a process for selecting the optimal synthesis model and performing structured data synthesis, provided in an embodiment of this application. For example... Figure 9 The diagram illustrates the core process of model selection and hierarchical execution in this application. The process begins by receiving a unified feature vector V, output from the previous module, containing information such as field distribution type and hierarchical index. Based on this vector, the system first calculates a set of quantified weights, such as the distribution weight w_dist and the correlation weight w_corr, to quantify the key characteristics of the current data level. Subsequently, the system evaluates each candidate data synthesis model (such as Gaussian Copula, CTGAN, etc.) using a multi-dimensional fitness scoring function. This function comprehensively considers the model's ability to handle specific distributions, correlations, privacy requirements, etc., and calculates a total score. Finally, the model with the highest total score is selected as the optimal model for the current level and used to perform the generation task.

[0156] After selecting the optimal model, the process enters the condition generation phase, which is executed in the order of field hierarchy. For example... Figure 9 As shown, the root layer data is generated unconditionally, while subsequent layers are generated based on samples from the previous layer, ensuring that dependencies between fields are maintained. After generation layer by layer, the system performs consistency verification and merges the results from each layer into a complete synthetic sample. The key to the process lies in a quality control and feedback loop mechanism: the system performs a comprehensive quality score on the merged sample and determines whether it meets the standards. If the score does not reach a preset threshold, the process will backtrack, triggering an update to the model selection strategy and re-executing the generation process until the generated data quality meets the requirements, ultimately outputting high-quality synthetic data.

[0157] Step 104: Based on the synthetic output data corresponding to all clustering levels, obtain structured synthetic data.

[0158] Step 104 will be described in detail below.

[0159] In some embodiments, after all clustering levels have completed their respective data generation in sequence, the synthetic output data corresponding to all levels are merged and assembled to obtain complete structured synthetic data. Since each layer of data is generated while satisfying its own peripheral distribution characteristics and following its dependency relationship with the data of the layer above, integrating these layer-by-layer data fragments containing partial fields constitutes a synthetic data table that is highly consistent with the original data in overall statistical characteristics. This table not only reproduces the distribution of individual fields but also successfully replicates the complex, hierarchical relationship structure between fields.

[0160] However, since residuals are inevitable in the combined data, compensation adjustments are required, as described below.

[0161] Reference Figure 10 Based on the synthetic output data corresponding to all clustering levels, structured synthetic data is obtained, including the following steps 1001 to 1002.

[0162] Step 1001: Based on the synthetic output data corresponding to all clustering levels, obtain the initial structured synthetic data.

[0163] Step 1002: Input the initial structured composite data into the residual mapping network to obtain the residual adjustment value, and adjust the initial structured composite data based on the residual adjustment value to obtain structured composite data corresponding to multiple structured data.

[0164] Steps 1001 to 1002 are described in detail below.

[0165] In some embodiments, initial structured composite data D_joint is first obtained based on the composite output data generated from all previous clustering levels. In the preceding hierarchical generation phase, each clustering level has generated its own field data. This step integrates and joins these scattered composite output data from different levels to assemble a complete data table with all fields. This data table is the initial structured composite data; it macroscopically possesses the marginal distribution and hierarchical dependency characteristics of the original data, but may still have biases in some minor statistical indicators, and is therefore considered an intermediate product awaiting refinement.

[0166] To further improve data quality, the initial structured synthetic data obtained in the previous step is input into a pre-trained residual mapping network, f_res. This "residual mapping network" is a neural network specifically designed to learn the statistical bias between real and synthetic data. Upon receiving the initial structured synthetic data, this network predicts a residual adjustment value. This residual adjustment value represents the amount of correction needed to make the statistical characteristics of the synthetic data more closely approximate those of the real data. Finally, based on this residual adjustment value, the system performs fine-tuning on each data point in the initial structured synthetic data, thereby producing the final structured synthetic data that is more statistically consistent with multiple sets of structured data.

[0167] The following section will further describe how to train the residual mapping network.

[0168] Reference Figure 11 The training process of the residual mapping network includes the following steps 1101 to 1104.

[0169] Step 1101: Obtain multiple structured sample data.

[0170] Step 1102: Perform data synthesis on each structured sample data to obtain synthetic sample data, and calculate the statistical deviation characteristics between each synthetic sample data and the corresponding structured sample data.

[0171] Step 1103: Perform missing data imputation on each structured sample data to obtain imputed data. Input the imputed data, statistical bias features, and contextual information of the structured sample data into the initial residual mapping network for data processing to obtain the training residual adjustment value.

[0172] Step 1104: The initial residual mapping network is trained and updated multiple times based on the loss function, and the residual mapping network is obtained based on the initial residual mapping network after multiple training and updates. The loss function is obtained based on the difference between the training residual adjustment value and the preset correction increment.

[0173] Steps 1101 to 1104 are described in detail below.

[0174] In some embodiments, before training the residual mapping network, a dataset needs to be prepared for offline training. The first step is to acquire multiple structured sample data. Here, "structured sample data" refers to multiple batches or subsets of data extracted from real, raw data sources. For example, the system might randomly sample thousands of independent batches of real samples from raw financial risk control data. These real data samples will serve as the "standard answer" or benchmark in the subsequent training process, used to calculate bias and define learning objectives.

[0175] Then, the system generates a corresponding synthetic copy for each acquired structured sample data and quantifies the differences between the two. Specifically, the system uses the data synthesis process of this invention to generate a corresponding synthetic sample data (D_joint) for each real structured sample data (D_true).

[0176] In one example, during the offline training phase, the system randomly samples real sample batches D_true from "user batches" (100 user records per batch) of real financial risk control data. The corresponding initial sample batch D_joint is generated using the hierarchical generation process (Copula+CTGAN) from the previous module. To build sufficient training data, the system repeats this pairing generation process thousands of times, independently sampling real data and generating synthetic data each time, ensuring that each pair (D_true, D_joint) has different statistical characteristics and bias patterns. A batch statistical vector is calculated for each field j: Where S_j is used to comprehensively describe the statistical characteristics of a single field j in the data batch, and μ_j represents the mean (average value) of field j in the data of that batch. This represents the variance of field j in this batch of data. These represent the quartiles of field j, which together describe the distribution and central tendency of the data. skew_j represents the skewness of field j, measuring the asymmetry of the data distribution.

[0177] Subsequently, the system calculates the statistical bias characteristics between the two samples by comparing their performance on various statistical indicators one by one. The "statistical bias characteristics" here are a feature vector that includes residuals of single-field statistics (such as the difference between the mean and variance), residuals of inter-field correlation, and distribution distances (such as Wasserstein distance). It accurately describes the degree of "distortion" of the synthetic sample relative to the real sample.

[0178] In one example, the residual objective is constructed by comparing the statistical characteristics of the real batch and the synthetic batch, including: single-field residuals. Correlation residuals: Distribution distance: These residual features capture the degree of deviation in single-field statistical bias and inter-field association. Here, S_true,j is the statistical vector of field j calculated using the real data D_true, S_joint,j is the statistical vector of field j calculated using the synthetic data D_joint, Corr_joint is the correlation matrix calculated based on the synthetic data batch, and Wasserstein() represents the Wasserstein distance, a metric for measuring the distance between two probability distributions.

[0179] Furthermore, the residual mapping network in this application is designed as follows: the residual mapping network f_res is trained with the following input: Where: x_initial is the initial field value in D_joint, The batch statistical error corresponds to the field, and context_features represents the context information of the sample within the batch. By learning from a large amount of paired training data, the network masters the correction rules of "how to adjust the values ​​of each field to make them closer to the features of the real data when the generated data exhibits a certain statistical bias". The target value of each training sample is the difference between the real sample and the synthetic sample (D_true[i] - D_joint[i]), representing the ideal correction increment.

[0180] The network architecture is: multilayer perceptron [256, 128, 64], LeakyReLU activation, outputting the correction increment of each field.

[0181] The loss function is: (L2 regularization).

[0182] Next, training instances are constructed and fed into the network for forward propagation. First, the system imputes missing data for each structured sample to ensure data integrity, resulting in imputed data. Then, the system combines the imputed data, the statistical bias features calculated in the previous step, and necessary contextual information (such as the sample's context within the batch) into an input vector. This vector is fed into an unoptimized initial residual mapping network for data processing. The network outputs a predicted adjustment value based on its current internal parameters; this value is the training residual adjustment value.

[0183] Then, this method performs network parameter optimization and updates. The system uses a predefined loss function to evaluate the difference between the network output training residual adjustment value and an ideal preset correction increment. Here, the "preset correction increment" is the training target value, i.e., the actual difference between the true sample value and the synthetic sample value (D_true[i] - D_joint[i]), representing the perfect correction direction and magnitude. The loss function is then calculated based on the difference between this predicted value and the target value (e.g., using mean squared error loss), i.e. (L2 regularization). Using the backpropagation algorithm, the system iteratively trains and updates the initial residual mapping network based on this loss function until the network converges. After multiple training and updates, the network becomes a fully trained residual mapping network capable of accurately predicting correction values.

[0184] Steps 1101 to 1104 above do not teach the network how to generate data from scratch, but rather focus on a more specific and simpler task: learning and correcting the systematic biases of the master generative model. By constructing “real-synthetic” paired samples and explicitly using statistical bias features as network input, the network is provided with direct and clear contextual information about “errors.” This goal-oriented supervised learning makes the training process efficient and stable. The resulting residual mapping network can act as a precise “calibrator” to make subtle adjustments to the synthetic data, thereby effectively improving the statistical fidelity of the final output data.

[0185] In one example, for each newly generated sample batch D_new, a statistical feature vector S_new is calculated and compared with the target feature S_target. This is done using a statistical significance threshold (e.g., ...). Quickly locate the fields that need correction.

[0186] The correction process includes: conditional residual prediction, which involves inputting statistical bias information into a pre-trained f_res network to predict field-level correction increments.

[0187] The following is a correction example. When the batch mean of the transaction field is 15% higher, the network's learning-based association pattern prediction includes: a negative adjustment of -300 yuan for the transaction field; and a positive adjustment of +20 points for the credit score, thereby achieving multi-field coordinated correction and trend consistency restoration.

[0188] The iterative optimization control in this application employs an iterative correction strategy, including: a convergence condition where the statistical deviation change rate between two consecutive iterations is <1% or the maximum number of iterations is reached (5); and a correction strength consisting of a dynamic adjustment coefficient. This is to avoid data distortion caused by excessive adjustments.

[0189] In addition, to enhance the stability of the correction, this application also uses multiple residual mapping networks with different network structures for residual mapping. It is understood that all residual mapping networks are obtained from the above training process.

[0190] The residual mapping networks employ different random initialization seeds and different network structures: [128,64], [256,128], and [512,256,128]. They also use different training subsets obtained through bootstrap sampling to ensure that each sub-network learns different data patterns and correction strategies.

[0191] Reference Figure 12 There are multiple residual mapping networks. The initial structured synthetic data is input into the residual mapping network to obtain the residual adjustment value, including the following steps 1201 to 1203.

[0192] Step 1201: Input the initial structured synthesis data into each residual mapping network one by one to obtain the initial residual adjustment value corresponding to each residual mapping network.

[0193] Step 1202: Based on the residual error and stability index of each residual mapping network, calculate the network comprehensive performance of each residual mapping network, and obtain the network weight of each residual mapping network based on the network comprehensive performance.

[0194] Step 1203: Based on the network weights of multiple residual mapping networks, the multiple initial residual adjustment values ​​are weighted and fused to obtain the residual adjustment values.

[0195] Steps 1201 to 1203 are described in detail below.

[0196] In some embodiments, this method employs an ensemble learning strategy, inputting the generated initial structured synthetic data one by one and in parallel into each pre-trained residual mapping network. Here, "multiple residual mapping networks" refers to a set of models consisting of multiple sub-networks with different structures and parameters, which may employ different random initializations, network structures, or training subsets to ensure diversity. Upon receiving the input, each independent residual mapping network independently outputs what it considers the optimal correction value based on its own learning patterns; this value is the initial residual adjustment value corresponding to each residual mapping network.

[0197] Then, to objectively evaluate and quantify the contribution of each sub-model, this method calculates the overall network performance of each residual mapping network and determines its network weights based on this performance. This calculation process is based on two core evaluation dimensions: residual error. and stability indicators The "residual error" measures the deviation between the adjusted value predicted by the network and the true target value, i.e., accuracy; while the "stability index" measures the consistency and volatility of the adjusted value output by the network. Here, `prediction_i` is the predicted correction value made by the i-th residual mapping network to the validation set samples, `mean()` represents the average of the squared errors of all samples on the validation set, `std(corrections_i)` is the standard deviation of all predicted correction values ​​of the i-th residual mapping network on the validation set, used to measure the magnitude of output volatility, and `std / mean` is the coefficient of variation, used to measure the relative dispersion of the data. Subtracting this value from 1 makes the index positive; that is, the higher the `Stability_i`, the more stable the network output.

[0198] The system will then combine these two metrics using a preset formula to create a comprehensive network performance score. Then, through normalization, the score is converted into the network weights corresponding to each residual mapping network, i.e. .

[0199] The system then performs a fusion decision to generate a unified, optimal adjustment value. Based on the network weights calculated for the multiple residual mapping networks in the previous step, the system performs a weighted fusion of the multiple initial residual adjustment values ​​generated by these networks. Specifically, this is a weighted averaging process, where each initial residual adjustment value is multiplied by its corresponding network weight, and then all weighted results are summed. Through this weighted fusion method, the opinions of the superior and more stable networks are given greater importance. The final calculated fusion result, known as the residual adjustment value, will be used to perform final correction on the initial structured synthetic data.

[0200] In one example, the sub-model errors for the three rating fields are [0.12, 0.18, 0.15], corresponding to weights of [0.45, 0.30, 0.25]. The final output at this point... Then, quality monitoring continues: when the variance of the fusion result exceeds the threshold, model retraining or weight reallocation is triggered to ensure consistency and reliability of the correction.

[0201] Through steps 1201 to 1203 above, by employing a strategy of integrating multiple residual mapping networks instead of relying on a single model, the robustness and stability of the entire correction process are greatly enhanced. Utilizing diverse sub-networks allows for learning the correction patterns of data bias from different perspectives, effectively avoiding overfitting or prediction bias that might occur with a single model. Furthermore, by introducing a dynamic weight allocation mechanism based on residual error and stability indices, the "opinions" of multiple models can be intelligently and adaptively integrated, allowing the model that contributes more and performs more reliably to dominate the final decision. This ensures that the final residual adjustment value is more accurate and reliable than the prediction of any single model, thereby further improving the final quality of the synthetic data.

[0202] Reference Figure 13 This is a schematic diagram illustrating the process of residual adjustment for initial structured synthetic data provided in an embodiment of this application. Figure 13 The diagram illustrates the detailed workflow of the joint fine-tuning and fusion module in this application. Its core is the use of a residual mapping network to refine the initially synthesized samples. The process begins by receiving the joint sample D_joint from the previous module and proceeding to the offline network construction stage. In this stage, the system generates paired batches of real samples (D_true) and synthesized samples (D_joint), calculates the residual characteristics of both in terms of single fields, correlation, and distribution, and uses these to train the residual mapping network f_res, enabling it to learn the mapping pattern from statistical bias to field correction increments. Subsequently, in the online correction stage, this network is used to detect the statistical bias of newly generated samples in real time, predict and perform coordinated correction of multiple fields, and iteratively optimize the control until the results converge.

[0203] To further improve the stability and accuracy of the calibration, a multi-model weighted fusion mechanism is introduced. The system trains multiple diverse f_res sub-networks with different structures, data, or initializations. During application, the system evaluates the performance (residual error) and stability of each sub-network on the validation set in real time and calculates dynamic fusion weights accordingly. Finally, the calibration results output by all sub-networks are weighted and fused according to these weights to obtain a more robust and accurate final calibration result, and outputs optimized high-quality samples.

[0204] Through steps 1001 to 1002 above, a post-processing correction stage is introduced. This stage does not stop at the initial generated results, but uses a dedicated residual mapping network to refine the initial structured synthetic data a second time. This design can effectively identify and compensate for systematic biases or minor errors that may exist in the basic generative model. By learning and applying this residual adjustment, the fidelity of the final output structured synthetic data in terms of multi-dimensional statistical indicators such as mean, variance, and quantiles can be significantly improved, making it highly consistent with the real data not only in terms of macro structure but also in terms of subtle features, thereby comprehensively improving the quality and usability of the synthetic data.

[0205] In addition, to ensure the security and privacy of the structured data synthesis process, appropriate protective measures are required for the data processing flow.

[0206] Reference Figure 14 The structured data synthesis method also includes the following steps 1401 to 1403.

[0207] Step 1401: Based on the sensitivity values ​​in the field feature vector corresponding to each field data, obtain the encryption budget allocation weight corresponding to each field data.

[0208] Step 1402: Based on the feature type encoding in the feature vector corresponding to each field data, obtain the noise addition mechanism corresponding to each field data.

[0209] Step 1403: Encrypt the corresponding field data based on the encryption budget allocation weight and noise addition mechanism.

[0210] Steps 1401 to 1403 are described in detail below.

[0211] In some embodiments, resources are first configured for privacy protection mechanisms during the data synthesis process. The system determines the encryption budget allocation weight for each field based on the sensitivity value contained in its feature vector. Here, the sensitivity value v_sens is a predefined metric that quantifies the sensitivity of the field information. Based on this value and, if necessary, considering business importance, the system proportionally allocates a total privacy budget. This allocation is applied to each field. The result is a weight assigned to the encryption budget of each field (i.e., its independent privacy budget). This parameter determines the strength of privacy protection on that field and is a key parameter for achieving differential privacy protection.

[0212] In one example, the system uses the sensitivity information v_sens from the feature vector constructed in the first stage, combined with business importance weights, to allocate the budget. (Set the total budget.) The principle of utility maximization is adopted: For example, sensitivity encoding based on feature vectors: transaction data v_sens=2 (high sensitivity), credit score v_sens=1 (medium sensitivity), transaction count v_sens=0 (low sensitivity), combined with business importance weights, to calculate budget allocation. .

[0213] In addition, dynamic budget adjustments are performed. The system monitors the budget consumption rate of each field in real time. When the budget of a certain field is consumed too quickly, the budget is transferred from the low-frequency field to the high-frequency field to ensure that the differential privacy combination property is not violated.

[0214] Then, this method selects the most suitable privacy protection technique based on the inherent characteristics of the field data. The system determines the noise addition mechanism for each field based on the feature type encoding (v_dist) contained in its feature vector. Here, the "feature type encoding" clarifies the data distribution type of the field, such as whether it is continuous (e.g., normal distribution, long-tailed distribution) or discrete. Based on this encoding, the system adaptively selects a matching noise addition mechanism, for example, choosing a Laplace mechanism for continuous fields and a discrete Laplace mechanism for discrete fields. This step ensures that the applied privacy protection technique is mathematically rigorous and applicable to the data type.

[0215] In one example, for continuous field noise: a Laplace mechanism is used for both normally distributed and long-tailed distributed fields. Sensitivity of transaction field Yuan, budget Noise scale λ=2500, generated For discrete field noise: For discrete distribution fields, a discrete Laplace mechanism is used, with a probability mass function... ,in Transaction count field Add when the original 5 times The probability of noise is approximately 0.22.

[0216] Then, based on the encryption budget weights assigned to each field and the selected noise-adding mechanism determined in the previous step, the corresponding field data is encrypted. Here, "encryption" does not refer to traditional data encryption, but rather to injecting precisely calibrated random noise into the data to meet differential privacy requirements. The selected noise-adding mechanism utilizes the encryption budget weights (… To define the scale of noise (e.g., the scale of Laplace noise). Then, the data is generated and added to the actual field data values. After this processing step, the output data retains usable statistical characteristics while also achieving mathematically provable privacy protection.

[0217] In addition, noise post-processing and constraint maintenance are required to ensure that the data after adding noise meets the business logic constraints. This includes: numerical truncation, flow_final = max(0, min(flow_max, flow_original +noise), and consistency adjustment. When noise disrupts the consistency between fields, it is fine-tuned according to priority rules without adding additional privacy leaks.

[0218] This application also provides a series of security protection measures, as shown below.

[0219] Real-time compliance monitoring includes: budget tracking, establishing a real-time budget tracking matrix B(t) = [ε_consumed, ε_remaining, ε_projected], using a sliding window to monitor consumption rate, and supporting fine-grained audit traceability; combinatorial computation, calculating budget consumption based on differential privacy combinatorial properties, parallel query ε_parallel = max(ε_i), and sequential query ε_sequential = Σε_i; a three-level early warning mechanism: green status (remaining budget > 30%), yellow warning (remaining budget 10%-30%) (reducing release frequency), and red alert (remaining budget < 10%) (suspending non-critical releases).

[0220] Seamless integration with the generation process, including: end-to-end privacy protection, with privacy protection mechanisms integrated into the entire data generation pipeline, including the following points.

[0221] (1) Model training stage: Inject training privacy protection; (2) Layered generation stage: Add generation privacy noise; (3) Residual correction stage: Consider the impact of noise on correction accuracy; (4) Final release stage: Implement release privacy protection; (5) Use privacy accounting technology to accurately track accumulated privacy loss and ensure end-to-end privacy protection.

[0222] Intelligent release decision-making includes: assessing the utility and quality of released data. When the Q value is below the threshold, an optimal balance is sought between privacy protection and data quality.

[0223] In one example: Privacy protection for the corrected sample D_corrected includes: transaction history: 5000 yuan + Laplace noise 180 yuan = 5180 yuan; credit score: 700 points + Laplace noise (-15 points) = 685 points; number of transactions: 5 times + discrete noise 1 time = 6 times. The cumulative privacy loss ε_total = max(0.4, 0.35, 0.25) = 0.4 (parallel combination), meeting the budget constraint. After publication, the budget status is updated, detailed audit logs are recorded, completing the entire process from raw data to privacy-protected published data.

[0224] Reference Figure 15 This is a schematic diagram illustrating a security and privacy protection process in structured data processing provided in an embodiment of this application. For example... Figure 15 The diagram illustrates the complete workflow of the privacy compliance management module in this application, designed to provide end-to-end differential privacy protection for synthetic data. The process begins with receiving corrected sample data D_corrected. First, the system performs differential privacy budget allocation, assigning a reasonable privacy budget to each field based on its sensitivity and business weight. It also supports dynamic adjustments to prioritize frequently used fields. Subsequently, the process enters the adaptive noise mechanism stage, where the system selects an appropriate noise model, such as Laplace or Discrete Laplace, based on the field's distribution type (continuous or discrete). After adding noise, necessary post-processing is performed, such as numerical truncation and consistency adjustments, to ensure data validity.

[0225] To ensure compliance throughout the process, a real-time compliance monitoring system is included. This system monitors privacy budget consumption through budget tracking and combinatorial calculations, and features a three-tiered warning mechanism (green, yellow, and red) to prevent budget overruns. A key feature of this module is its seamless integration with the overall generation process, injecting privacy protection measures into every stage of training, generation, correction, and release. The system utilizes privacy accounting techniques to accurately track accumulated privacy losses and makes intelligent release decisions before final output, balancing the strength of privacy protection with data quality. Through this rigorous process, the final output is data that satisfies both statistical characteristics and stringent privacy guarantees.

[0226] Through steps 1401 to 1403 above, intelligent and refined management of privacy resources is achieved by allocating budgets based on field sensitivity values ​​and selecting noise-adding mechanisms based on data distribution types. This "locally tailored" strategy can maximize the preservation of the utility and quality of synthetic data while meeting strict privacy protection requirements, achieving a better balance between privacy and usability, thus making the generated synthetic data more secure, compliant, and usable.

[0227] Reference Figure 16 This is a schematic diagram of the framework of a structured data synthesis system provided in an embodiment of this application. Figure 16 The diagram illustrates the four core modules of the structured data synthesis system and their interrelationships. The process begins with the "Data Preprocessing and Feature Analysis Module," which cleans and analyzes the raw data and outputs hierarchical features containing hierarchical dependencies. These features are then fed into the "Model Selection and Execution Module," which dynamically selects the optimal generative model based on the input features. The selected model is then executed by the "Joint Fine-tuning and Fusion Module," generating preliminary synthesized data. Finally, this synthesized data flows to the "Privacy Compliance Management Module," where differential privacy protection is implemented, producing the final secure and compliant data for output.

[0228] The advanced nature of this system architecture lies not only in its modular, sequential process, but also in the dynamic feedback and collaboration mechanisms between modules. For example... Figure 16 As indicated by the middle arrow, the system comprises two key feedback loops. First, the "Joint Fine-tuning and Fusion Module" feeds back the residual information of the generated data to the "Model Selection and Execution Module" to dynamically adjust and optimize the model selection strategy, forming a closed loop that ensures data quality. Second, the "Privacy Compliance Management Module" monitors the consumption of the privacy budget in real time and feeds it back to the "Model Selection and Execution Module," enabling it to balance the cost and efficiency of privacy protection when selecting models. This design makes the entire framework not a simple linear pipeline, but an intelligent collaborative system that adaptively balances data utility and privacy security.

[0229] This application also provides a structured data synthesis apparatus that can implement the above-described structured data synthesis method, referring to... Figure 17 The device 1700 includes: The feature extraction module 1710 is used to extract features from the field data of the acquired multiple structured data, and obtain the field feature vector corresponding to each field data. The field feature vector includes a clustering level index used to indicate the clustering level of the field data. The synthetic model selection module 1720 is used to select the target data synthetic model corresponding to the clustering level from multiple data synthetic models based on the field feature vector of all corresponding field data for each clustering level. The hierarchical synthesis processing module 1730 is used to obtain the marginal distribution of the field type corresponding to each cluster level in multiple structured data. For each cluster level, the input data is processed using the corresponding target data synthesis model to obtain the synthesized output data. The input data of the first layer is the corresponding marginal distribution, and the input data of other layers is the synthesized output data of the previous layer and the marginal distribution of the current layer. The overall synthesis module 1740 is used to obtain structured synthetic data based on the synthetic output data corresponding to all clustering levels.

[0230] In some embodiments, the feature extraction module 1710 is further configured to: Obtain multiple initial structured data sets, identify missing structured data sets with missing data from the multiple initial structured data sets, and determine the marginal distribution of each field type based on the complete observation data from the multiple initial structured data sets; By combining the complete observation data and marginal distribution corresponding to each field type, the cumulative probability value of each complete observation data is obtained; Based on the cumulative probability values, the correlation matrix between field types is obtained, and the correlation matrix is ​​used as the connection parameter to construct a multivariate Gaussian connection model. The observed data in the missing structured data is input into the marginal distribution corresponding to the field type to obtain the cumulative probability value of the observation. The cumulative probability value is then input into the multivariate Gaussian connection model to obtain the missing cumulative probability value corresponding to the missing data in the missing structured data. Input the cumulative probability value of missing data into the inverse distribution of the marginal distribution corresponding to the field type of the missing data to obtain the filling data, and update the missing structured data with the filling data to obtain the structured data.

[0231] In some embodiments, the feature extraction module 1710 is further configured to: When the field data is a standard type field, calculate the statistical characteristics of each field data one by one. Based on the statistical characteristics and statistical test methods, determine the distribution type of each field data, and based on the distribution type, obtain the corresponding feature type code of the field data. When the field data is a non-standard type field, calculate the empirical quantile vector of the field data; Calculate the correlation coefficient between each field of data and the data of other fields one by one, and construct a correlation matrix between all fields of data based on all correlation coefficients; Based on the correlation matrix, the correlation distance between all field data is determined, and hierarchical clustering is performed on all field data based on the correlation distance to obtain multiple clustering levels and the clustering level index corresponding to each field data. Based on feature type encoding, clustering hierarchy index and / or empirical quantile vector, the field feature vector corresponding to each field data is obtained.

[0232] In some embodiments, the feature extraction module 1710 is further configured to: The field data is tested and calculated based on the normality test method to obtain the normality distribution confidence value. Based on the numerical relationship between the normality distribution confidence value and the preset distribution threshold, the normality distribution type result is obtained. The field skewness is compared with the preset skewness threshold, and the field kurtosis is compared with the preset kurtosis threshold to obtain the long-tail distribution type result; The field data is tested and calculated using statistical testing methods to obtain the confidence value of the multimodal distribution. Based on the numerical relationship between the confidence value of the multimodal distribution and the preset distribution threshold, the result of the multimodal distribution type is obtained. Based on the results of the normal distribution type, the long-tailed distribution type, and the multimodal distribution type, the distribution type corresponding to the field data is obtained.

[0233] In some embodiments, the feature extraction module 1710 is further configured to: Based on the field type corresponding to each field of data, obtain the corresponding sensitivity value; Obtain sample values ​​from multiple structured data sets; Based on feature type encoding, clustering level index, sensitivity value, sample value and / or empirical quantile vector, the field feature vector corresponding to each field data is obtained.

[0234] In some embodiments, the synthesis model selection module 1720 is further configured to: Based on the field feature vectors of all field data at the clustering level, we obtain the quantized weights corresponding to multiple feature types. Each data synthesis model is scored based on the quantized weights and scoring functions corresponding to each feature type, resulting in a type performance score for each data synthesis model corresponding to each feature type. The performance scores of multiple types are weighted to obtain the performance score of each data synthesis model, and the data synthesis model with the highest performance score is selected as the target data synthesis model corresponding to the clustering level.

[0235] In some embodiments, the synthesis model selection module 1720 is further configured to: Euclidean norm calculation is performed on the feature type encoding of all field feature vectors at the clustering level to obtain the distribution quantization weights corresponding to the distribution features; Extract the largest correlation coefficient from all field feature vectors at the cluster level and use it as the correlation quantification weight for the corresponding correlation feature. The sensitivity values ​​are extracted from the feature vectors of all fields at the clustering level and used as the sensitivity quantification weights corresponding to the sensitivity features. Extract sample values ​​from all field feature vectors at the cluster level, and perform logarithmic processing on the sample values ​​to obtain the scale quantization weights corresponding to the scale features.

[0236] In some embodiments, the overall synthesis module 1740 is further configured to: Based on the synthetic output data corresponding to all clustering levels, the initial structured synthetic data is obtained.

[0237] The initial structured composite data is input into the residual mapping network to obtain residual adjustment values. Based on the residual adjustment values, the initial structured composite data is adjusted to obtain structured composite data corresponding to multiple structured data.

[0238] In some embodiments, the overall synthesis module 1740 is further configured to: Acquire multiple structured sample data; Each structured sample data is synthesized to obtain synthetic sample data, and the statistical deviation characteristics between each synthetic sample data and the corresponding structured sample data are calculated. Missing data imputation is performed on each structured sample data to obtain imputed data. The imputed data, statistical bias features, and contextual information of the structured sample data are then input into the initial residual mapping network for data processing to obtain the training residual adjustment value. The initial residual mapping network is trained and updated multiple times based on the loss function, and the residual mapping network is obtained based on the initial residual mapping network after multiple training and updates. The loss function is obtained based on the difference between the training residual adjustment value and the preset correction increment.

[0239] In some embodiments, the overall synthesis module 1740 is further configured to: The initial structured synthesis data is input into each residual mapping network one by one to obtain the initial residual adjustment value corresponding to each residual mapping network; Based on the residual error and stability index of each residual mapping network, the overall network performance of each residual mapping network is calculated, and the network weight of each residual mapping network is obtained based on the overall network performance. The adjusted residual values ​​are obtained by weighting and fusing multiple initial residual adjustment values ​​based on the network weights of multiple residual mapping networks.

[0240] In some embodiments, the overall synthesis module 1740 is further configured to: Based on the sensitivity values ​​in the field feature vector corresponding to each field data, the encryption budget allocation weight corresponding to each field data is obtained; Based on the feature type encoding in the feature vector corresponding to each field data, the noise addition mechanism corresponding to each field data is obtained; The corresponding field data is encrypted based on the weighting of the encrypted budget and the noise-adding mechanism.

[0241] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, the specific implementation of the structured data synthesis device is basically the same as the specific implementation of the structured data synthesis method described above, and will not be repeated here.

[0242] This application also provides an electronic device, including: At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the structured data synthesis method described above in this application. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0243] Please see Figure 18 , Figure 18 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1802 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1802 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1802 and is called and executed by the processor 1801 using the structured data synthesis method of the embodiments of this application. The input / output interface 1803 is used to implement information input and output; The communication interface 1804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1805 transmits information between various components of the device (e.g., processor 1801, memory 1802, input / output interface 1803, and communication interface 1804); The processor 1801, memory 1802, input / output interface 1803 and communication interface 1804 are connected to each other within the device via bus 1805.

[0244] This application also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described structured data synthesis method.

[0245] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0246] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0247] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0248] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0249] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0250] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0251] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0252] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0253] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0254] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0255] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0256] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method of structured data synthesis, characterized by, The method comprises: feature extraction is performed on field data in the obtained plurality of structured data to obtain a field feature vector corresponding to each of the field data, the field feature vector comprising a clustering level index for indicating a clustering level of the field data; for each of the clustering levels, a target data synthesis model corresponding to the clustering level is selected from a plurality of data synthesis models based on the field feature vectors of all the field data corresponding to the clustering level; for each of the clustering levels, an input data is processed by using the target data synthesis model corresponding to the clustering level to obtain a synthesized output data, the input data of a first layer being the marginal distribution corresponding to the clustering level, and the input data of other layers being the synthesized output data of a previous layer and the marginal distribution of a current layer; based on the synthesized output data corresponding to all the clustering levels, structured synthesized data is obtained.

2. The structured data synthesis method of claim 1, wherein, The acquisition process of the plurality of structured data comprises: a plurality of initial structured data are acquired, missing structured data in which missing data exist are determined from the plurality of initial structured data, and the marginal distribution of each of the field types is determined based on complete observation data in the plurality of initial structured data; the complete observation data corresponding to each of the field types and the marginal distribution are used to obtain a cumulative probability value of each of the complete observation data; based on the cumulative probability value, a correlation matrix between the field types is obtained, and the correlation matrix is used as a connection parameter to construct a multivariate Gaussian connection model; observation data in the missing structured data are input into the marginal distribution corresponding to the field type corresponding to the observation data to obtain an observation cumulative probability value, and the cumulative probability value is input into the multivariate Gaussian connection model to obtain a missing cumulative probability value corresponding to the missing data in the missing structured data; the missing cumulative probability value is input into an inverse distribution of the marginal distribution corresponding to the field type corresponding to the missing data to obtain filling data, and the filling data is used to update the missing structured data to obtain the structured data.

3. The structured data synthesis method of claim 1, wherein, The feature extraction performed on the field data in the obtained plurality of structured data to obtain a field feature vector corresponding to each of the field data comprises: when the field data is a standard type field, statistical features of each of the field data are calculated one by one, a distribution type of each of the field data is determined based on the statistical features and a statistical test method, and a feature type code of the corresponding field data is obtained based on the distribution type; when the field data is a non-standard type field, an empirical quantile vector of the field data is calculated; correlation coefficients between each of the field data and other field data are calculated one by one, and a correlation matrix between all the field data is constructed based on all the correlation coefficients; Based on the correlation matrix, the correlation distance between all the field data is determined, and hierarchical clustering is performed on all the field data based on the correlation distance to obtain multiple clustering levels and the clustering level index corresponding to each field data. Based on the feature type encoding, the clustering level index, and / or the empirical quantile vector, the field feature vector corresponding to each field data is obtained.

4. The structured data synthesis method of claim 3, wherein, The statistical features include field skewness and field kurtosis; the statistical tests include normality tests and multimodal tests; and determining the distribution type of each field data based on the statistical features and statistical tests includes: The field data is tested and calculated based on the normality test method to obtain a normal distribution confidence value. Based on the numerical relationship between the normal distribution confidence value and the preset distribution threshold, the normal distribution type result is obtained. The field skewness is compared with a preset skewness threshold, and the field kurtosis is compared with a preset kurtosis threshold to obtain the long-tail distribution type result; The field data is tested and calculated using statistical testing methods to obtain a multi-peak distribution confidence value. Based on the numerical relationship between the multi-peak distribution confidence value and a preset distribution threshold, the multi-peak distribution type result is obtained. Based on the normal distribution type result, the long-tailed distribution type result, and the multimodal distribution type result, the distribution type corresponding to the field data is obtained.

5. The method of claim 3, wherein, The step of obtaining the field feature vector corresponding to each field data based on the feature type encoding, the clustering level index, and / or the empirical quantile vector includes: Based on the field type corresponding to each of the field data, obtain the corresponding sensitivity value; Obtain sample values ​​from multiple pieces of the structured data; Based on the feature type encoding, the clustering level index, the sensitivity value, the sample value, and / or the empirical quantile vector, the field feature vector corresponding to each field data is obtained.

6. The structured data synthesis method of claim 1, wherein, For each clustering level, based on the field feature vectors of all the corresponding field data, the selection of the target data synthesis model corresponding to the clustering level from multiple data synthesis models includes: Based on the field feature vectors of all the field data in the clustering level, quantized weights corresponding to multiple feature types are obtained; Based on the quantization weight and scoring function corresponding to each feature type, a score is calculated for each data synthesis model to obtain a type performance score for each data synthesis model corresponding to each feature type. The performance scores of the multiple types are weighted to obtain the performance score of each data synthesis model, and the data synthesis model with the highest performance score is selected from the multiple data synthesis models as the target data synthesis model corresponding to the clustering level.

7. The method of claim 6, wherein, The feature types include distribution features, correlation features, sensitivity features, and scale features. The quantization weights include distribution quantization weights, correlation quantization weights, sensitivity quantization weights, and scale quantization weights. The quantization weights corresponding to multiple feature types are obtained based on the field feature vectors of all field data at the clustering level, including: Euclidean norm calculation is performed on the feature type encoding of all field feature vectors at the clustering level to obtain the distribution quantization weights corresponding to the distribution features; Extract the largest correlation coefficient from all the field feature vectors of the clustering level, and use it as the correlation quantification weight corresponding to the correlation feature; The sensitivity values ​​are extracted from all the field feature vectors of the clustering level and used as the sensitivity quantization weights corresponding to the sensitivity features. Extract sample values ​​from all the field feature vectors of the clustering level, and perform logarithmic processing on the sample values ​​to obtain the scale quantization weights corresponding to the scale features.

8. The structured data synthesis method of claim 1, wherein, The process of obtaining structured synthetic data based on the synthetic output data corresponding to all the clustering levels includes: Based on the synthetic output data corresponding to all the clustering levels, the initial structured synthetic data is obtained. The initial structured synthetic data is input into a residual mapping network to obtain residual adjustment values. Based on the residual adjustment values, the initial structured synthetic data is adjusted to obtain multiple pieces of structured synthetic data corresponding to the structured data.

9. The method of claim 8, wherein, The training process of the residual mapping network includes: Acquire multiple structured sample data; Each of the structured sample data is synthesized to obtain synthesized sample data, and the statistical deviation characteristics between each of the synthesized sample data and the corresponding structured sample data are calculated. Missing data imputation is performed on each of the structured sample data to obtain sample imputation data. The sample imputation data, the statistical bias features, and the context information of the structured sample data are then input into the initial residual mapping network for data processing to obtain the training residual adjustment value. The initial residual mapping network is trained and updated multiple times based on the loss function, and the residual mapping network is obtained based on the initial residual mapping network after multiple training and updates. The loss function is obtained based on the difference between the training residual adjustment value and the preset correction increment.

10. The method of claim 8, wherein, The residual mapping network comprises multiple networks. The step of inputting the initial structured synthesis data into the residual mapping network to obtain residual adjustment values ​​includes: The initial structured synthesis data is input into each of the residual mapping networks one by one to obtain the initial residual adjustment value corresponding to each residual mapping network; Based on the residual error and stability index of each residual mapping network, the overall network performance of each residual mapping network is calculated, and the network weight of each residual mapping network is obtained based on the overall network performance. The initial residual adjustment values ​​are weighted and fused based on the network weights of the multiple residual mapping networks to obtain the residual adjustment values.

11. The method of claim 1, wherein, The method further includes: Based on the sensitivity value in the feature vector of each field data, the encryption budget allocation weight corresponding to each field data is obtained; Based on the feature type encoding in the feature vector corresponding to each field data, the noise addition mechanism corresponding to each field data is obtained; The corresponding field data is encrypted based on the encryption budget allocation weight and the noise-adding mechanism.

12. A structured data synthesis apparatus, characterized by The device includes: The feature extraction module is used to extract features from the field data in the acquired multiple structured data to obtain a field feature vector corresponding to each field data. The field feature vector includes a clustering level index for indicating the clustering level of the field data. The synthesis model selection module is used to select the target data synthesis model corresponding to the clustering level from multiple data synthesis models based on the field feature vector of all the corresponding field data for each clustering level. The hierarchical synthesis processing module is used to obtain the marginal distribution of the field type corresponding to each clustering level in multiple pieces of structured data. For each clustering level, the input data is processed using the corresponding target data synthesis model to obtain synthesized output data. The input data of the first layer is the corresponding marginal distribution, and the input data of other layers is the synthesized output data of the previous layer and the marginal distribution of the current layer. The overall synthesis module is used to obtain structured synthesized data based on the synthesis output data corresponding to all the clustering levels.

13. An electronic device, comprising: It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the structured data synthesis method according to any one of claims 1 to 11.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the structured data synthesis method according to any one of claims 1 to 11.

Images