A highly efficient generative anonymization method and system for big data

Abstract

This invention relates to the field of data processing, providing a highly efficient generative anonymization method and system for big data. The method includes: preprocessing and feature modeling the original large dataset to obtain a feature vector set; constructing privacy risk indicators and task utility indicators based on the configuration parameters of the target sharing scenario, respectively, for the privacy leakage degree of the generated data and the performance of downstream tasks, resulting in a privacy utility quantification index system; inputting the feature vector set and the privacy utility quantification index system into a pre-built model, training the pre-built model through a multi-objective optimization loss function to obtain a generative anonymization model; inputting the scenario configuration vector into the generative anonymization model to generate a target anonymized dataset; evaluating the target anonymized dataset according to the privacy utility quantification index system; and outputting the evaluated target anonymized dataset to a data application system. This invention balances privacy security and data utility, improving scenario adaptability.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a highly efficient generative anonymization method and system for big data. Background Technology

[0002] With the rapid development of cloud computing, the Internet of Things (IoT), and artificial intelligence (AI) technologies, industries such as healthcare, finance, telecommunications, and the internet have accumulated massive amounts of user behavior and business data. Modeling and mining this data can support applications such as risk control, disease prediction, and precision marketing. However, this data contains a large amount of personal privacy and sensitive attributes. As countries and regions continuously introduce stricter data security and privacy protection regulations, the conflict between data sharing and privacy protection is becoming increasingly prominent. How to fully unleash the potential value of big data while meeting privacy compliance requirements has become a crucial issue that urgently needs to be addressed.

[0003] Existing anonymization methods mainly include removing direct identifiers, masking or generalizing certain fields, and desensitization techniques based on classic models such as k-anonymity and l-diversity. In recent years, synthetic data methods based on generative models such as generative adversarial networks and variational autoencoders have also emerged. However, these methods generally lack a quantifiable and controllable privacy-utility joint optimization mechanism. When anonymizing data, attributes are often strongly generalized or suppressed to meet privacy constraints, resulting in a large deviation between the statistical distribution of the generated data and the original data. This leads to a significant decrease in the modeling performance of downstream tasks and a serious lack of data utility. Summary of the Invention

[0004] This invention provides an efficient generative anonymization method and system for big data, in order to overcome the shortcomings of existing technologies.

[0005] This invention provides an efficient generative anonymization method for big data, comprising: S1. Preprocess and model the features of the original large dataset to obtain a feature vector set; S2. Based on the configuration parameters of the target sharing scenario, construct privacy risk indicators and task utility indicators for the degree of privacy leakage of generated data and the performance of downstream tasks respectively, and obtain a privacy utility quantitative indicator system. S3. Input the feature vector set and the privacy utility quantification index system into the pre-built model, and train the pre-built model through multi-objective optimization loss function to obtain a generative anonymization model; S4. Input the scenario configuration vector into the generative anonymization model to generate the target anonymization dataset. Evaluate the target anonymization dataset according to the privacy utility quantification index system. Output the target anonymization dataset that has passed the evaluation and publish it to the data application system.

[0006] According to the efficient generative anonymization method for big data provided by the present invention, step S1 further includes: S11. Extract the original large dataset from multiple business data sources, and divide the fields of each record in the original large dataset into identifier fields, quasi-identifier fields, sensitive fields and business fields to obtain a field-classified record set; S12. The missing values ​​in the field classification record set are processed by statistical imputation, and outliers are marked and corrected by threshold rules to obtain a cleaned record set. S13. Normalize, encode, and perform periodic feature transformation on the numerical field, category field, and time field in the cleaned record set, and concatenate the processing results of multiple fields into a feature vector of a unified dimension to obtain a feature vector set.

[0007] According to the efficient generative anonymization method for big data provided by the present invention, step S2 further includes: S21. Determine the set of attack types and sensitive fields that need to be protected, and form privacy configuration parameters; S22. Based on the privacy configuration parameters, for each of the multiple attack proxy models, during the training phase, the samples in the feature vector set are periodically input into the attack proxy model, the attack success rate of each type of attack proxy model is statistically analyzed, and multiple risk values ​​are obtained. S23. Weight and sum multiple risk values ​​according to preset weight coefficients to obtain the overall privacy risk index and form a privacy risk index model. S24. Determine the downstream task set and the evaluation index and task weight corresponding to the task. Take real data and samples in the feature vector set as inputs respectively, and independently perform training and inference on the preset task models of multiple downstream tasks to obtain the performance benchmark value of multiple tasks on real data and the task performance value on samples in the feature vector set. S25. The ratios of the task performance values ​​of multiple tasks to their corresponding performance benchmark values ​​are weighted and summed according to the task weights to obtain the overall task utility index, thus forming a task utility index model. S26. The privacy risk indicator model and the task utility indicator model together constitute the privacy utility quantitative indicator system.

[0008] According to the efficient generative anonymization method for big data provided by the present invention, the multiple attack proxy models in step S22 include: re-identification attack proxy model, member inference attack proxy model and attribute inference attack proxy model.

[0009] According to the efficient generative anonymization method for big data provided by the present invention, step S3 further includes: S31. Construct a pre-built model; the pre-built model consists of a generation sub-network and an evaluation sub-network. The generation sub-network receives random latent vectors and scene configuration vectors, and outputs anonymous candidate samples with the same dimension as the feature vector set after multi-layer nonlinear mapping. The evaluation sub-network receives the anonymous candidate samples and real samples, and calculates intermediate indicators of privacy risk and intermediate indicators of task loss. S32. Based on the intermediate privacy risk index and the intermediate task loss index, calculate the total loss according to the multi-objective optimization loss function, and update the parameters of the generative subnetwork and the evaluation subnetwork through the backpropagation algorithm to continuously optimize the generative anonymization model. When the overall privacy risk index and the overall task utility index on the validation set both reach the preset standard, save the converged generative anonymization model.

[0010] According to the present invention, a highly efficient generative anonymization method for big data is provided. In step S31, the generative sub-network adopts one or more combined structures of generative adversarial networks, variational autoencoders, or diffusion models. The output layer of the generative sub-network uses corresponding activation functions according to the original types of multiple fields to restore the output vector to an anonymous record that matches the field classification record set format in the feature vector set.

[0011] According to the efficient generative anonymization method for big data provided by the present invention, step S4 further includes: S41. Assign a scenario number to each shared scenario, record the corresponding privacy threshold, task weight and compliance requirements, form a scenario configuration file, and package the parameters in the scenario configuration file into a scenario configuration vector. S42. Input the scene configuration vector and the random latent vector sampled from the latent space into the generative subnetwork of the generative anonymization model, and output the anonymized data records in batches to obtain the target anonymized dataset. S43. Evaluate the target anonymized dataset, output a qualified version of the target anonymized dataset, and publish it to the data application system.

[0012] According to the present invention, a highly efficient generative anonymization method for big data is provided, wherein step S43 specifically includes: S431. Evaluate the target anonymized dataset, specifically including: performing re-identification, member inference, and attribute inference tests on the target anonymized dataset under a preset attack model; comparing the overall privacy risk index with the current scenario privacy threshold to obtain a first comparison difference; calculating the overall task utility index on the target anonymized dataset and comparing it with the real data performance baseline to obtain a second comparison difference. S432. When both the first comparison difference and the second comparison difference are less than or equal to the corresponding preset difference, the target anonymized dataset is marked as a qualified version and published to the data application system. When either the first comparison difference or the second comparison difference is greater than or equal to the corresponding preset difference, the weight coefficient of the loss term and the task weight are adjusted according to the deviation information to continuously optimize the generative anonymization model.

[0013] According to the efficient generative anonymization method for big data provided by the present invention, in step S43, when publishing a qualified version of the target anonymized dataset, the model version number, parameter configuration snapshot and evaluation results used in this release are synchronously recorded in the scenario configuration file as an auditable basis.

[0014] This invention also provides an efficient generative anonymization system for big data, comprising: Modeling module: Used to preprocess and model features of the original large dataset to obtain a set of feature vectors; The building module is used to construct privacy risk indicators and task utility indicators based on the configuration parameters of the target sharing scenario, respectively, for the degree of privacy leakage of the generated data and the performance of downstream tasks, so as to obtain a privacy utility quantitative indicator system. Training module: used to input the feature vector set and the privacy utility quantification index system into the pre-built model, and train the pre-built model through multi-objective optimization loss function to obtain a generative anonymization model; Evaluation module: This module is used to input the scenario configuration vector into the generative anonymization model, generate the target anonymization dataset, evaluate the target anonymization dataset according to the privacy utility quantification index system, and output the target anonymization dataset that passes the evaluation and publish it to the data application system.

[0015] This invention first divides the fields of the original large dataset into identifier fields, quasi-identifier fields, sensitive fields, and general business fields, and performs unified feature modeling. This ensures that subsequent privacy risk quantification and task utility evaluation are based on a clearly structured and dimensionally unified feature vector set, thus avoiding the metric calculation bias caused by mixed field types. Second, this invention constructs attack proxy models for re-identification attacks, member inference attacks, and attribute inference attacks, and forms an overall privacy risk index through weighted summation. Simultaneously, it combines the performance benchmark ratio of downstream tasks on real data and generated data to construct an overall task utility index. This allows for a calculable and comparable quantitative expression of both privacy protection effectiveness and data usability, completely changing the existing methods' reliance on static rules and inability to dynamically weight... The invention addresses the limitations of balancing privacy and utility. Secondly, because the generative and evaluation subnetworks are jointly updated during training through multi-objective optimization of the loss function, and the weight coefficients of each loss term can be dynamically adjusted according to privacy thresholds and task performance requirements, the anonymized data generated by this invention is statistically highly similar to the original data, effectively preserving the modeling performance of downstream tasks. Furthermore, by using scenario configuration files to independently manage privacy thresholds, task weights, and compliance requirements for different industries and partners, the same convergent generative anonymization model can flexibly generate multiple versions of target anonymized datasets using scenario configuration vectors as conditional inputs. Each release synchronously records the model version number, parameter configuration snapshot, and evaluation results, providing a traceable and auditable complete basis for subsequent privacy compliance reviews. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A schematic diagram of a highly efficient generative anonymization method using big data provided by this invention; Figure 2 This invention provides a schematic diagram of a highly efficient generative anonymization system architecture for big data. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, embodiments of this invention, and should not be construed as limiting the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. In the description of this invention, it should be understood that the terminology used is for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0019] The embodiments of the present invention are described below with reference to the figures.

[0020] like Figure 1 As shown, this invention provides an efficient generative anonymization method for big data, comprising: S1. Preprocess and model the features of the original large dataset to obtain a set of feature vectors.

[0021] Step S1 further includes: S11. Extract the original large dataset from multiple business data sources, and divide the fields of each record in the original large dataset into identifier fields, quasi-identifier fields, sensitive fields, and business fields to obtain a field-classified record set.

[0022] In step S11, the present invention extracts raw large datasets from one or more business data sources such as business databases, log systems, or data warehouses. Each record in the raw large dataset contains multiple fields. Based on data governance standards and business meaning, the present invention divides the fields of each record into four categories: identifier fields, which can directly point to a specific individual's identity, such as name and ID number; quasi-identifier fields, which are insufficient to identify an individual when used alone, but pose a risk of re-identification when combined, such as age, postal code, and occupation; sensitive fields, which involve personal privacy and require special protection, such as disease diagnosis and income level; and general business fields, which are ordinary business attribute fields with low relevance to privacy protection. After the above division, the present invention finally obtains a field-classified record set.

[0023] S12. The missing values ​​in the field classification record set are processed by statistical imputation, and outliers are marked and corrected by threshold rules to obtain a cleaned record set.

[0024] In step S12, the present invention performs data cleaning on the field-classified record set. Specifically, for missing values, the present invention uses statistical imputation, that is, it fills in the missing value positions according to the statistical distribution of the field in the full set of records (such as mean, mode or median) to make the records complete and usable; for outliers, the value range of each field is constrained by threshold rules, and values ​​exceeding the preset upper and lower boundaries are marked as outliers, and they are corrected or removed according to preset rules to obtain a cleaned record set.

[0025] S13. Normalize, encode, and perform periodic feature transformation on the numerical field, category field, and time field in the cleaned record set, and concatenate the processing results of multiple fields into a feature vector of a unified dimension to obtain a feature vector set.

[0026] In step S13, after obtaining the cleaned record set in step S12, this invention performs feature transformations for different field types. For numerical fields, normalization or standardization is performed to map values ​​of different dimensions and ranges to a unified numerical interval, eliminating the interference of dimension differences on subsequent model training. For category fields, encoding is performed to convert discrete category labels into numerical vector representations, allowing them to directly participate in matrix operations of the neural network. For time fields, periodic feature transformation is performed to decompose the original timestamp into feature components that reflect periodic patterns, such as converting hourly information into sine and cosine components to capture the periodic structure of time. After completing the feature transformations for each field, this invention concatenates the processing results of all fields in the same record into a feature vector of a unified dimension in a fixed order, repeating the above operations on the entire cleaned record set to finally obtain a feature vector set.

[0027] S2. Based on the configuration parameters of the target sharing scenario, construct privacy risk indicators and task utility indicators for the degree of privacy leakage of generated data and the performance of downstream tasks, respectively, to obtain a privacy utility quantitative indicator system.

[0028] Step S2 further includes: S21. Determine the set of attack types and sensitive fields that need to be protected, and form privacy configuration parameters.

[0029] In step S21, after the feature vector set is constructed, the data provider and security management personnel jointly determine the set of attack types and sensitive fields that need to be protected in the current shared scenario, forming privacy configuration parameters. The set of attack types clarifies the types of privacy attacks that need to be defended in this scenario, while the set of sensitive fields specifies the range of fields that need to be protected. Together, they form the basis for the subsequent attack proxy model selection and evaluation process configuration.

[0030] S22. Based on the privacy configuration parameters, for each of the multiple attack proxy models, during the training phase, samples from the feature vector set are periodically input into the attack proxy model to calculate the attack success rate of each type of attack proxy model and obtain multiple risk values. The multiple attack proxy models in step S22 include: re-identification attack proxy model, member inference attack proxy model, and attribute inference attack proxy model. S23. The multiple risk values ​​are weighted and summed according to preset weight coefficients to obtain the overall privacy risk index, forming a privacy risk index model.

[0031] In step S22, the present invention selects corresponding attack proxy models for three types of privacy attacks based on privacy configuration parameters. The re-identification attack proxy model simulates an attacker's behavior of matching anonymous records with original records using quasi-identifier field combinations. Its inputs are anonymous data samples and original data samples, and its output is the probability of a successful match, i.e., the re-identification risk value. The membership inference attack proxy model simulates an attacker's behavior of determining whether a specific record has participated in the model training process. Its inputs are the target record and the output distribution information of the generative anonymization model, and its output is the confidence that the record belongs to the training set, i.e., the membership inference risk value. The attribute inference attack proxy model simulates an attacker's behavior of inferring the values ​​of sensitive fields given the known values ​​of some fields. Its inputs are the record features after removing sensitive fields, and its output is the prediction accuracy for the values ​​of sensitive fields, i.e., the attribute inference risk value. During the training phase, the present invention periodically inputs samples from the feature vector set into the above three types of attack proxy models, calculates the attack success rate of each model, and obtains three independent risk values: the re-identification risk value, the membership inference risk value, and the attribute inference risk value.

[0032] Furthermore, after obtaining the three types of risk values, the present invention sums them up according to preset weight coefficients to obtain an overall privacy risk index, forming a privacy risk index model. The preset weight coefficients are independently set by security managers according to the threat level of each type of attack in the current scenario. The higher the weight, the more important it is to suppress this type of attack in this scenario.

[0033] S24. Determine the downstream task set and the corresponding evaluation index and task weight. Using real data and samples from the feature vector set as inputs, independently perform training and inference on the preset task models of multiple downstream tasks to obtain the performance benchmark values ​​of multiple tasks on real data and the task performance values ​​on samples from the feature vector set. S25. Sum the ratios of the task performance values ​​of multiple tasks to their corresponding performance benchmark values ​​according to the task weights to obtain the overall task utility index and form a task utility index model.

[0034] In step S24, after completing the privacy risk indicator modeling, the present invention further determines the downstream task set and the evaluation indicators and task weights corresponding to each task. The downstream task set covers specific modeling tasks such as risk prediction, behavior classification, indicator regression, and customer segmentation. Each task is configured with an independent evaluation indicator. For example, the classification task uses accuracy or F1 score, the regression task uses mean squared error, and the clustering task uses silhouette coefficient.

[0035] Subsequently, this invention uses real data and samples from the feature vector set as inputs, and independently performs training and inference on the preset task models of each downstream task to obtain the performance benchmark value of each task on real data and the task performance value on the feature vector set samples. The ratio of the task performance value to the performance benchmark value reflects the degree to which the generated data retains the performance of the real data on the task. This invention weights and sums these ratios for each task according to the task weights to obtain the overall task utility index, forming a task utility index model.

[0036] S26. The privacy risk indicator model and the task utility indicator model together constitute the privacy utility quantitative indicator system.

[0037] In step S26, the privacy risk indicator model and the task utility indicator model are combined to form a privacy utility quantitative indicator system, which serves as a unified quantitative basis for subsequent generative anonymization model training and generation evaluation.

[0038] S3. Input the feature vector set and the privacy utility quantification index system into the pre-built model, and train the pre-built model through multi-objective optimization loss function to obtain a generative anonymization model.

[0039] Step S3 further includes: S31. Construct a pre-built model; the pre-built model consists of a generation sub-network and an evaluation sub-network. The generation sub-network receives random latent vectors and scene configuration vectors, and outputs anonymous candidate samples with the same dimension as the feature vector set after multi-layer nonlinear mapping. The evaluation sub-network receives the anonymous candidate samples and real samples, and calculates intermediate indicators of privacy risk and task loss.

[0040] In step S31, the generating subnetwork adopts one or more combined structures of generative adversarial network, variational autoencoder or diffusion model; the output layer of the generating subnetwork uses the corresponding activation function according to the original type of multiple fields to restore the output vector to an anonymous record that matches the field classification record set form in the feature vector set.

[0041] After completing the construction of the privacy utility quantification index system, this invention constructs a pre-built model based on the feature vector set and the privacy utility quantification index system. The pre-built model consists of two parts: a generation sub-network and an evaluation sub-network.

[0042] The generator subnetwork receives random latent vectors and scene configuration vectors. The random latent vectors are sampled from the latent space and are a set of random numerical vectors following a standard normal distribution, representing the latent semantic encoding of the anonymized samples to be generated. The scene configuration vectors are vectors formed by numerically packaging configuration parameters such as the privacy threshold and task weights of the current shared scene. The generator subnetwork concatenates these two types of vectors as input, performs multiple layers of nonlinear mapping, and outputs anonymized candidate samples with the same dimension as the feature vector set.

[0043] The specific structure of the generative subnetwork employs one or more combinations of Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), or diffusion models. GANs train the generator and discriminator adversarially to make the distribution of generated samples approximate the real data distribution. VAEs map real samples to probability distribution parameters in the latent space through an encoder, and the decoder then samples from this distribution to reconstruct the samples. Diffusion models train a denoising network by progressively adding noise to real samples, gradually denoising from pure noise during the generation phase to obtain generated samples. The output layer of the generative subnetwork uses the corresponding activation function according to the original type of each field in the field classification record set. For numerical fields, a linear activation function is used to directly output continuous values. For categorical fields, a Softmax activation function is used to output the category probability distribution, and the category corresponding to the highest probability is selected. For time fields, an inverse transform (the opposite of the periodic feature transformation) is used to restore the output vector to a time representation, thus restoring the output vector to an anonymous record matching the format of the field classification record set.

[0044] Subsequently, the evaluation subnetwork receives anonymous candidate samples output by the generation subnetwork and real samples extracted from the feature vector set, and calculates intermediate privacy risk and task loss metrics respectively. The privacy risk intermediate metric is calculated as follows: anonymous candidate samples are input into the re-identification attack proxy model, the membership inference attack proxy model, and the attribute inference attack proxy model respectively; the attack success rate of each model on the current batch of anonymous candidate samples is calculated; and the results are weighted and summed according to preset weight coefficients to obtain the privacy risk intermediate metric for the current training batch. The task loss intermediate metric is calculated as follows: anonymous candidate samples are input into the preset task models of each downstream task for inference; the inference results are compared with the performance benchmark values ​​on real samples; the loss values ​​of each task are calculated; and the results are weighted and summed according to task weights to obtain the task loss intermediate metric for the current training batch.

[0045] S32. Based on the intermediate privacy risk index and the intermediate task loss index, calculate the total loss according to the multi-objective optimization loss function, and update the parameters of the generative subnetwork and the evaluation subnetwork through the backpropagation algorithm to continuously optimize the generative anonymization model. When the overall privacy risk index and the overall task utility index on the validation set both reach the preset standard, save the converged generative anonymization model.

[0046] In step S32, this invention substitutes the intermediate privacy risk index and the intermediate task loss index into the multi-objective optimization loss function to calculate the total loss value, which includes a privacy loss term, a task loss term, and a regularization loss term. The regularization loss term is used to constrain the diversity and stability of the output samples of the generator sub-network, preventing pattern collapse in the generated samples. This invention uses a backpropagation algorithm to propagate the gradient of the total loss value layer by layer from the evaluation sub-network back to the generator sub-network, updating all trainable parameters in both the generator and evaluation sub-networks according to the gradient direction. After each training round, this invention calculates the overall privacy risk index and the overall task utility index on the validation set and compares them with preset standards. When both indices reach the preset standards, training stops and the model parameters at this point are saved, resulting in a converged generative anonymization model.

[0047] S4. Input the scenario configuration vector into the generative anonymization model to generate the target anonymization dataset. Evaluate the target anonymization dataset according to the privacy utility quantification index system. Output the target anonymization dataset that has passed the evaluation and publish it to the data application system.

[0048] Step S4 further includes: S41. Assign a scene number to each shared scene, record the corresponding privacy threshold, task weight and compliance requirements, form a scene configuration file, and package the parameters in the scene configuration file into a scene configuration vector.

[0049] In step S41, after obtaining the converged generative anonymization model, the present invention enters the generation and publication stage of the target anonymized dataset. Specifically, the scene management module assigns a unique scene number to each shared scene and records information such as the privacy threshold, task weight, industry type, partner category, and compliance requirements corresponding to the scene, forming a scene configuration file. Subsequently, all parameters in the scene configuration file are concatenated into a numerical vector in a fixed order to obtain the scene configuration vector.

[0050] S42. Input the scene configuration vector and the random latent vector sampled from the latent space into the generative subnetwork of the generative anonymization model, and output the anonymous data records in batches to obtain the target anonymized dataset.

[0051] In step S42, during the generation phase, the present invention inputs the scene configuration vector and random latent vectors sampled from the latent space into the generative subnetwork of the converged generative anonymization model. The generative subnetwork performs multi-layer nonlinear mapping on the two types of vectors and outputs anonymized data records in batches. The above sampling and output process is repeated until the cumulative number of anonymized data records generated reaches the scale requirement of the current scene, thus obtaining the target anonymized dataset.

[0052] S43. Evaluate the target anonymized dataset, output a qualified version of the target anonymized dataset, and publish it to the data application system.

[0053] Specifically, step S43 includes: S431. Evaluate the target anonymized dataset, specifically including: performing re-identification, member inference, and attribute inference tests on the target anonymized dataset under a preset attack model; comparing the overall privacy risk index with the current scenario privacy threshold to obtain a first comparison difference; calculating the overall task utility index on the target anonymized dataset and comparing it with the real data performance baseline to obtain a second comparison difference.

[0054] In step S43, the present invention aims to perform an evaluation on the target anonymized dataset. In the privacy risk assessment stage, the target anonymized dataset is input into preset re-identification attack proxy models, member inference attack proxy models, and attribute inference attack proxy models, respectively. The success rates of the three types of attacks are statistically analyzed and weighted by preset weight coefficients to calculate the overall privacy risk index. The difference between this index value and the current scenario privacy threshold is used to obtain the first comparison difference value. In the task utility assessment stage, the target anonymized dataset is input into preset task models for each downstream task to perform training and inference. The task performance value of each task is calculated, and the ratio of each task performance value to the corresponding performance benchmark value is weighted and summed by task weights to obtain the overall task utility index. The difference between this index value and the real data performance baseline is used to obtain the second comparison difference value.

[0055] S432. When both the first comparison difference and the second comparison difference are less than or equal to the corresponding preset difference, the target anonymized dataset is marked as a qualified version and published to the data application system; when either the first comparison difference or the second comparison difference is greater than or equal to the corresponding preset difference, the loss term weight coefficient and task weight are adjusted according to the deviation information to continuously optimize the generative anonymization model; in step S43, when publishing the qualified version of the target anonymized dataset, the model version number, parameter configuration snapshot and evaluation results used in this publication are synchronously recorded in the scenario configuration file as an auditable basis.

[0056] In step S432, after obtaining the first comparison difference and the second comparison difference, the present invention compares the two differences with their corresponding preset differences. When both the first comparison difference and the second comparison difference are less than or equal to their corresponding preset differences, the target anonymized dataset is marked as a qualified version, exported to the downstream data application system, and the model version number, parameter configuration snapshot, and evaluation results used in this release are synchronously recorded in the scenario configuration file as an auditable basis for subsequent privacy compliance review. When any difference is greater than its corresponding preset difference, the weight coefficients of the loss term and the task weights in the multi-objective optimization loss function are adjusted according to the deviation information, and the converged generative anonymization model is retrained and optimized. Then, the generation and evaluation process is re-executed until the target anonymized dataset meets all the requirements of the current scenario.

[0057] like Figure 2 As shown, the present invention also provides an efficient generative anonymization system for big data, comprising: Modeling module 100: Used to preprocess and model features of the original large dataset to obtain a feature vector set; Module 200: Based on the configuration parameters of the target sharing scenario, it is used to construct privacy risk indicators and task utility indicators for the degree of privacy leakage of generated data and the performance of downstream tasks, respectively, and obtain a privacy utility quantitative indicator system. Training module 300: used to input the feature vector set and the privacy utility quantification index system into the pre-built model, and train the pre-built model through multi-objective optimization loss function to obtain a generative anonymization model; Evaluation module 400: is used to input the scenario configuration vector into the generative anonymization model, generate the target anonymization dataset, evaluate the target anonymization dataset according to the privacy utility quantification index system, and output the target anonymization dataset that passes the evaluation and publish it to the data application system.

[0058] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the efficient generative anonymization method for big data described in various embodiments or some parts of embodiments.

[0060] This invention achieves generative anonymization of big data through multi-objective optimization. Based on preprocessing and feature modeling, it constructs a quantitative indicator system that balances privacy risks and task utility, effectively balancing data privacy protection and downstream task usability. Secondly, it employs generative model training, which can dynamically adapt to different sharing needs according to scenario configurations, significantly reducing the risk of privacy leakage while preserving key features and statistical distributions of the original data, thus improving the utility of anonymized data in practical applications. The anonymized dataset generated by this invention is of stable quality and reliable for evaluation, meeting data security compliance requirements and supporting the efficient operation of various data application systems. It is suitable for large-scale data sharing and open scenarios, possessing strong practicality and versatility.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A highly efficient generative anonymization method for big data, characterized in that, include: S1. Preprocess and model the features of the original large dataset to obtain a feature vector set; S2. Based on the configuration parameters of the target sharing scenario, construct privacy risk indicators and task utility indicators for the degree of privacy leakage of generated data and the performance of downstream tasks respectively, and obtain a privacy utility quantitative indicator system. S3. Input the feature vector set and the privacy utility quantification index system into the pre-built model, and train the pre-built model through multi-objective optimization loss function to obtain a generative anonymization model; S4. Input the scenario configuration vector into the generative anonymization model to generate the target anonymization dataset. Evaluate the target anonymization dataset according to the privacy utility quantification index system. Output the target anonymization dataset that has passed the evaluation and publish it to the data application system.

2. The efficient generative anonymization method for big data according to claim 1, characterized in that, Step S1 further includes: S11. Extract the original large dataset from multiple business data sources, and divide the fields of each record in the original large dataset into identifier fields, quasi-identifier fields, sensitive fields and business fields to obtain a field-classified record set; S12. The missing values ​​in the field classification record set are processed by statistical imputation, and outliers are marked and corrected by threshold rules to obtain a cleaned record set. S13. Normalize, encode, and perform periodic feature transformation on the numerical field, category field, and time field in the cleaned record set, and concatenate the processing results of multiple fields into a feature vector of a unified dimension to obtain a feature vector set.

3. The efficient generative anonymization method for big data according to claim 1, characterized in that, Step S2 further includes: S21. Determine the set of attack types and sensitive fields that need to be protected, and form privacy configuration parameters; S22. Based on the privacy configuration parameters, for each of the multiple attack proxy models, during the training phase, the samples in the feature vector set are periodically input into the attack proxy model, the attack success rate of each type of attack proxy model is statistically analyzed, and multiple risk values ​​are obtained. S23. Weight and sum multiple risk values ​​according to preset weight coefficients to obtain the overall privacy risk index and form a privacy risk index model. S24. Determine the downstream task set and the evaluation index and task weight corresponding to the task. Take real data and samples in the feature vector set as inputs respectively, and independently perform training and inference on the preset task models of multiple downstream tasks to obtain the performance benchmark value of multiple tasks on real data and the task performance value on samples in the feature vector set. S25. The ratios of the task performance values ​​of multiple tasks to their corresponding performance benchmark values ​​are weighted and summed according to the task weights to obtain the overall task utility index, thus forming a task utility index model. S26. The privacy risk indicator model and the task utility indicator model together constitute the privacy utility quantitative indicator system.

4. The efficient generative anonymization method for big data according to claim 3, characterized in that, The multiple attack proxy models in step S22 include: re-identification attack proxy model, membership inference attack proxy model, and attribute inference attack proxy model.

5. The efficient generative anonymization method for big data according to claim 1, characterized in that, Step S3 further includes: S31. Construct a pre-built model; the pre-built model consists of a generation sub-network and an evaluation sub-network. The generation sub-network receives random latent vectors and scene configuration vectors, and outputs anonymous candidate samples with the same dimension as the feature vector set after multi-layer nonlinear mapping. The evaluation sub-network receives the anonymous candidate samples and real samples, and calculates intermediate indicators of privacy risk and intermediate indicators of task loss. S32. Based on the intermediate privacy risk index and the intermediate task loss index, calculate the total loss according to the multi-objective optimization loss function, and update the parameters of the generative subnetwork and the evaluation subnetwork through the backpropagation algorithm to continuously optimize the generative anonymization model. When the overall privacy risk index and the overall task utility index on the validation set both reach the preset standard, save the converged generative anonymization model.

6. The efficient generative anonymization method for big data according to claim 5, characterized in that, In step S31, the generating subnetwork adopts one or more combined structures of generative adversarial network, variational autoencoder or diffusion model; the output layer of the generating subnetwork adopts the corresponding activation function according to the original type of multiple fields to restore the output vector to an anonymous record that matches the field classification record set form in the feature vector set.

7. The efficient generative anonymization method for big data according to claim 1, characterized in that, Step S4 further includes: S41. Assign a scenario number to each shared scenario, record the corresponding privacy threshold, task weight and compliance requirements, form a scenario configuration file, and package the parameters in the scenario configuration file into a scenario configuration vector. S42. Input the scene configuration vector and the random latent vector sampled from the latent space into the generative subnetwork of the generative anonymization model, and output the anonymized data records in batches to obtain the target anonymized dataset. S43. Evaluate the target anonymized dataset, output a qualified version of the target anonymized dataset, and publish it to the data application system.

8. The efficient generative anonymization method for big data according to claim 7, characterized in that, Step S43 specifically includes: S431. Evaluate the target anonymized dataset, specifically including: The target anonymized dataset is subjected to re-identification, member inference, and attribute inference tests under a preset attack model. The overall privacy risk index is compared with the current scenario privacy threshold to obtain the first comparison difference. Calculate the overall task utility index on the target anonymized dataset and compare it with the real data performance baseline to obtain a second comparison difference; S432. When both the first comparison difference and the second comparison difference are less than or equal to the corresponding preset difference, the target anonymized dataset is marked as a qualified version and published to the data application system. When either the first comparison difference or the second comparison difference is greater than or equal to the corresponding preset difference, the weight coefficient of the loss term and the task weight are adjusted according to the deviation information to continuously optimize the generative anonymization model.

9. The efficient generative anonymization method for big data according to claim 7, characterized in that, In step S43, when publishing a qualified version of the target anonymized dataset, the model version number, parameter configuration snapshot, and evaluation results used in this publication are synchronously recorded in the scenario configuration file as an auditable basis.

10. A highly efficient generative anonymization system for big data, characterized in that, include: Modeling module: Used to preprocess and model features of the original large dataset to obtain a set of feature vectors; The building module is used to construct privacy risk indicators and task utility indicators based on the configuration parameters of the target sharing scenario, respectively, for the degree of privacy leakage of the generated data and the performance of downstream tasks, so as to obtain a privacy utility quantitative indicator system. Training module: used to input the feature vector set and the privacy utility quantification index system into the pre-built model, and train the pre-built model through multi-objective optimization loss function to obtain a generative anonymization model; Evaluation module: This module is used to input the scenario configuration vector into the generative anonymization model, generate the target anonymization dataset, evaluate the target anonymization dataset according to the privacy utility quantification index system, and output the target anonymization dataset that passes the evaluation and publish it to the data application system.

Images