Data anonymization processing methods, apparatus, computer equipment and storage media
By constructing a pool of target algorithms with multiple anonymization strategies and utilizing Pareto optimization and evolutionary algorithms, the problem of dynamically balancing privacy protection and data availability in multimodal environments for data anonymization methods is solved, thereby improving anonymization efficiency and reducing data loss rate.
Patent Information
- Application Number
- CN202611630394.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing data anonymization methods are difficult to adapt to the multimodal and dynamically changing IoT environment, and cannot achieve a dynamic balance between privacy protection and data availability under different data types and business scenarios.
By constructing a pool of target algorithms for multiple anonymization strategies, and using the Pareto algorithm to determine candidate anonymization strategies based on the sensitivity type and sensitivity level of the fields, the strategies are optimized by combining genetic and differential evolution algorithms, thereby achieving a balance between privacy protection and data availability.
It improves anonymization efficiency, reduces data loss rate, and achieves a flexible balance between privacy protection and data availability in different scenarios.
Smart Images

Figure CN123116344A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a data anonymization processing method, apparatus, computer equipment, and storage medium. Background Technology
[0002] With the deepening of market-oriented allocation reforms of data elements, the value of data as a new type of production factor is becoming increasingly prominent. The data element market requires data to be frequently uploaded, dynamically circulated, and compliantly verifiable, which places higher demands on data security processing technologies. Among these, data anonymization, as a key privacy protection measure in data release and circulation, aims to achieve a reasonable balance between protecting personal privacy and maintaining data availability.
[0003] In related technologies, data anonymization methods use fixed algorithm selection and static parameter configuration. When faced with multimodal and dynamically changing data environments such as the Internet of Things, it is difficult to adaptively adjust the strategy and achieve a dynamic balance between privacy protection and data availability under different data types and business scenarios. Summary of the Invention
[0004] In view of this, this application provides a data anonymization processing method, apparatus, computer equipment, and storage medium.
[0005] In a first aspect, embodiments of this application provide a data anonymization processing method, including: Retrieve raw data from multiple access data sources; Based on the name and sample value of each field in the original data, determine the sensitivity type and sensitivity level of the field; Based on the various anonymization algorithms set, a target algorithm pool is constructed, which includes multiple anonymization strategies. Each anonymization strategy includes: the anonymization algorithm corresponding to the field and / or the execution parameters of the anonymization algorithm. At least some fields in different anonymization strategies have different anonymization algorithms and / or different execution parameters. Based on the sensitivity type and sensitivity level of each field, as well as the original data, determine the privacy protection strength value and data availability index value of each anonymization strategy in the target algorithm pool; Based on the privacy protection strength value and data availability index value of each anonymization strategy, at least one candidate anonymization strategy is determined using the Pareto algorithm; and the target anonymization strategy is determined from the at least one candidate anonymization strategy. The original data is anonymized based on the target anonymization strategy to generate target data.
[0006] In one optional implementation, after generating the target data, the method further includes: The privacy protection strength assessment index and data availability assessment index for this anonymization process are determined. The privacy protection strength assessment index includes at least one of the following: re-identification risk, information entropy, L-diversity achievement rate, and differential privacy budget consumption. The data availability assessment index includes at least one of the following: information loss, statistical error, and task utility index. Based on the privacy protection strength assessment index and the data availability assessment index, the assessment result of this anonymization process is determined; When the evaluation result is "evaluation passed", an evaluation report for this anonymization process will be generated. If the evaluation result is "not passed", the target algorithm pool is updated and the target anonymization strategy is redefined.
[0007] In one optional implementation, the method further includes: In response to a data query request, the query data is determined from the target data, and the data usage scenario of the query data is determined. Based on the data usage scenario, determine the current privacy budget allocated to the data query request; Based on the current privacy budget, the query data is adjusted, and the adjusted query data is generated and fed back.
[0008] In one optional implementation, the method further includes: Based on the identifier of the dataset to which the queried data belongs, determine the remaining privacy budget of the dataset; The step of adjusting the query data according to the current privacy budget, generating and returning the adjusted data includes: When the remaining privacy budget is greater than or equal to the current privacy budget, the query data is adjusted according to the current privacy budget, and the adjusted query data is generated and fed back.
[0009] In one optional implementation, the target anonymization strategy is multiple, and the target data includes: multiple anonymized data with different levels of anonymity generated after processing the original data using multiple target anonymization strategies respectively; The method further includes: Upon receiving a data retrieval request, determine the data user of the data retrieval request; Based on the authorization level of the data user, the target anonymous data matching the data user is determined from the various anonymous data, and the call data corresponding to the data call request is determined and returned from the target anonymous data.
[0010] In one optional implementation, the method further includes: The system stores the target data, the original data, and the intermediate state data corresponding to the target data; wherein the intermediate state data is auxiliary data generated during the anonymization process and persistently stored, and the auxiliary data includes at least one of the following: semantic vector representation, generalization hierarchical structure, clustering grouping results, and noise seed; The system periodically detects the multiple data sources, and when new data is detected in the multiple data sources, it determines similar data corresponding to the new data from the original data. Based on the intermediate state data corresponding to the similar data, the newly added data is processed to generate processed data.
[0011] In one optional implementation, the construction of a target algorithm pool comprising multiple anonymization strategies based on the set multiple anonymization algorithms includes: Based on the various anonymization algorithms set, an initial population including multiple anonymization strategies after initialization is constructed; The initial population is used as the population to be processed. A genetic algorithm is used to screen, crossover, and mutate multiple anonymization strategies in the population to be processed to generate an updated population. Determine the updated diversity index values for the population; When the diversity index value is greater than the set diversity threshold, the updated population is treated as a population to be processed and returned to the process of processing using a genetic algorithm. When the diversity index value is less than or equal to the diversity threshold, the updated population is processed in multiple rounds using the differential evolution algorithm to generate a target algorithm pool that includes multiple anonymization strategies.
[0012] In one optional implementation, determining the sensitivity type and sensitivity level of each field based on its name and sample value in the original data includes: The name and sample value of each field in the original data are input into the trained field encoding model to generate the semantic vector representation of the field. Using a named entity recognition header and a conditional random field (CRF) module, the semantic vector representation of the field is processed, and the start-middle-not-entity BIO tag of each character is output; based on the BIO tags of each character, the sensitivity type of the field is determined. Based on the set sensitivity assessment dimensions, the sensitivity level of the field is determined; wherein the sensitivity assessment dimensions include: the inherent sensitivity of the entity type, the identifiability strength of the field, the risk of combined re-identification of fields, and the rule requirements used.
[0013] Secondly, embodiments of this application also provide a data anonymization processing apparatus, comprising: The access module is used to acquire raw data from various access data sources; The first determining module is used to determine the sensitivity type and sensitivity level of each field based on the name and sample value of each field in the original data. The second determining module is used to construct a target algorithm pool including multiple anonymization strategies based on a set number of anonymization algorithms, wherein each anonymization strategy includes: an anonymization algorithm corresponding to a field and / or execution parameters of the anonymization algorithm, and at least some fields in different anonymization strategies have different anonymization algorithms and / or different execution parameters; The third determining module is used to determine the privacy protection strength value and data availability index value of each anonymization strategy in the target algorithm pool based on the sensitivity type, sensitivity level, and the original data corresponding to each field. The fourth determining module is used to determine at least one candidate anonymization strategy based on the privacy protection strength value and data availability index value of each anonymization strategy using the Pareto algorithm; and to determine the target anonymization strategy from the at least one candidate anonymization strategy. The generation module is used to anonymize the original data based on the target anonymization strategy to generate target data.
[0014] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described data anonymization processing method.
[0015] This application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described data anonymization processing method.
[0016] The data anonymization processing method, apparatus, computer equipment, and storage medium provided in this application, after determining the sensitivity type and sensitivity level of a field, construct a target algorithm pool including multiple anonymization strategies. Based on the sensitivity type, sensitivity level, and original data corresponding to each field, the privacy protection strength value and data usability index value of each anonymization strategy in the target algorithm pool are determined. Then, based on the privacy protection strength value and data usability index value of each anonymization strategy, at least one candidate anonymization strategy is determined using the Pareto algorithm. This ensures that the candidate anonymization strategies balance privacy protection and data usability. Subsequently, the target anonymization strategy can be flexibly determined from at least one candidate anonymization strategy according to the application scenario, achieving a balance between privacy protection and data usability in different scenarios. This overcomes the limitations of using fixed algorithms in related technologies, effectively improves anonymization efficiency, and reduces data loss rate. Attached Figure Description
[0017] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating a data anonymization processing method according to an exemplary embodiment of this application; Figure 2 This is a schematic diagram of a data anonymization processing apparatus shown in an exemplary embodiment of this application; Figure 3 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation
[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0019] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0020] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0021] To facilitate understanding of this embodiment, a detailed description of the data anonymization method disclosed in this application embodiment will be provided first. The execution subject of the data anonymization method provided in this application embodiment is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, embedded device, etc. In some possible implementations, this data anonymization method can be implemented by the processor calling computer-readable instructions stored in memory.
[0022] See Figure 1 The diagram shows a flowchart of a data anonymization processing method provided in an embodiment of this application. The method includes steps S101 to S106, wherein: S101. Obtain raw data from multiple access data sources.
[0023] This application allows for the establishment of a unified data access module to interface with multiple data sources and acquire raw data from these sources. The raw data may include, but is not limited to, at least one field and sample values of that field. Data sources include, but are not limited to, relational databases, NoSQL databases, data interfaces, and file systems.
[0024] After obtaining the raw data from the data source, the raw data can be standardized. Standardization processes include normalization, metadata extraction, and data quality testing. Format normalization converts heterogeneous data from different data sources into a unified internal data format, standardized field type definitions, and coding standards. Metadata extraction automatically extracts metadata information such as field names, field types, data distribution, and missing rates from the dataset, providing a foundation for subsequent classification and grading. Data quality testing checks the completeness, consistency, and accuracy of the data, identifies outliers and missing values, and can generate data quality reports based on the test results.
[0025] S102. Based on the name and sample value of each field in the original data, determine the sensitivity type and sensitivity level of the field.
[0026] After obtaining the raw data, it is necessary to detect the sensitivity type and sensitivity level of the data before publishing it. Based on the sensitivity type and sensitivity level, a matching anonymization algorithm should be selected to anonymize the raw data, protect the data information of the raw data, and avoid the leakage of sensitive information.
[0027] In practice, a trained neural network model for sensitive information detection can be used to detect each field in the original data and determine the sensitivity type and sensitivity level of each field.
[0028] Optionally, determining the sensitivity type and sensitivity level of each field based on its name and sample value in the original data includes: Step a1: Input the name and sample value of each field in the original data into the trained field encoding model to generate the semantic vector representation corresponding to the field; Step a2: Using the Named Entity Recognition Header and Conditional Random Field (CRF) module, process the semantic vector representation of the field and output the start-middle-not-entity BIO tag for each character; determine the sensitivity type of the field based on the BIO tags of each character. Step a3: Determine the sensitivity level of the field based on the set sensitivity assessment dimensions; wherein the sensitivity assessment dimensions include: the inherent sensitivity of the entity type, the identifiability strength of the field, the risk of combined re-identification of fields, and the rule requirements used.
[0029] During implementation, the names and sample values of fields in the original data can be input into the trained field encoding model to generate semantic vector representations of the fields. The field encoding model can be a Bidirectional Encoder Representations from Transformers (BERT) model, and the dimension of the semantic vector representation can be set as needed.
[0030] The Named Entity Recognition (NER) head is fine-tuned using sample data from the acquired business scenario to enable it to recognize entities within that scenario. Then, the fine-tuned NER head and a Conditional Random Field (CRF) module are used to process the semantic vector representation of the fields, outputting a Begin-Inside-Outside (BIO) label for each character. For example, B-PER indicates the beginning of a person's name, I-PER indicates the inside of a person's name, and B-ID indicates the beginning of an ID number.
[0031] In this application, sensitive entity identification adopts a BIO labeling strategy, which labels each token in the field value sequence with its position and role in the sensitive entity, and combines it with a CRF module to optimize the entity boundary recognition accuracy, ensuring the accuracy and precision of the BIO labels of characters.
[0032] Based on the BIO tags of each character, determine the sensitivity type of the field. Sensitive types include, but are not limited to: personal identification, financial, health, and behavioral types. Personal identification types may include: name, ID number, mobile phone number, email address, and address; financial types may include: bank card number, transaction amount, and credit score; health types may include: diagnosis results, medical record number, and medication records; and behavioral types may include: location tracking, browsing history, and social relationships.
[0033] Furthermore, the sensitivity level of a field can be determined based on the set sensitivity assessment dimensions. These dimensions include: the inherent sensitivity of the entity type, the identifiability strength of the field, the risk of combined re-identification of fields, and the rules required for its use. The inherent sensitivity of the entity type reflects the sensitivity of the entity type itself. This inherent sensitivity can be set according to actual circumstances; for example, the sensitivity of an ID card number can be set higher than that of a mobile phone number, and vice versa. The identifiability strength of a field reflects its ability to uniquely identify an individual. For example, the higher the uniqueness of a field, the higher its sensitivity level. ID card numbers have high uniqueness, hence their high identifiability strength; conversely, height has low uniqueness, hence its low identifiability strength. The risk of combined re-identification of fields reflects the joint identification capability when multiple fields are combined, i.e., whether the combination of multiple fields can uniquely identify an individual. This can be achieved by calculating the joint distinguishability of the combined fields. The rules required for its use can be specifically limited according to relevant laws and regulations (such as the requirements for sensitive personal information under the Personal Information Protection Law) and rules set for business scenarios.
[0034] For example, the sensitivity level can be set to four levels, such as L1, L2, L3, and L4, with L1 being the least sensitive and L4 being the most sensitive. For instance, L4 could include ID card numbers, bank card numbers, biometrics, or other legally protected characteristics; L3 could include mobile phone numbers, email addresses, etc.; L2 could include gender, age, occupation, etc.; and L1 could include city, etc.
[0035] For example, the rules for determining sensitivity levels are shown in the table below:
[0036] In practical implementation, after generating the sensitivity type and sensitivity level of the fields, the recognition results can be displayed on the interactive interface. Maintenance personnel can manually review the results and make manual adjustments when misjudgments are detected to ensure the accuracy of data recognition. Furthermore, after correction, correction records can be generated and used as feedback data for incremental training of the model, achieving continuous improvement in hierarchical accuracy.
[0037] S103. Based on the set multiple anonymization algorithms, construct a target algorithm pool including multiple anonymization strategies, wherein each anonymization strategy includes: the anonymization algorithm corresponding to the field and / or the execution parameters of the anonymization algorithm, and at least some fields in different anonymization strategies have different anonymization algorithms and / or different execution parameters.
[0038] Various anonymization algorithms can be selected based on actual needs. These include, but are not limited to, k-anonymity, l-diversity, t-approximation, differential privacy, generalization, pseudonymization, and data suppression. Specifically, k-anonymity ensures that each record is indistinguishable from at least k-1 records through generalization or suppression, reducing the risk of re-identification; l-diversity ensures that sensitive attributes in equivalence classes have at least l distinct values, preventing attribute leakage attacks; t-approximation controls the difference between the distribution of sensitive attributes in equivalence classes and the global distribution to not exceed t, further constraining attribute inference; differential privacy adds calibration noise through Laplace or Gaussian mechanisms; generalization replaces specific attribute values with more general values (e.g., replacing "25 years old" with "20-30 years old"); pseudonymization replaces identifiers with pseudonyms; and data suppression deletes or hides specific field values.
[0039] In implementation, anonymization algorithms can be encapsulated as pluggable algorithm modules. These modules, representing various anonymization algorithms, can be integrated into a unified algorithm capability pool, allowing users to call these modules to process data. Each algorithm module exposes a unified structural specification, including but not limited to interfaces for configuring algorithm parameters, execution, information loss calculation, and privacy budget consumption.
[0040] During implementation, one or more anonymization algorithms from a variety of anonymization algorithms can be used to initialize and generate multiple anonymization strategies, thus constructing a target algorithm pool; alternatively, all possible anonymization algorithms and parameter combinations can be enumerated to construct the target algorithm pool. Each anonymization strategy includes: the anonymization algorithm corresponding to the field and / or the execution parameters of the anonymization algorithm. In different anonymization strategies, at least some fields correspond to different anonymization algorithms and / or have different execution parameters, and each field can correspond to one or more anonymization algorithms.
[0041] Optionally, the pool of target algorithms, comprising multiple anonymization strategies, is constructed based on the set anonymization algorithms, including: Step b1: Based on the various anonymization algorithms set, construct an initial population including multiple anonymization strategies after initialization; Step b2: Using the initial population as the population to be processed, a genetic algorithm is used to screen, crossover, and mutate multiple anonymization strategies in the population to be processed to generate an updated population. Step b3: Determine the updated diversity index value of the population; Step b4: When the diversity index value is greater than the set diversity threshold, the updated population is taken as the population to be processed and returned to the process of processing using the genetic algorithm. Step b5: When the diversity index value is less than or equal to the diversity threshold, the updated population is processed in multiple rounds using the differential evolution algorithm to generate a target algorithm pool that includes multiple anonymization strategies.
[0042] In this application, a hierarchical adaptive evolutionary algorithm can be used for policy space search. For example, the first layer can use a genetic algorithm for global exploration to generate an initial policy population; the second layer uses differential evolution for local optimization to refine policy parameters, and uses an adaptive policy switching mechanism to dynamically balance exploration and development to avoid getting trapped in local optima.
[0043] In practice, multiple anonymization strategies can be initialized first to obtain an initial population. This initial population is then used as the population to be processed. Using a genetic algorithm, the initialized anonymization strategies are evaluated, considering factors such as privacy protection strength, data availability, and processing latency. Based on the evaluation results, the selected anonymization strategies are selected from the population to be processed and subjected to crossover and mutation processes to generate new anonymization strategies, thus constructing an updated population.
[0044] The updated population diversity index is determined, which represents the dispersion of all strategies in the parameter space. For example, the average of the parameter vector distances between all strategies can be calculated and normalized to between 0 and 1. If the diversity index exceeds the set diversity threshold, it indicates that the strategies in the current population are significantly different and are still in the exploratory stage. Therefore, the updated population is treated as a population to be processed again using the genetic algorithm.
[0045] If the diversity index is less than or equal to the diversity threshold, it indicates that the strategies in the current population have been clustered into a specific area, and the fine-tuning selection stage can begin. Therefore, the differential evolution algorithm is used to process the updated population in multiple rounds. Each round of processing can include mutation, crossover, selection, etc., until the iteration cutoff condition is met, such as the number of iterations reaching a threshold or the fitness change being less than a threshold, etc., generating a target algorithm pool containing multiple anonymization strategies. This allows for more accurate selection of target anonymization strategies from the target algorithm pool in subsequent processes.
[0046] After generating the target algorithm pool, the process also includes: S104, determining the privacy protection strength value and data availability index value of each anonymization strategy in the target algorithm pool based on the sensitivity type, sensitivity level, and the original data corresponding to each field.
[0047] The methods for determining the privacy protection strength value and data usability indicator value can be set according to business needs. For example, the privacy protection strength can be evaluated based on the risk quantification method of the attack model. This method calculates the attack success rate or exposure risk by simulating the actual behavior of attackers. Evaluation indicators include, but are not limited to, attack success rate, ε-differential privacy budget, k-anonymity, etc. Alternatively, the amount of personal identity information remaining in the anonymized data can be measured based on the information entropy quantification method. Evaluation indicators include conditional entropy, mutual information, etc. For the evaluation of data usability, for example, it can be evaluated based on the accuracy loss of downstream tasks. This can be achieved by selecting a representative machine learning / analysis task and comparing the performance on the original data and the anonymized data to evaluate usability. Evaluation indicators include, for example, the accuracy of classification tasks and the mean squared error of regression tasks. Alternatively, it can be evaluated based on the statistical characteristic retention rate. This method does not depend on downstream tasks and can be directly measured from the perspective of data distribution. Evaluation indicators include, for example, mean / variance retention rate, correlation retention rate, distribution similarity, etc.
[0048] Then, in S105, based on the privacy protection strength value and data availability index value of each anonymization strategy, at least one candidate anonymization strategy is determined using the Pareto algorithm; and the target anonymization strategy is determined from the at least one candidate anonymization strategy.
[0049] This application models the selection of anonymization strategies as a multi-objective optimization problem, namely, maximizing data availability under given privacy constraints, and introduces Pareto Optimality theory to solve for the optimal strategy combination.
[0050] The objective function is defined as follows: Let the anonymization strategy be θ, its privacy protection strength be P(θ), and the data availability be U(θ). The strategy optimization objective is to maximize {P(θ), U(θ)}.
[0051] The constraints are: P(θ) ≥ P_min (Privacy protection strength is not lower than the minimum requirement); U(θ) ≥ U_min (data availability is not lower than the minimum requirement); Latency(θ) ≤ L_max (processing latency does not exceed the upper limit).
[0052] Based on the set policy optimization objective and constraints, Pareto optimality is used to solve the problem: For any two policies θ_i and θ_j, θ_i is said to dominate θ_j if and only if P(θ_i) ≥ P(θ_j) and U(θ_i) ≥ U(θ_j), and at least one strict inequality holds.
[0053] The Pareto front consists of all strategies that are not dominated by other strategies, and it includes at least one candidate anonymization strategy. The target anonymization strategy can then be determined from the Pareto front. For example, the optimal strategy can be selected based on the current use case preferences. If the current use case prioritizes privacy, the candidate anonymization strategy with the highest P can be selected; if the current use case prioritizes usability, the candidate anonymization strategy with the highest U can be selected.
[0054] For example, the algorithm can output the optimal algorithm combination and the corresponding parameter configuration (i.e., the target anonymization strategy). For example, use case: highly sensitive financial data + cross-border transmission → Recommendation: differential privacy (ε=0.5) + k-anonymity (k=20) + SM4 encryption; use case: low-sensitivity statistical reports + internal use → Recommendation: generalization (moderate granularity) + pseudonymization.
[0055] S106. Based on the target anonymization strategy, the original data is anonymized to generate target data.
[0056] After determining the target anonymization strategy, the original data can be anonymized using this strategy to generate the target data. This target data can then be published to the relevant data users.
[0057] In this application, after determining the sensitivity type and sensitivity level of a field, a target algorithm pool including multiple anonymization strategies is constructed. Based on the sensitivity type, sensitivity level, and original data corresponding to each field, the privacy protection strength value and data availability index value of each anonymization strategy in the target algorithm pool are determined. Then, based on the privacy protection strength value and data availability index value of each anonymization strategy, the Pareto algorithm is used to determine at least one candidate anonymization strategy. This ensures that the candidate anonymization strategies balance privacy protection and data availability. Subsequently, the target anonymization strategy can be flexibly determined from at least one candidate anonymization strategy according to the application scenario. This achieves a balance between privacy protection and data availability in different scenarios, overcomes the limitations of using fixed algorithms in related technologies, effectively improves anonymization efficiency, and reduces data loss rate.
[0058] Optionally, after generating the target data, the method further includes: Step c1: Determine the privacy protection strength assessment index and data availability assessment index for this anonymization process. The privacy protection strength assessment index includes at least one of the following: re-identification risk, information entropy, L-diversity achievement rate, and differential privacy budget consumption. The data availability assessment index includes at least one of the following: information loss, statistical error, and task utility index. Step c2: Based on the privacy protection strength assessment index and the data availability assessment index, determine the assessment result of this anonymization process; Step c3: If the evaluation result is "evaluation passed", generate an evaluation report for this anonymization process; if the evaluation result is "evaluation failed", update the target algorithm pool and redetermine the target anonymization strategy.
[0059] After generating the target data, the anonymization process can be evaluated to ensure data security and usability. During implementation, the original data, target data, field sensitivity types, sensitivity levels, and other data can be used to determine privacy protection strength assessment indicators and data usability assessment indicators for this anonymization process. The privacy protection strength assessment indicators include at least one of the following: re-identification risk, information entropy, L-diversity compliance rate, and differential privacy budget consumption.
[0060] Re-identification risk can be assessed based on a record-linking attack model, evaluating the probability of re-identification when anonymized data is associated with an external dataset. The calculation can be as follows: for each equivalence class, Risk = 1 / |G| (under the uniform assumption). Further adjustments can be made by considering the availability of external knowledge, for example, through a "Bayesian attack model" or a "scenario-based weighted approach." An equivalence class is a set of all records that have the exact same generalization value on a quasi-identifier (such as age, gender, or postal code). |G| represents the number of records included in that equivalence class.
[0061] Information entropy measures the diversity and uncertainty of sensitive attributes within an equivalence class; the formula is: Entropy(G) = -Σ p(s) · log2p(s). Here, Entropy(G) represents the information entropy of the equivalence class G, with a higher value indicating better privacy protection. p(s) represents the probability (or proportion) of the sensitive attribute s appearing in the equivalence class G. For example, we can determine the number of records with sensitive attribute s in the equivalence class G, and then divide this number by the total number of records in the equivalence class G to obtain p(s).
[0062] The l-diversity compliance rate is used to statistically determine the proportion of equivalence classes that meet the l-diversity requirements, reflecting the defense capability against attribute leakage attacks. Differential privacy budget consumption is used to record the cumulative privacy budget ε consumption, ensuring that the total consumption does not exceed a preset upper limit (the combinatorics theorem of differential privacy requires strict control over the cumulative budget published multiple times).
[0063] Data usability assessment metrics include at least one of the following: information loss, statistical error, and task utility. Specifically, information loss quantifies the degree of information degradation caused by anonymization; in implementation, normalized mean squared error can be used for numerical attributes, and the height loss of the generalization tree can be used for categorical attributes. Statistical error measures the deviation of the statistical distribution of the anonymized data from the original data, including mean error, quantile error, and correlation error. Task utility compares the decline in model performance before and after anonymization for specific application tasks (such as classification, regression, and clustering).
[0064] Furthermore, the evaluation result of this anonymization process can be determined based on privacy protection strength assessment indicators and data availability assessment indicators. For example, thresholds corresponding to multiple indicators can be set. By comparing the indicator values with the thresholds, if all requirements are met, the evaluation result is considered to be passed; otherwise, the evaluation result is considered to be failed.
[0065] Alternatively, weights can be assigned to each indicator, and the indicators can be summed with weights. The sum is then compared to a set threshold to determine the evaluation result. For example, this application designs a Relative Utility-Threat (RUT) comprehensive indicator, unifying privacy protection strength assessment indicators and data availability assessment indicators onto a 0-1 rating scale for comparison and weighting, so as to determine the evaluation result based on the RUT comprehensive indicator. The determination of the RUT comprehensive indicator is as follows: RUT = α · U_norm + (1-α) · (1 - R_norm).
[0066] Where U_norm and R_norm are the normalized usability score (i.e., the score value determined according to the data usability assessment index) and the risk score (i.e., the score value determined according to the privacy protection strength index), respectively, and α is the scenario preference coefficient (e.g., it can be set to 0.4, 0.5, etc.).
[0067] When the assessment result is determined to be passed, an anonymized assessment report can be generated. This assessment report may include, but is not limited to, the values of indicators for each dimension, the judgment of compliance, the comprehensive score of RUT, etc., and the assessment report can be exported to the regulatory filing format.
[0068] When the evaluation result is determined to be unsuccessful, the target algorithm pool can be updated and the target anonymization strategy can be redefined. The original data can then be anonymized again using the newly defined target anonymization strategy to ensure the availability and confidentiality of the processed data.
[0069] For example, the evaluation process for this application is as follows: Step 1. After the anonymization process is completed and the target data is generated, the above-mentioned indicators are automatically collected, namely the privacy protection strength assessment indicator and the data availability assessment indicator.
[0070] Step 2. Standardize each indicator and map it to the 0-1 range.
[0071] Step 3. Compare the standardized indicators with the preset thresholds.
[0072] Step 4. If the evaluation fails (e.g., insufficient privacy protection or low usability), a closed loop of policy adjustment and reprocessing is triggered. That is, the system automatically adjusts the algorithm parameters (e.g., increasing the k value, reducing noise accuracy, or selecting other anonymization algorithms) and re-executes anonymization until the evaluation passes.
[0073] Step 5. After the assessment is passed, a compliance assessment report will be automatically generated, which includes the indicator values of each dimension, the compliance judgment, and the RUT comprehensive score. It can be exported to the regulatory filing format.
[0074] Research has found that related technologies typically assess privacy or usability in isolation, lacking a unified quantitative measurement framework. Risk assessments and utility evaluations for re-identification often employ fragmented and incompatible indicators, making it difficult to objectively verify the anonymization effect and provide quantifiable compliance evidence for regulators. This application, however, achieves a unified quantitative assessment of the privacy protection strength and data usability of anonymization processing by setting privacy protection strength and data usability assessment indicators, ensuring both the usability and privacy protection of the processed data. Furthermore, it can automatically generate compliance assessment reports, providing objective and verifiable evidence for data uploading and regulatory filing, effectively meeting the compliance requirements of the data element market.
[0075] Optionally, the method further includes: in response to a data query request, determining query data from the target data and determining the data usage scenario of the query data; determining a current privacy budget allocated to the data query request based on the data usage scenario; adjusting the query data based on the current privacy budget, generating and feeding back the adjusted query data.
[0076] This application can also adaptively adjust the privacy budget based on the data's intended use. For example, given the different availability requirements for data sold online versus internal circulation, the data processing system can automatically adjust the privacy budget allocation based on the data usage scenario. This allows for dynamic adjustment of the data to be used according to different privacy budgets, ensuring data security. The privacy budget represents the amount of privacy data allowed to be consumed when the data is used. Generally, a larger privacy budget means less noise is added to the data, resulting in more accurate query results, but weaker privacy protection. Conversely, a smaller privacy budget means more noise is added to the data, resulting in more ambiguous query results, but stronger privacy protection.
[0077] Specifically, after a data user initiates a data query request, the data processing system responds by identifying the query data from the target data and determining the data usage scenario. This scenario can be set when the data user initiates the query request, or it can be set based on the data user's permissions. Then, based on the data usage scenario and the mapping between the set scenario and the budget, the system determines the current privacy budget allocated to the data query request. For example, if the data usage scenario is internal statistical analysis, the current privacy budget can be set larger; if the data user's permissions indicate external publishing, the current privacy budget can be set smaller. Furthermore, the system can adjust the query data based on the current privacy budget. For instance, it can perform differential privacy processing based on the current privacy budget to add noise to the query data, generating and feeding back the adjusted query data.
[0078] The above process can further ensure data security and enable the display of data with different levels of anonymity for different use scenarios.
[0079] Optionally, the method further includes: determining the remaining privacy budget of the dataset based on the identifier of the dataset to which the queried data belongs.
[0080] In practical implementation, to avoid data leakage caused by multiple queries on the dataset and to ensure data privacy and security, this application allows setting a corresponding total privacy budget for each dataset. Different datasets may have different total privacy budgets. The system can also maintain a budget list in real time to update and record the remaining privacy budget for each dataset. Upon receiving a data query request and identifying the data to be queried, the system can also determine the identifier of the dataset to which the queried data belongs, and use this identifier to determine the remaining privacy budget for that dataset from the budget list.
[0081] When the remaining privacy budget is greater than or equal to the current privacy budget, the query data is adjusted according to the current privacy budget, and the adjusted query data is generated and fed back. Conversely, when the remaining privacy budget is less than the current privacy budget, the query request can be rejected; alternatively, manual review can be used to determine whether to respond to the data query request.
[0082] As can be seen, this application sets up a privacy budget cumulative control strategy. The combinatorial requirements of differential privacy consume privacy budget with each release. This application adopts a privacy budget accounting mechanism to track cumulative privacy consumption in real time. When the cumulative consumption approaches the preset limit, it automatically switches to a more lenient budget strategy (such as a strategy with a smaller budget) or triggers manual review.
[0083] This application also includes a user-level adaptive strategy, meaning that users with different authorization levels can obtain anonymized data with different levels of sophistication, enabling flexible publication of anonymized data. Optionally, the target anonymization strategy can be multiple, and the target data includes: multiple anonymized data with different levels of anonymity generated after processing the original data using multiple target anonymization strategies respectively.
[0084] The method further includes: upon receiving a data call request, determining the data user of the data call request; based on the authorization level of the data user, determining the target anonymous data matching the data user from the multiple anonymous data, and determining and returning the call data corresponding to the data call request from the target anonymous data.
[0085] In implementation, after determining the candidate anonymization strategies, multiple target anonymization strategies can be selected. These strategies will produce different levels of anonymity when processing the original data; for example, strategies with varying degrees of privacy protection can be chosen. In other words, the target data includes anonymized data with different levels of anonymity generated after processing the original data using various target anonymization strategies.
[0086] Furthermore, upon receiving a data request, it can identify the data user and determine the authorization level of that user; it can also identify the target anonymous data matching the authorization level from a variety of anonymous data, and identify and return the request data corresponding to the data request from the target anonymous data, thereby returning data with different levels of granularity to different data users.
[0087] In another implementation, after generating a Pareto front including multiple candidate anonymization strategies, these strategies can be associated with the original data. Upon receiving a data request, a target anonymization strategy is determined from the candidate strategies based on the data user's authorization level. For example, if the data user has a high authorization level, a candidate anonymization strategy with high availability (i.e., high U) and low privacy protection strength (i.e., low P) can be selected as the target anonymization strategy. The target anonymization strategy is then used to anonymize the data requested in real time, and the processed anonymized data is returned to the data requester.
[0088] Optionally, this application may also include a processing rule base containing various rules that can be used to restrict anonymity requirements for specific attributes in the data. During implementation, the rules in the rule base can be updated in real time according to laws and regulations to ensure the compliance of the processed data.
[0089] Optionally, the method further includes: storing the target data, the original data, and intermediate state data corresponding to the target data; wherein the intermediate state data is auxiliary data generated during the anonymization process and persistently stored, and the auxiliary data includes at least one of the following: semantic vector representation, generalization hierarchical structure, clustering grouping results, and noise seed; periodically detecting the multiple data sources, and when new data is detected in the multiple data sources, determining similar data corresponding to the new data from the original data; and processing the new data according to the intermediate state data corresponding to the similar data to generate processed data.
[0090] After generating the target data, this application can also store the target data, the original data, and the intermediate state data corresponding to the target data, so that the incremental data can be processed more efficiently using the intermediate state data later.
[0091] Intermediate state data is auxiliary data generated during the anonymization process, including at least one of the following: semantic vector representation, generalization hierarchy (which represents the level to which each field is generalized), clustering grouping results (representing the equivalence class to which each original record is assigned), noise seeds (such as the source identifier of random noise in differential privacy), etc.
[0092] The system periodically checks multiple data sources to determine if there are any updates, i.e., if new data has been added. When new data is found, similar data can be identified. For example, the semantic vector representation of fields in the new data can be determined, and similar data can be identified from the original data using these semantic vectors. Then, based on the intermediate state data corresponding to the similar data, the incremental data (i.e., the new data) is processed. For instance, based on clustering results, the incremental data can be assigned to the equivalence classes corresponding to the similar data; alternatively, it can be processed based on noise seeds.
[0093] Research has found that the data element market is characterized by high-frequency updates. To avoid the inefficiency caused by full-scale processing, this application designs an incremental update decision based on differential privacy. Specifically, when the initial dataset is first uploaded, a complete anonymization process can be executed, while saving intermediate state data generated during processing (such as clustering results, generalization hierarchical structures, noise seeds, etc.). When detecting new data, the data processing system performs specific processing only on the new data. For example, it can assign the new data to an existing equivalence class structure (e.g., finding the nearest equivalence class based on distance metrics); it can also perform generalization or noise addition operations only on the new data; and it can perform incremental correction on published statistical results, i.e., calculating the incremental impact of new data on statistics, rather than recalculating global statistics. Through the above process, rapid processing of incremental data is achieved.
[0094] Optionally, the incremental update process may include: 1. Change detection: Periodically scan the source dataset to identify newly added, deleted, and modified records. 2. Incremental grouping: Map changed records to existing equivalence class structures. If a newly added record cannot be assigned to an existing equivalence class, a new equivalence class is created. 3. Incremental perturbation: Perform perturbation operations (generalization or noise addition) only on changed records; existing records remain unchanged. 4. Incremental release: Release the updated anonymized dataset and update the audit log to record the changes. 5. Budget update: Update the cumulative privacy budget consumption value and check if budget threshold alarms have been triggered.
[0095] Research has found that differential privacy models in related technologies are mainly designed for static datasets and cannot efficiently handle incremental update scenarios. Furthermore, traditional methods, under the high-frequency update demands of the data element market, require re-anonymizing all data with each arrival of incremental data, resulting in high computational overhead, high processing latency, and low efficiency, making it difficult to meet real-time requirements. This application, however, utilizes an incremental anonymization mechanism and a privacy budget accounting mechanism, requiring only incremental processing instead of full recalculation for newly added or changed data. This significantly reduces computational overhead and processing latency, improves incremental data processing efficiency, and enables the system to efficiently meet the business needs of high-frequency updates in the data element market. Related research has demonstrated that differential privacy publishing algorithms in incremental update scenarios can maintain high availability and scalability.
[0096] This application proposes an intelligent data anonymization system for the data element market, whose architecture consists of six layers from bottom to top: 1. Data Access Layer: This layer interfaces with various data sources (databases, data interfaces, file systems, etc.) to achieve standardized data collection and preprocessing. It performs data access operations, namely, standardized collection and metadata extraction of multi-source data.
[0097] 2. Intelligent Hierarchy Layer: Based on NLP and machine learning technologies, this layer automatically identifies sensitive field types and labels them with sensitivity levels, automating data classification and grading. This layer performs the grading and classification process, enabling automatic identification of sensitive fields and labeling of their sensitivity levels.
[0098] 3. Strategy Decision Layer: An adaptive multi-algorithm collaborative anonymization framework is constructed. Based on data security level, business scenario, and usage requirements, a multi-objective optimization model automatically selects the optimal combination of algorithms such as k-anonymity, differential privacy, generalization, and pseudonymization. Here, a strategy matching operation is performed, i.e., the optimal anonymization strategy combination is automatically selected based on multi-objective optimization.
[0099] 4. Anonymization Execution Layer: Executes the selected anonymization strategy, supports incremental anonymization processing, and only performs incremental operations on newly added or modified data. That is, it performs anonymization operations and supports both full and incremental processing methods.
[0100] 5. Quantitative Assessment Layer: Establish a two-dimensional quantitative assessment system for privacy protection strength and data availability, and automatically generate compliance assessment reports through indicators such as re-identification risk, information entropy, and statistical error.
[0101] 6. Operations Management (i.e., the output and deployment stage): Implement end-to-end process control, including task orchestration, status monitoring, log traceability, and compliance certification output.
[0102] This application implements end-to-end log traceability. Each level of operation log is structured and includes: operation timestamp (accurate to milliseconds), operation type (e.g., access / tiering / policy matching / de-identification / evaluation / output), operation object (e.g., dataset identifier, number of records), policy parameters (e.g., selected algorithm, key parameter values), evaluation results (e.g., values of various dimensions, pass / fail status), and operator / system identifier. Logs are simultaneously written to local audit databases and blockchain systems, achieving tamper-proof evidence storage and meeting the data element market's requirements for regulatory verification.
[0103] The core concept of this application is to upgrade anonymization from "static one-time processing" to "intelligent decision-making system throughout the entire life cycle" by constructing a five-in-one intelligent anonymization system that integrates "intelligent classification, adaptive combination, incremental processing, dual-dimensional evaluation, and end-to-end control". This provides a systematic solution for the specific needs of the data element market, which requires "high-frequency listing, dynamic circulation, and verifiable supervision".
[0104] This application constructs an end-to-end automated compliance closed loop, which forms a complete automated management and control process from data access, classification and grading, policy matching, de-identification, effect evaluation to output and uploading; the closed-loop mechanism of "assessment failure → policy adjustment → reprocessing" ensures the compliance of output data; full log traceability and blockchain evidence storage provide reliable technical support for supervision.
[0105] Research has shown that the implementation of this application will significantly improve the efficiency and compliance of data listing and circulation in the data element market, reduce the cost of manual intervention, effectively support the high-frequency, large-scale, and compliant circulation of data elements, and provide a reliable technical foundation for the healthy development of the data element market.
[0106] Corresponding to the aforementioned embodiments of the data anonymization processing method, this application also provides embodiments of the data anonymization processing apparatus. Figure 2 A schematic diagram of the data anonymization processing apparatus provided in this application specifically includes: Access module 201 is used to obtain raw data from multiple access data sources; The first determining module 202 is used to determine the sensitivity type and sensitivity level of each field based on the name and sample value of each field in the original data; The second determining module 203 is used to construct a target algorithm pool including multiple anonymization strategies based on the set multiple anonymization algorithms, wherein each anonymization strategy includes: the anonymization algorithm corresponding to the field and / or the execution parameters of the anonymization algorithm, and at least some fields in different anonymization strategies have different anonymization algorithms and / or different execution parameters; The third determining module 204 is used to determine the privacy protection strength value and data availability index value of each anonymization strategy in the target algorithm pool based on the sensitivity type, sensitivity level and the original data corresponding to each field. The fourth determining module 205 is used to determine at least one candidate anonymization strategy based on the privacy protection strength value and data availability index value of each anonymization strategy using the Pareto algorithm; and to determine the target anonymization strategy from the at least one candidate anonymization strategy. The generation module 206 is used to anonymize the original data based on the target anonymization strategy to generate target data.
[0107] In one optional implementation, after generating the target data, the apparatus further includes an evaluation module, configured to: The privacy protection strength assessment index and data availability assessment index for this anonymization process are determined. The privacy protection strength assessment index includes at least one of the following: re-identification risk, information entropy, L-diversity achievement rate, and differential privacy budget consumption. The data availability assessment index includes at least one of the following: information loss, statistical error, and task utility index. Based on the privacy protection strength assessment index and the data availability assessment index, the assessment result of this anonymization process is determined; When the evaluation result is "evaluation passed", an evaluation report for this anonymization process will be generated. If the evaluation result is "not passed", the target algorithm pool is updated and the target anonymization strategy is redefined.
[0108] In one optional implementation, the apparatus further includes a publishing module for: In response to a data query request, the query data is determined from the target data, and the data usage scenario of the query data is determined. Based on the data usage scenario, determine the current privacy budget allocated to the data query request; Based on the current privacy budget, the query data is adjusted, and the adjusted query data is generated and fed back.
[0109] In one optional implementation, the publishing module is further configured to: Based on the identifier of the dataset to which the queried data belongs, determine the remaining privacy budget of the dataset; When the publishing module adjusts the query data according to the current privacy budget and generates and returns the adjusted query data, it is configured to: adjust the query data according to the current privacy budget when the remaining privacy budget is greater than or equal to the current privacy budget, and generate and return the adjusted query data.
[0110] In one optional implementation, the target anonymization strategy is multiple, and the target data includes: multiple anonymized data with different levels of anonymity generated after processing the original data using multiple target anonymization strategies respectively; The publishing module is further configured to: upon receiving a data call request, determine the data user of the data call request; based on the authorization level of the data user, determine the target anonymous data matching the data user from the various anonymous data, and determine and return the call data corresponding to the data call request from the target anonymous data.
[0111] In one optional implementation, the apparatus further includes a processing module for: The system stores the target data, the original data, and the intermediate state data corresponding to the target data; wherein the intermediate state data is auxiliary data generated during the anonymization process and persistently stored, and the auxiliary data includes at least one of the following: semantic vector representation, generalization hierarchical structure, clustering grouping results, and noise seed; The system periodically detects the multiple data sources, and when new data is detected in the multiple data sources, it determines similar data corresponding to the new data from the original data. Based on the intermediate state data corresponding to the similar data, the newly added data is processed to generate processed data.
[0112] In one optional implementation, the second determining module, when constructing a target algorithm pool including multiple anonymization strategies based on a set of anonymization algorithms, is used to: Based on the various anonymization algorithms set, an initial population including multiple anonymization strategies after initialization is constructed; The initial population is used as the population to be processed. A genetic algorithm is used to screen, crossover, and mutate multiple anonymization strategies in the population to be processed to generate an updated population. Determine the updated diversity index values for the population; When the diversity index value is greater than the set diversity threshold, the updated population is treated as a population to be processed and returned to the process of processing using a genetic algorithm. When the diversity index value is less than or equal to the diversity threshold, the updated population is processed in multiple rounds using the differential evolution algorithm to generate a target algorithm pool that includes multiple anonymization strategies.
[0113] In an optional implementation, when the first determining module determines the sensitivity type and sensitivity level of each field based on its name and sample value in the original data, it is used to: The name and sample value of each field in the original data are input into the trained field encoding model to generate the semantic vector representation of the field. Using a named entity recognition header and a conditional random field (CRF) module, the semantic vector representation of the field is processed, and the start-middle-not-entity BIO tag of each character is output; based on the BIO tags of each character, the sensitivity type of the field is determined. Based on the set sensitivity assessment dimensions, the sensitivity level of the field is determined; wherein the sensitivity assessment dimensions include: the inherent sensitivity of the entity type, the identifiability strength of the field, the risk of combined re-identification of fields, and the rule requirements used.
[0114] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.
[0115] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0116] This application also provides a computer-readable storage medium storing a computer program that can be used to execute the data anonymization processing method described in the above embodiments.
[0117] This application also provides a computer device, see [link to relevant documentation] Figure 3 The diagram shown illustrates the structure of the computer device provided in this application. At the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for various business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the data anonymization processing method described in the above embodiments. Of course, besides software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.
[0118] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs) containing computer-usable program code. The form of a computer program product implemented on ROM, optical memory, etc.
[0119] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0122] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0123] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, and optical disc read-only memory (CD). ROM, digital multifunction optical disc (DVD) or other optical storage, magnetic cassette tape, magnetic magnetic disk storage or other magnetic storage devices or any other non-transferable medium, may be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0124] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0125] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0127] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.
Claims
1. A data anonymization processing method, characterized in that, The method includes: Retrieve raw data from multiple access data sources; Based on the name and sample value of each field in the original data, determine the sensitivity type and sensitivity level of the field; Based on the various anonymization algorithms set, a target algorithm pool is constructed, which includes multiple anonymization strategies. Each anonymization strategy includes: the anonymization algorithm corresponding to the field and / or the execution parameters of the anonymization algorithm. At least some fields in different anonymization strategies have different anonymization algorithms and / or different execution parameters. Based on the sensitivity type and sensitivity level of each field, as well as the original data, determine the privacy protection strength value and data availability index value of each anonymization strategy in the target algorithm pool; Based on the privacy protection strength value and data availability index value of each anonymization strategy, at least one candidate anonymization strategy is determined using the Pareto algorithm; and the target anonymization strategy is determined from the at least one candidate anonymization strategy. The original data is anonymized based on the target anonymization strategy to generate target data.
2. The method according to claim 1, characterized in that, After generating the target data, the method further includes: The privacy protection strength assessment index and data availability assessment index for this anonymization process are determined. The privacy protection strength assessment index includes at least one of the following: re-identification risk, information entropy, L-diversity achievement rate, and differential privacy budget consumption. The data availability assessment index includes at least one of the following: information loss, statistical error, and task utility index. Based on the privacy protection strength assessment index and the data availability assessment index, the assessment result of this anonymization process is determined; When the evaluation result is "evaluation passed", an evaluation report for this anonymization process will be generated. If the evaluation result is "not passed", the target algorithm pool is updated and the target anonymization strategy is redefined.
3. The method according to claim 1, characterized in that, The method further includes: In response to a data query request, the query data is determined from the target data, and the data usage scenario of the query data is determined. Based on the data usage scenario, determine the current privacy budget allocated to the data query request; Based on the current privacy budget, the query data is adjusted, and the adjusted query data is generated and fed back.
4. The method according to claim 3, characterized in that, The method further includes: Based on the identifier of the dataset to which the queried data belongs, determine the remaining privacy budget of the dataset; The step of adjusting the query data according to the current privacy budget, generating and returning the adjusted query data includes: When the remaining privacy budget is greater than or equal to the current privacy budget, the query data is adjusted according to the current privacy budget, and the adjusted query data is generated and fed back.
5. The method according to claim 1, characterized in that, The target anonymization strategy is multiple, and the target data includes: multiple anonymized data with different levels of anonymity generated after processing the original data using multiple target anonymization strategies respectively; The method further includes: Upon receiving a data retrieval request, determine the data user of the data retrieval request; Based on the authorization level of the data user, the target anonymous data matching the data user is determined from the various anonymous data, and the call data corresponding to the data call request is determined and returned from the target anonymous data.
6. The method according to claim 1, characterized in that, The method further includes: The system stores the target data, the original data, and the intermediate state data corresponding to the target data; wherein the intermediate state data is auxiliary data generated during the anonymization process and persistently stored, and the auxiliary data includes at least one of the following: semantic vector representation, generalization hierarchical structure, clustering grouping results, and noise seed; The system periodically detects the multiple data sources, and when new data is detected in the multiple data sources, it determines similar data corresponding to the new data from the original data. Based on the intermediate state data corresponding to the similar data, the newly added data is processed to generate processed data.
7. The method according to claim 1, characterized in that, The aforementioned multiple anonymization algorithms, based on the settings, construct a target algorithm pool comprising multiple anonymization strategies, including: Based on the various anonymization algorithms set, an initial population including multiple anonymization strategies after initialization is constructed; The initial population is used as the population to be processed. A genetic algorithm is used to screen, crossover, and mutate multiple anonymization strategies in the population to be processed to generate an updated population. Determine the updated diversity index values for the population; When the diversity index value is greater than the set diversity threshold, the updated population is treated as a population to be processed and returned to the process of processing using a genetic algorithm. When the diversity index value is less than or equal to the diversity threshold, the updated population is processed in multiple rounds using the differential evolution algorithm to generate a target algorithm pool that includes multiple anonymization strategies.
8. The method according to claim 1, characterized in that, The process of determining the sensitivity type and sensitivity level of each field based on its name and sample value in the original data includes: The name and sample value of each field in the original data are input into the trained field encoding model to generate the semantic vector representation of the field. Using a named entity recognition header and a conditional random field (CRF) module, the semantic vector representation of the field is processed, and the start-middle-not-entity BIO tag of each character is output; based on the BIO tags of each character, the sensitivity type of the field is determined. Based on the set sensitivity assessment dimensions, the sensitivity level of the field is determined; wherein the sensitivity assessment dimensions include: the inherent sensitivity of the entity type, the identifiability strength of the field, the risk of combined re-identification of fields, and the rule requirements used.
9. A data anonymization processing device, characterized in that, The device includes: The access module is used to acquire raw data from various access data sources; The first determining module is used to determine the sensitivity type and sensitivity level of each field based on the name and sample value of each field in the original data. The second determining module is used to construct a target algorithm pool including multiple anonymization strategies based on a set number of anonymization algorithms, wherein each anonymization strategy includes: an anonymization algorithm corresponding to a field and / or execution parameters of the anonymization algorithm, and at least some fields in different anonymization strategies have different anonymization algorithms and / or different execution parameters; The third determining module is used to determine the privacy protection strength value and data availability index value of each anonymization strategy in the target algorithm pool based on the sensitivity type, sensitivity level, and the original data corresponding to each field. The fourth determining module is used to determine at least one candidate anonymization strategy based on the privacy protection strength value and data availability index value of each anonymization strategy using the Pareto algorithm; and to determine the target anonymization strategy from the at least one candidate anonymization strategy. The generation module is used to anonymize the original data based on the target anonymization strategy to generate target data.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.
11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor performs the steps of the method according to any one of claims 1-8.
