A data de-identification method and system that can maintain desired data characteristics

Abstract

The application relates to the technical field of data processing, in particular to a data desensitization method and system capable of keeping expected data characteristics, which comprises the following steps: firstly, determining a key data column and a characteristic set of an original data set; secondly, performing standardization processing on the original data set; thirdly, dividing the original data set into a plurality of data clusters with similar characteristics by using a clustering analysis technology; fourthly, replacing data points in the data clusters; fifthly, calculating characteristic differences between the original data set and a desensitized data set; and finally, performing k-anonymity and l-diversity tests on the desensitized data set. While ensuring data privacy security, the application effectively keeps key characteristics of the original data, significantly improves the usability of the desensitized data, accurately controls desensitization intensity through a dynamic clustering and characteristic weighting mechanism, avoids data distortion problems caused by excessive desensitization in traditional methods, and ensures that the data meets strict privacy protection requirements through the k-anonymity and l-diversity tests.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data desensitization method and system that can maintain the expected characteristics of data. Background Technology

[0002] Power grid supply chain management relies on the effective use of supply chain data, which has become one of the core elements supporting enterprise operations. However, supply chain data contains a large amount of sensitive information, including important content such as trade secrets, personal information, and non-public information. Once this information is leaked, enterprises will face severe risks, including weakened business competitiveness, potential legal and regulatory risks, and significant reputational damage.

[0003] Especially in the power grid supply chain sector, as the digitalization level of the power grid supply chain deepens, data flow has become an indispensable and important way for supply chain collaboration. Digital transformation has generated massive amounts of data in all aspects of the power grid supply chain, such as material procurement, warehousing and logistics, production and manufacturing, operation and maintenance. This data flows efficiently among upstream and downstream enterprises in the supply chain, power grid operation departments, and third-party service organizations, greatly improving the transparency and collaboration efficiency of the supply chain. At the same time, the risk of sensitive information leakage is becoming increasingly prominent in the data flow, display and application stages. For example, un-anonymized supplier quotation information may reveal the company's procurement strategy, and unprocessed power grid equipment operation data may expose security vulnerabilities in critical infrastructure.

[0004] Currently, data anonymization technology has become an important means of protecting sensitive data. However, existing anonymization technologies show certain limitations when applied to complex and interconnected supply chain data. For example, some technologies may not provide sufficient anonymization strength and are easily cracked; others may have excessive anonymization strength, leading to data distortion and seriously affecting the effectiveness of subsequent data analysis and applications. More importantly, existing technologies often lack customized anonymization solutions for the characteristics of supply chain data, making it difficult to achieve an ideal balance between anonymization strength and fidelity. They also lack continuous management and monitoring mechanisms for the anonymization process.

[0005] More specifically, the limitations of traditional methods are particularly prominent in the area of ​​data anonymization in the power grid supply chain. Traditional data anonymization methods, such as simple replacement and masking, often struggle to precisely control the intensity of data anonymization. If the anonymization intensity is too high, while it can reduce the risk of sensitive information leakage, it may also excessively damage the inherent structure and usability of the data, resulting in the anonymized data being unable to effectively support the digital applications of the power grid supply chain, such as intelligent demand forecasting, refined operational optimization, and real-time risk warnings. Conversely, if the anonymization intensity is too low, it cannot fully eliminate sensitive information, and there is still a risk of improper leakage of data during the flow and display application stages, failing to meet the stringent requirements for data security and compliance in the power grid supply chain.

[0006] Therefore, how to balance the strength of data anonymization with data availability while ensuring the secure flow of power grid supply chain data has become an urgent technical challenge. In particular, how to design a data anonymization method and system that can flexibly adjust the anonymization strategy according to the characteristics of power grid supply chain data and effectively evaluate the anonymization effect to meet the growing digital security needs of the power grid supply chain is the key issue that this invention focuses on and solves.

[0007] Therefore, in order to address the above problems, this invention proposes a data desensitization method and system that can maintain the expected data characteristics. Summary of the Invention

[0008] To overcome the problem that existing desensitization technologies cannot accurately control the intensity of data desensitization, which can easily lead to the loss of key features of the dataset after desensitization, thereby impairing the usability and value of the dataset, this invention proposes a data desensitization method and system that can maintain the expected data features.

[0009] The technical solution of the present invention is: a data desensitization method that can maintain the expected data characteristics, comprising: S1, determine the key data columns and their feature sets in the original dataset; S2, standardizes the original dataset; S3 uses cluster analysis to divide the original dataset into several data clusters with similar characteristics and replaces the data points within the data clusters; S4, calculate the feature differences between the original dataset and the anonymized dataset; S5, perform k-anonymity and l-diversity tests on the anonymized dataset.

[0010] Preferably, step S1 includes: The first step is to obtain the original dataset from the data source and calculate the statistical and structural characteristics of all data columns in the original dataset. The statistical characteristics include mean, sum, median, variance, standard deviation, minimum, maximum, quantile, skewness, kurtosis, distribution shape, and correlation between data columns. The structural characteristics include the relationships and organization of data columns in the dataset. The second step is to calculate the statistical and structural features of all data columns in the original dataset, and then select key data columns and the statistical or structural features to be retained for the key data columns according to the purpose of data desensitization and application scenario. The key data columns and their features are then placed into the feature set. The third step is to determine the weight of each feature in the feature set to be retained, based on the purpose of data anonymization and the application scenario.

[0011] Preferably, step S2 includes: The first step is to perform data standardization on non-critical data columns based on the already determined set of features to be retained. The second step is to process the key data column cases based on the already determined set of features to be retained.

[0012] As a preferred option, for the distribution pattern of the feature system to be retained in the key data column, a standardization algorithm that can preserve the distribution pattern is used to process the key data column. For the requirement of the feature coefficient value to be retained in the key data column, no standardization processing is required.

[0013] Preferably, step S3 includes: The first step is to calculate the silhouette coefficients corresponding to different K values ​​for the original dataset, and select the K value with the highest silhouette coefficient. The representative value is calculated by averaging. The second step is to use the already determined K value as a parameter for the key data column and perform clustering using the K-means++ algorithm to generate K data clusters. The representative value is calculated using the median. The third step is to calculate the representative value of the key data column for each data cluster, and replace the key data column of all data points in the data cluster with the representative value. The method for calculating the representative value is to add random noise to the average value.

[0014] Preferably, step S4 includes: The first step is to calculate the fidelity of a single feature, D(X,X'), as follows: D(X,X') = ‖Statistic(X) - Statistic(X')‖, where X is the original dataset and X' is the de-identified dataset; statistic(X) and statistic(X') are the calculation functions for the retained features. The second step is to calculate the overall feature fidelity as a weighted average of the fidelity of individual features. Third, if the overall feature fidelity is less than the preset threshold, the desensitized dataset is output directly; if the overall feature fidelity is greater than the preset threshold, step S3 is executed again, and different K values ​​are selected.

[0015] Preferably, step S5 includes: The first step is to determine the k and l parameters based on the different purposes and scenarios of desensitization; The second step is to test whether the anonymized dataset meets the requirements of k-anonymity and l-diversity. If it does not meet the requirements, return to step S3 to adjust the parameters; if it does meet the requirements, the anonymized dataset will be officially output.

[0016] As a preferred embodiment, a data anonymization system that can maintain the expected data characteristics includes: The feature extraction module is used to determine the key data columns and their feature sets in the original dataset; The standardization module is used to standardize the original dataset. The clustering desensitization module is used to generate desensitized datasets through cluster analysis; The fidelity assessment module is used to calculate feature differences and adjust desensitization parameters; The privacy testing module is used to verify whether the anonymized dataset meets the requirements of k-anonymity and l-diversity.

[0017] Preferably, the clustering desensitization module determines the optimal number of clusters K by using the silhouette coefficient, generates data clusters using the K-means++ algorithm, calculates representative values ​​to replace data within the clusters, and dynamically adjusts the clustering parameters based on the fidelity evaluation results.

[0018] Preferably, the privacy testing module allows users to customize the k-anonymity and l-diversity parameters according to their needs, and provides a visual report on the desensitization effect, showing the balance between fidelity and privacy protection indicators.

[0019] The beneficial effects of this invention are: 1. This invention effectively preserves the key features of the original data while ensuring data privacy and security, significantly improving the usability of the anonymized data. Through dynamic clustering and feature weighting mechanisms, this invention can precisely control the anonymization intensity, avoiding the data distortion problem caused by excessive anonymization in traditional methods. At the same time, k-anonymity and l-diversity tests ensure that the data meets strict privacy protection requirements.

[0020] 2. The systematic fidelity assessment and parameter adjustment mechanism makes this method particularly suitable for complex scenarios such as power grid supply chains. It can support data analysis needs while avoiding the risk of sensitive information leakage, providing a reliable solution for secure data flow in digital transformation. Attached Figure Description

[0021] Figure 1 The diagram shown illustrates the workflow of this invention. Figure 2 The diagram shown is a schematic representation of the system structure of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see Figure 1 The present invention provides an embodiment of a data desensitization method that can maintain the expected data characteristics, comprising: S1, determine the key data columns and their feature sets in the original dataset; S2, standardizes the original dataset; S3 uses cluster analysis to divide the original dataset into several data clusters with similar characteristics and replaces the data points within the data clusters; S4, calculate the feature differences between the original dataset and the anonymized dataset; S5, perform k-anonymity and l-diversity tests on the anonymized dataset.

[0024] Furthermore, the specific steps of the present invention are as follows: S1. First, it is necessary to identify the key data columns and their feature sets in the original dataset to ensure that the anonymized data can still retain the necessary statistical and structural characteristics. The original dataset is obtained from the data source, and the statistical and structural characteristics of all data columns are calculated. The characteristics include mean, variance, quantile, skewness, kurtosis, distribution pattern, etc. The structural characteristics include correlation between data columns, time series periodicity, network topology, etc. For example, in power grid supply chain data, the distribution pattern of supplier quotations and the time series pattern of purchase orders may be crucial for subsequent analysis.

[0025] Then, based on the purpose of de-identification and the application scenario, select the key data columns whose characteristics need to be retained. For example, if the de-identified data is to be used for demand forecasting, then the mean and trend characteristics of historical procurement data need to be retained; if it is to be used for supplier evaluation, then the distribution characteristics of price fluctuations need to be retained.

[0026] Finally, different features are assigned weights to reflect their importance in the desensitization process. For example, in the desensitization of financial data, the sum and mean may be more important than skewness, so they can be given higher weights.

[0027] S2. First, use conventional standardization methods for non-critical data columns, such as Z-score standardization and Min-Max normalization, to eliminate the influence of dimensions. For example, in the operation data of power grid equipment, numerical data such as temperature and voltage can be transformed into a uniform range through standardization.

[0028] If the key data column is to retain a distribution pattern, such as a normal distribution or a long-tailed distribution, then quantile normalization or logarithmic transformation is used to ensure that the data retains its original distribution characteristics after desensitization. If the key data column needs to retain numerical features such as sum or mean, then standardization is skipped and the subsequent clustering steps are directly entered.

[0029] S3, Cluster analysis is the core technology of this invention for achieving data anonymization. By grouping similar data and replacing them with representative values, it can protect privacy while retaining data characteristics. First, the silhouette coefficient is used to evaluate the clustering effect of different K values. The K value that maximizes the similarity within data clusters and the difference between clusters is selected. For example, in the historical transaction data of suppliers, K=5 may be better at distinguishing different procurement patterns than K=3. Then, the key data columns are clustered to generate K data clusters. For example, in the fault data of power grid equipment, clustering can group data with similar fault patterns into one group.

[0030] Among them, mean replacement replaces the original value with the mean of the data within the cluster, which is suitable for scenarios that need to preserve the overall trend; median replacement can enhance the robustness of data with outliers; and random noise replacement can enhance anonymity in scenarios that need to balance privacy and randomness.

[0031] S4. To ensure that the anonymized data meets the usability requirements, it is necessary to quantify the feature differences between the data and the original data and dynamically optimize the anonymization strategy. First, for each retained feature, such as mean and variance, calculate the difference between the original data and the anonymized data, for example, D = ||mean(X) - mean(X')||.

[0032] Then, calculate the weighted average fidelity based on the feature weights. If the result exceeds the threshold, return to step S3 to adjust the K value or replace the strategy.

[0033] Finally, the cluster size or representative value calculation method is adjusted iteratively until the fidelity is achieved. For example, in power grid load data, it may be necessary to try K=3 to K=6 multiple times to find the optimal balance point.

[0034] S5 verifies the anonymization effect through a privacy protection model to ensure that the data meets compliance requirements. First, based on data sensitivity and application scenarios, thresholds for k-anonymity (each record is indistinguishable from at least k-1 other records) and l-diversity (each sensitive attribute has at least l different values) are set. Then, it is checked whether the anonymized dataset meets the set parameters. If the test passes, the final anonymized data is output; if it fails, the process returns to step S3 to adjust the K value or replace the strategy until both privacy and usability meet the standards.

[0035] Please see Figure 2 A data anonymization system that can maintain the expected data characteristics includes: The feature extraction module is used to determine the key data columns and their feature sets in the original dataset; The standardization module is used to standardize the original dataset. The clustering desensitization module is used to generate desensitized datasets through cluster analysis; The fidelity assessment module is used to calculate feature differences and adjust desensitization parameters; The privacy testing module is used to verify whether the anonymized dataset meets the requirements of k-anonymity and l-diversity.

[0036] The clustering desensitization module determines the optimal number of clusters K using the silhouette coefficient, generates data clusters using the K-means++ algorithm, calculates representative values ​​to replace data within clusters, and dynamically adjusts clustering parameters based on the fidelity evaluation results.

[0037] The privacy testing module allows users to customize k-anonymity and l-diversity parameters according to their needs, and provides a visual report on the desensitization effect, showing the balance between fidelity and privacy protection indicators.

[0038] Furthermore, the modules work together to implement this method. The feature extraction module outputs a feature set to the standardization module, which then passes the preprocessed data to the clustering and desensitization module. The clustering and desensitization module generates desensitized data and sends it to the fidelity evaluation module. If the evaluation result fails, it is fed back to the clustering module for iterative optimization. If it passes, the desensitized dataset is output and sent to the privacy testing module. If the test result fails, it is fed back to the clustering module for iterative optimization. If it passes, the desensitized dataset is officially output.

[0039] Through the above steps, this invention effectively preserves the key features of the original data while ensuring data privacy and security, significantly improving the usability of the anonymized data. Through dynamic clustering and feature weighting mechanisms, this invention can precisely control the anonymization intensity, avoiding the data distortion problem caused by excessive anonymization in traditional methods. At the same time, through k-anonymity and l-diversity tests, it ensures that the data meets strict privacy protection requirements, thus solving the problem that existing anonymization technologies cannot accurately control the anonymization intensity, which easily leads to the loss of key features of the dataset after anonymization, thereby impairing the usability and value of the dataset.

Claims

1. A data anonymization method that can maintain the expected data characteristics, characterized in that, Including: S1, determine the key data columns and their feature sets in the original dataset; S2, standardizes the original dataset; S3 uses cluster analysis to divide the original dataset into several data clusters with similar characteristics and replaces the data points within the data clusters; S4, calculate the feature differences between the original dataset and the anonymized dataset; S5, perform k-anonymity and l-diversity tests on the anonymized dataset.

2. The data desensitization method according to claim 1, which can maintain the expected data characteristics, is characterized in that, Step S1 includes: The first step is to obtain the original dataset from the data source and calculate the statistical and structural characteristics of all data columns in the original dataset. The statistical characteristics include mean, sum, median, variance, standard deviation, minimum, maximum, quantile, skewness, kurtosis, distribution shape, and correlation between data columns. The structural characteristics include the relationships and organization of data columns in the dataset. The second step is to calculate the statistical and structural features of all data columns in the original dataset, and then select key data columns and the statistical or structural features to be retained for the key data columns according to the purpose of data desensitization and application scenario. The key data columns and their features are then placed into the feature set. The third step is to determine the weight of each feature in the feature set to be retained, based on the purpose of data anonymization and the application scenario.

3. The data desensitization method according to claim 1, which can maintain the expected data characteristics, is characterized in that, Step S2 includes: The first step is to perform data standardization on non-critical data columns based on the already determined set of features to be retained. The second step is to process the key data column cases based on the already determined set of features to be retained.

4. A data desensitization method that can maintain the expected data characteristics according to claim 3, characterized in that: For the distribution pattern of the features to be retained in the key data column, a standardization algorithm that can preserve the distribution pattern is used to process the key data column. For the requirement of the feature coefficients to be retained in the key data column, no standardization processing is performed.

5. A data desensitization method that maintains the expected data characteristics according to claim 1, characterized in that, Step S3 includes: The first step is to calculate the silhouette coefficients corresponding to different K values ​​for the original dataset, and select the K value with the highest silhouette coefficient. The representative value is calculated by averaging. The second step is to use the already determined K value as a parameter for the key data column and perform clustering using the K-means++ algorithm to generate K data clusters. The representative value is calculated using the median. The third step is to calculate the representative value of the key data column for each data cluster, and replace the key data column of all data points in the data cluster with the representative value. The method for calculating the representative value is to add random noise to the average value.

6. A data desensitization method that maintains the expected data characteristics according to claim 1, characterized in that, Step S4 includes: The first step is to calculate the fidelity of a single feature, D(X,X'), as follows: D(X,X') = ‖Statistic(X) - Statistic(X')‖, where X is the original dataset and X' is the de-identified dataset; statistic(X) and statistic(X') are the calculation functions for the retained features. The second step is to calculate the overall feature fidelity as a weighted average of the fidelity of individual features. Third, if the overall feature fidelity is less than the preset threshold, the desensitized dataset is output directly; if the overall feature fidelity is greater than the preset threshold, step S3 is executed again, and different K values ​​are selected.

7. A data desensitization method that maintains the expected data characteristics according to claim 1, characterized in that, Step S5 includes: The first step is to determine the k and l parameters based on the different purposes and scenarios of desensitization; The second step is to test whether the anonymized dataset meets the requirements of k-anonymity and l-diversity. If it does not meet the requirements, return to step S3 to adjust the parameters; if it does meet the requirements, the anonymized dataset will be officially output.

8. A data anonymization system that preserves expected data characteristics, employing the data anonymization method for preserving expected data characteristics as described in claims 1-7, characterized in that, Including: The feature extraction module is used to determine the key data columns and their feature sets in the original dataset; The standardization module is used to standardize the original dataset. The clustering desensitization module is used to generate desensitized datasets through cluster analysis; The fidelity assessment module is used to calculate feature differences and adjust desensitization parameters; The privacy testing module is used to verify whether the anonymized dataset meets the requirements of k-anonymity and l-diversity.

9. A data desensitization system that can maintain the expected data characteristics according to claim 8, characterized in that: The clustering desensitization module determines the optimal number of clusters K using the silhouette coefficient, generates data clusters using the K-means++ algorithm, calculates representative values ​​to replace data within clusters, and dynamically adjusts clustering parameters based on the fidelity evaluation results.

10. A data desensitization system that can maintain the expected data characteristics according to claim 8, characterized in that: The privacy testing module allows users to customize k-anonymity and l-diversity parameters according to their needs, and provides a visual report on the desensitization effect, showing the balance between fidelity and privacy protection indicators.

Images