Data intelligent processing method and device based on improved clustering analysis understanding algorithm
By improving cluster analysis understanding algorithms and privacy protection technology, the existing distributed experimental data processing problems are solved, and automated processing and efficient analysis are realized, and the accuracy and efficiency of data processing are improved.
Patent Information
- Application Number
- CN202411195948.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-29
- Publication Date
- 2024-12-27
AI Technical Summary
The existing distributed experimental data processing methods mainly rely on manual labor, time-consuming and labor-intensive, and have bias and accuracy problems, and cannot effectively process complex data, resulting in poor analysis and understanding.
Using intelligent processing methods based on improved cluster analysis understanding algorithms, including data cleaning, data conversion, data regulations, data preprocessing and privacy protection cluster analysis calculations, using the decision tree's CluDP algorithm and TCS algorithm for privacy protection and usability analysis .
Through automated processing, reduce errors in human subjective judgments, improve data processing efficiency and accuracy, reduce labor costs, and enhance the ability to analyze and understand complex data.
Smart Images
Figure CN119202758A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a data intelligent processing method and device based on an improved cluster analysis comprehension algorithm. Background Art
[0002] With the development of the times, system tests involve many professions and complex combat forces. System test conditions, test methods and the scope of professional knowledge involved in the test are becoming more and more complex and diverse. Distributed system test data are also becoming more and more complex and diverse. This complexity and diversity of system test data complicates the processing of distributed system test data, and the mutual verification of combat systems makes system test data processing particularly urgent. At present, large-scale system test data is difficult to process manually, so analysis and understanding algorithms have become the mainstream in the field of distributed test data collection and processing.
[0003] At present, the commonly used distributed test data processing methods are mainly manual, based on the characteristics of the specific test equipment, targeted based on prior knowledge to screen the obtained test data, perform data processing on the statistical objects, and combine the characteristics of the test equipment and the obtained data to manually adapt, classify and analyze and summarize. The manual distributed test data processing method is time-consuming and labor-intensive, and is easily dominated by human consciousness, and cannot guarantee accurate processing of test data and subsequent analysis and understanding.
[0004] like Figure 1 As shown in the figure, the process of distributed test data processing method based on manual operation is given. This technology is one of the most commonly used distributed test data processing methods. The method is people-oriented throughout the process. It processes data based on prior knowledge for parameter equipment, divides data into indicator data, numerical type data, other key data and irrelevant data, discards irrelevant data, and manually classifies it into data related to the test equipment.
[0005] The distributed test data processing method based on manual work requires human independent judgment throughout the entire process, which is time-consuming and labor-intensive, and increases labor costs. This method has a certain bias, and the processing results of the same test data vary from person to person, and it is impossible to complete it accurately. In addition, the analysis and understanding effect of data with complex data and unclear features is poor. Summary of the invention
[0006] In view of this, the present invention provides a test data intelligent processing method based on an improved cluster analysis understanding algorithm, the method comprising the following steps:
[0007] Intelligent processing steps for test data are used to clean, convert and reduce test data, remove non-critical data and reduce the amount of data;
[0008] The test data analysis and understanding step is used to perform data preprocessing and privacy-preserving cluster analysis and calculation on the test data in sequence; wherein the privacy-preserving cluster analysis and calculation step includes: performing privacy-preserving cluster analysis and calculation using a CluDP algorithm based on a decision tree or a TCS algorithm according to different data types;
[0009] The usability analysis step is used to perform usability analysis on the above privacy-preserving cluster analysis calculation results, and send the results of the usability analysis to data users.
[0010] In particular, the data reduction includes at least one of: merging attributes, stepwise forward selection, stepwise backward deletion, decision tree induction and principal component analysis.
[0011] In particular, the data preprocessing of the test data in sequence specifically includes: first, the data user sends a clustering analysis request to the data owner; then, according to the data user's request, the data owner selects a clustering algorithm to process the original data set D and obtains a labeled data set D * , D * The attributes in the data record are represented by A * ={A1, ..., A d , Class}, where Class represents the attributes in each record, A1,…,A d Indicates that D * The class labels in , can be used to help identify the clustering structure of the original data set D during the anonymization process.
[0012] In particular, the privacy protection calculations using the decision tree-based CluDP algorithm or the TCS algorithm are performed according to different data types, including:
[0013] Performing privacy protection calculation on the preprocessed continuous test data using the CluDP algorithm based on the decision tree;
[0014] For the classified data in the test data formed by the continuous data discretization step during the above data conversion, or the existing classified data in the test data, the TCS algorithm is used to perform privacy protection calculations.
[0015] In particular, the decision tree-based CluDP algorithm includes:
[0016] First, initialize the root node of the classification tree and set the number of numeric attributes to d n ; Initialize the division value Divide0, which represents the initial division value set for the numeric attribute during initialization;
[0017] Then, the utility score of each candidate attribute is calculated and a split value Divide is selected that is proportional to its probability. i , Divide i Indicates the i-th partition value selected for a numeric attribute during the partition process; if Divide i If is a numeric attribute, a partition value is selected for it; a privacy budget of ε / 2 is used in the entire partition process, and the rest is used to partition leaf nodes; therefore, the privacy budget ε / 2 is evenly distributed to all steps of the partition process, and the privacy budgets allocated to these steps are set to ε′ represents the privacy budget allocated to each step of the partitioning process, and h represents the maximum height of the decision tree.
[0018] In particular, the TCS algorithm includes: first, through the label data set D * The attributes in are generalized according to the classification tree, all attributes are generalized to the initial root node, and then the child nodes are generalized separately; then, the information gain between any attribute and the class label is calculated:
[0019]
[0020] Among them, H g (D) represents the information entropy of the dataset D about the attribute g, and Ω(Class) represents all possible class values |D g | represents all the data about attribute g in the dataset D, Indicates that in the dataset D, attribute g
[0021] The number of data records with the value cls; calculate the available fraction of any attribute generalized to its child nodes:
[0022]
[0023] Where: q(g) represents the information gain value when attribute g is used as the partition attribute, H g (D) represents the information entropy of the data set D about the attribute g, which measures the degree of disorder of the attribute g; |D c | represents the number of data records in child node c, |D g | represents the number of data records of attribute g in data set D; H c (D) represents the information entropy of data set D about attribute c;
[0024] Children(g) represents the child nodes after data partitioning of attribute g; D g =∑ c∈Children(g) D c ; Calculate the utility score at each generalization operation and select the node to generalize downward based on the exponential mechanism.
[0025] In particular, after being processed by the TCS algorithm, the original data records are divided into different groups; the threshold of the noise size of each group is calculated to determine whether the group is retained; for the numerical attribute d n , by dynamically generating or expanding the corresponding classification tree to probabilistically process numerical attributes, calculate the utility score of each attribute value within the numerical partition range, and use the exponential mechanism to select an attribute value g as the probability of the numerical partition s:
[0026]
[0027] Among them, I(s) represents the set of attribute values within the partition p; dividevalue represents the selected partition attribute value; ε represents the privacy budget parameter of differential privacy.
[0028] In particular, the usability analysis of the above privacy-preserving cluster analysis calculation results includes:
[0029] Analyze the clustering accuracy, inter-cluster distance and cluster quantity indicators for the numerical data clustering results generated by the CluDP algorithm;
[0030] For the classified data protected by the TCS algorithm, the degree of information loss and data query availability indicators are evaluated.
[0031] The present invention also discloses an intelligent processing device for test data based on an improved cluster analysis and understanding algorithm, comprising:
[0032] The test data intelligent processing module is used to clean, convert and reduce the test data, remove non-critical data and reduce the data volume;
[0033] The test data analysis and understanding module is used to perform data preprocessing and privacy-preserving cluster analysis and calculation on the test data in sequence; wherein the privacy-preserving cluster analysis and calculation step comprises: performing privacy-preserving cluster analysis and calculation using the CluDP algorithm based on a decision tree or the TCS algorithm according to different data types;
[0034] The usability analysis module is used to perform usability analysis on the above privacy-preserving cluster analysis calculation results and send the results of the usability analysis to data users.
[0035] Beneficial effects:
[0036] Based on actual needs, this application designs an intelligent test data processing technology based on an improved cluster analysis and understanding algorithm, targeting the time-consuming and labor-intensive and poor analysis results of the most commonly used distributed test data processing methods. Using automated methods to process distributed test data saves time and effort and can reduce the errors caused by human subjective judgment.
[0037] This application divides the application process into two parts: the first is the test data intelligent processing stage, which performs data cleaning, data conversion and data reduction according to the characteristics of the test data, removes non-critical data and reduces the amount of data; the second is the test data analysis and understanding stage, which uses cluster analysis to understand the algorithm framework, pre-processes the data, analyzes data privacy, and performs availability analysis, and finally sends the results of the availability analysis to the data users. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is a schematic diagram of the most commonly used distributed test data processing method in the prior art;
[0039] Figure 2 Schematic diagram of the test data intelligent processing method of the present invention;
[0040] Figure 3 Schematic diagram of the intelligent processing process of test data in the present invention;
[0041] Figure 4 Schematic diagram of the framework of the privacy-preserving clustering analysis algorithm CluDP in the present invention. DETAILED DESCRIPTION
[0042] The present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0043] The present invention provides a method for intelligently processing test data based on an improved cluster analysis comprehension algorithm, the method comprising the following steps: Figure 2 Shown:
[0044] Intelligent processing steps for test data are used to clean, convert and reduce test data, remove non-critical data and reduce data volume;
[0045] 1. Intelligent processing technology of test data
[0046] The test data intelligent processing technology mainly includes data cleaning, data conversion and data reduction. The test data intelligent processing process is as follows: Figure 3 shown.
[0047] 1.1 Test data cleaning
[0048] Data cleaning mainly involves deleting irrelevant and duplicate data in the original data set, smoothing noise data, filtering out data that is irrelevant to the calculation of evaluation indicators, and processing missing values and outliers.
[0049] (1) Missing value processing
[0050] Statistically speaking, missing data may lead to biased estimates, so that the sample data cannot represent the population well. In reality, most data contain missing values, so it is important to know how to deal with missing values.
[0051] Generally speaking, the processing of missing values includes two steps, namely, the identification of missing data and the processing of missing values. After determining whether there are missing values, missing value processing is required. Commonly used methods include deletion, replacement, interpolation, etc. The detailed description is as follows:
[0052] 1) Deletion method
[0053] The deletion method is the simplest method for dealing with missing values. It can be divided into two types: deleting observed samples and deleting variables according to different angles of data processing.
[0054] 2) Replacement method
[0055] Variables can be divided into numerical and non-numerical types according to their attributes, and the two are handled differently:
[0056] If the variable containing the missing value is numerical, the missing value of the variable is generally replaced by the mean of the values of the variable in all other objects;
[0057] If it is a non-numeric variable, it will be replaced by the median or mode of all other valid observations of the variable.
[0058] 3) Interpolation method
[0059] Although the deletion method is simple and easy to implement, there will be a problem of information waste and the data structure will change, resulting in biased statistical results. The replacement method also has similar problems. When facing the problem of missing values, commonly used interpolation methods include regression interpolation and multiple interpolation. The regression interpolation method uses the regression model to take the variable that needs to be interpolated as the dependent variable and other related variables as independent variables. The value of the dependent variable is predicted by the regression function to fill the missing variable; the principle of the multiple interpolation method is to generate a complete set of data from a data set containing missing values, and perform it multiple times to generate a random sample of missing values.
[0060] (2) Outlier processing
[0061] Before processing outliers, it is necessary to identify the outliers. Generally, a univariate scatter plot or box plot can be used to achieve this goal.
[0062] In data preprocessing, whether to remove outliers depends on the specific situation, because some outliers may contain useful information. Common methods for outlier processing are shown in the following table.
[0063] Table 2.1 Common methods for dealing with outliers
[0064]
[0065]
[0066] It is simple and easy to directly delete records containing outliers, but the disadvantage is also obvious. When there are few observations, this deletion will result in insufficient sample size and may change the original distribution of the variable, resulting in inaccurate analysis results. The advantage of treating it as missing values is that the information of existing variables can be used to fill in outliers (missing values).
[0067] In many cases, it is necessary to first analyze the possible reasons for the occurrence of outliers, and then determine whether the outliers should be discarded. If the data is correct, mining modeling can be performed directly on the data set with outliers.
[0068] 1.2 Data Conversion
[0069] Data transformation mainly involves normalizing data, discretizing continuous variables, and constructing variable attributes, so as to convert data into an "appropriate" form to meet the needs of evaluation calculations and algorithms.
[0070] (1) Data type conversion
[0071] The data field type in the database is defined as character type. Character type data cannot meet the evaluation and calculation requirements. If character type data needs to be converted to numeric type, conversion rules need to be defined to achieve the conversion from character type to numeric type.
[0072] (2) Simple function conversion
[0073] Simple function transformation is to perform some mathematical function transformation on the original data, commonly used ones include square, square root, logarithm, difference operation, etc., namely:
[0074] x′=x 2
[0075]
[0076] x=log(x)
[0077]
[0078] Simple function transformations are often used to transform data that does not have a normal distribution into data that has a normal distribution; in time series analysis, sometimes a simple logarithmic transformation or difference operation can convert a non-stationary series into a stationary series.
[0079] (3) Standardization
[0080] Data standardization (normalization) is a basic work of data mining. Different evaluation indicators often have different dimensions, and the difference between values may be large. Failure to process may affect the results of data analysis. In order to eliminate the impact of the differences in dimensions and value ranges between indicators, standardization is required to scale the data in proportion so that it falls into a specific area for comprehensive analysis.
[0081] Data normalization is particularly important for distance-based mining algorithms.
[0082] 1) Minimum and maximum normalization
[0083] Also known as deviation standardization, it is a linear transformation of the original data, mapping the values to between [0, 1].
[0084] The conversion formula is as follows:
[0085]
[0086] Among them, max is the maximum value of the sample data, and min is the minimum value of the sample data. max-min is the range. Deviation standardization retains the relationship existing in the original data and is the simplest way to eliminate the influence of dimension and data value range. The disadvantage of this processing method is that if a certain value in the value set is very large, the values will be close to 0 after normalization, and the difference will not be large. If it exceeds the current attribute value range in the future, it will cause a system error and min and max need to be re-determined.
[0087] 2) Zero-mean normalization
[0088] Also called standard deviation standardization, the mean of the processed data is 0 and the standard deviation is 1. The conversion formula is:
[0089]
[0090] in, is the mean of the original data, and σ is the standard deviation of the original data. This method is the most commonly used data normalization method, but the mean and standard deviation are greatly affected by outliers, so it is usually necessary to modify the above transformation. First, replace the mean with the median M, and then replace the standard deviation with the absolute standard deviation. W is the mean or median.
[0091] 3) Standardization of decimal calibration
[0092] The attribute value is mapped to the range [-1,1] by moving the decimal places of the attribute value. The number of decimal places moved depends on the maximum absolute value of the attribute value.
[0093] The conversion formula is:
[0094]
[0095] 4) Discretization of continuous attributes
[0096] Some data mining algorithms, especially some classification algorithms such as ID3 algorithm and Apriori algorithm, require data to be in the form of categorical attributes. In this way, it is often necessary to transform continuous attributes into categorical attributes, that is, to discretize continuous attributes.
[0097] (a) Discretization process
[0098] The discretization of continuous attributes is to set a number of discrete division points within the value range of the data, divide the value range into some discretized intervals, and finally use different symbols or integer values to represent the data values falling in each sub-interval. Therefore, discretization involves two subtasks: determining the number of categories and how to map continuous attribute values to these category values.
[0099] (b) Commonly used discretization methods
[0100] Commonly used discretization methods include equal-width method, equal-frequency method and (one-dimensional) clustering.
[0101] a) Equal-width method: Divide the attribute value range into intervals of the same width. The number of intervals is determined by the characteristics of the data itself or specified by the user, similar to making a frequency distribution table.
[0102] b) Equal frequency method: put the same number of records into each interval.
[0103] Both methods are simple and easy to operate, but they both require artificially specifying the number of intervals to be divided. At the same time, the disadvantage of the equal-width method is that it is sensitive to outliers and tends to distribute attribute values unevenly to each interval. Some intervals contain a lot of data, while other intervals have very little data, which will seriously damage the established decision model. Although the equal-frequency method avoids the above problems, it may divide the same data values into different intervals to meet the fixed number of data in each interval.
[0104] (c) Methods based on cluster analysis: The one-dimensional clustering method includes two steps. First, the continuous attribute values are clustered using a clustering algorithm (such as the K-Means algorithm). Then, the clusters obtained by clustering are processed and the continuous attribute values of a cluster are merged into the same label. The discretization method of cluster analysis also requires the user to specify the number of clusters, thereby determining the number of intervals generated.
[0105] 1.3 Data Specification
[0106] It will take a long time to perform complex data analysis and mining on a large data set. Data reduction can produce a new data set that is smaller but maintains the integrity of the original data. Evaluation calculations on the reduced data set will be more efficient.
[0107] The significance of data specification is:
[0108] (a) Reduce the impact of invalid and erroneous data on modeling and improve the accuracy of modeling;
[0109] (b) Small and representative data will significantly reduce the time required for data mining;
[0110] (c) Reduce the cost of storing data.
[0111] Attribute reduction creates new attribute dimensions by merging attributes, or directly reduces data dimensionality by deleting irrelevant attributes (dimensions), thereby improving the efficiency of data mining and reducing computational costs. The goal of attribute reduction is to find the smallest attribute subset and ensure that the probability distribution of the new data subset is as close as possible to the probability distribution of the original data set. Common methods of attribute reduction are shown in the following table.
[0112] Common methods for attribute reduction
[0113]
[0114]
[0115] Stepwise forward selection, stepwise backward deletion and decision tree induction are methods for directly deleting irrelevant attributes (dimensions). Principal component analysis is a data dimensionality reduction method for continuous attributes. It constructs an orthogonal transformation of the original data. The basis of the new space removes the correlation of the data under the basis of the original space. Only a few new variables are needed to explain most of the variation in the original data. In applications, several new variables that are less than the number of original variables and can explain most of the variables in the data, namely the so-called principal components, are usually selected to replace the original variables for modeling.
[0116] The calculation steps of principal component analysis are as follows:
[0117] (1) Assume that the observation data matrix of the original variable is:
[0118]
[0119] (2) The data matrix is center-normalized by column. For convenience, the normalized data matrix is still denoted as X.
[0120] (3) Find the correlation coefficient matrix R, R = (r ij ) p×p , r ij is defined as:
[0121]
[0122] Among them, r ij =r ji , r ii =1.
[0123] (4) Find the characteristic root λ1 of the characteristic equation of R, det(R-λE)=0.
[0124] (5) Determine the number of principal components m: α is determined according to the actual problem and is generally 80%.
[0125] (6) Calculate m corresponding unit eigenvectors:
[0126]
[0127] (7) Calculate the principal components:
[0128] Z i =β 1i X1+β 2i X2+…+β pi X p ,
[0129] i=1,2,...,m
[0130] 1.3 Data Specification
[0131] Data reduction reduces the amount of data by selecting alternative, smaller data. There are two types of methods: parametric and non-parametric. Parametric methods use a model to evaluate data, and only need to store parameters without storing actual data, such as regression (linear regression and multiple regression) and log-linear models. Non-parametric methods need to store actual data, such as histograms, clustering, and sampling.
[0132] 2 Steps to analyze and understand the experimental data
[0133] Since heterogeneous data contain numerical, categorical, and set-valued attributes, designing a privacy-preserving model for cluster analysis requires the model to satisfy differential privacy to maintain the availability of cluster analysis, while also solving the problem of processing efficiency for different data types. Therefore, a privacy-preserving heterogeneous data clustering analysis algorithm (CluDP) is adopted to achieve privacy protection of heterogeneous data for cluster analysis, which is used to solve the analysis and understanding of experimental data.
[0134] The test data analysis and understanding step is used to perform data preprocessing and privacy-preserving cluster analysis and calculation on the test data in sequence; wherein the privacy-preserving cluster analysis and calculation step includes: performing privacy-preserving cluster analysis and calculation using a CluDP algorithm based on a decision tree or a TCS algorithm according to different data types;
[0135] The data preprocessing of the test data in sequence specifically includes: first, the data user sends a clustering analysis request to the data owner; then, according to the data user's request, the data owner selects a clustering algorithm to process the original data set D and obtains a labeled data set D * , D * The attributes in the data record are represented by A * ={A1, ..., A d , Class}, where Class represents the attribute in each record, represented by A * ={A1, ..., A d , Class}, record r i In D * The class labels in , can be used to help identify the clustering structure of the original data set D during the anonymization process.
[0136] According to different data types, the privacy protection calculations are performed using the decision tree-based CluDP algorithm or the TCS algorithm, including:
[0137] Performing privacy protection calculation on the preprocessed continuous test data using the CluDP algorithm based on the decision tree;
[0138] The TCS algorithm is used to perform privacy protection calculations on the classified data formed by the continuous data discretization step during the above data conversion in the test data, or the classified data already in the test data. 1) Data preprocessing based on cluster analysis. First, the data owner performs clustering processing on the original data set D according to the cluster analysis request of the data user, and obtains a data set D containing cluster structures and class labels. * .
[0139] 2) Collaborative privacy protection based on anonymization and differential privacy. * Perform privacy protection processing to obtain an anonymous data set D′; finally, measure the privacy and availability of the privacy-preserving data set D′, and publish D′ that meets the privacy and availability requirements to data users.
[0140] Algorithm 1 Privacy-preserving clustering analysis algorithm CluDP, its framework is as follows Figure 4 As shown,
[0141] Input: Dataset D *, generalization number h, privacy budget ε
[0142] Output: Privacy-preserving dataset D
[0143] 1) Initialize each value in D as the root value of the classification tree. All root node values D = ∪Dicide0 = {all root node values};
[0144] 2) Calculate the privacy budget after partitioning
[0145] 3) According to the formula Calculate the utility score of each candidate attribute value g,
[0146] 4) for i = 1 to h do;
[0147] 5) Probability of selection and Proportional to g∈∪Divide i-1 ;
[0148] 6) if Divide i ∈d s then;
[0149] 7) Use the Laplace mechanism with a privacy budget of ε′ to calculate the noise size of each non-empty child node;
[0150] 8) For g noise >1 / ε attribute to generalize and update ∪Divide i ;
[0151] 9) if Divide i ∈d n then;
[0152] 10) According to the formula Select a split value for each numeric attribute;
[0153] 11) Update ∪ Divide i The utility score of each new partition in ;
[0154] 12)for do;
[0155] 13) Add a privacy budget to the number of data records in the node: Laplace noise;
[0156] 14) D′=D′∪{node in data record};
[0157] 15) Return the privacy-preserving dataset D′.
[0158] As shown in Algorithm 1, we first initialize the root node of the classification tree and set it as a numerical attribute d n Initialize the split value Divide0; then calculate the utility score of each candidate attribute and select the split value Divide proportional to its probability i ; If Divide i Is a set-valued attribute, verify that its child nodes are not empty; if Divide i If is a numeric attribute, a partition value is selected for it; the number of attributes may be different for different data records, so this difference is masked by adding noise to the number of records in each node. A privacy budget of ε / 2 is used throughout the partitioning process, and the remainder is used to partition leaf nodes. Therefore, the privacy budget ε / 2 is evenly distributed to all steps of the partitioning process, and the privacy budget allocated to these steps is set to
[0159] The CluDP algorithm based on decision tree includes first initializing the root node of the classification tree and setting the number of numerical attributes to d n ; Initialize the partition value Divide0, which represents the initial partition value set for the numeric attribute at initialization; then calculate the utility score of each candidate attribute and select the partition value Divide proportional to its probability i , Divide i Indicates the i-th division value selected for the numeric attribute during the division process; if Divide i Is a set-valued attribute, verify that its child nodes are not empty; if Divide i is a numeric attribute, a partition value is selected for it; the number of attributes may be different for different data records, so this difference is masked by adding noise to the number of records in each node; a privacy budget of ε / 2 is used in the entire partitioning process, and the rest is used to partition leaf nodes; therefore, the privacy budget ε / 2 is evenly distributed to all steps of the partitioning process, and the privacy budget allocated to these steps is set to ε represents the privacy budget allocated to each step of the partitioning process, and h represents the maximum height of the decision tree.
[0160] The TCS algorithm includes: first, through the label data set D * The attributes in are generalized according to the classification tree, all attributes are generalized to the initial root node, and then the child nodes are generalized separately; then, the information gain between any attribute and the class label is calculated:
[0161]
[0162] Among them, H g (D) represents the information entropy of the dataset D about the attribute g, and Ω(Class) represents all possible class values |Dg | represents all the data about attribute g in the dataset D, Indicates that in the dataset D, attribute g
[0163] The number of data records with the value cls; calculate the available fraction of any attribute generalized to its child nodes:
[0164]
[0165] Where: q(g) represents the information gain value when attribute g is used as the partition attribute, H g (D) represents the information entropy of the data set D about the attribute g, which measures the degree of disorder of the attribute g; |D c | represents the number of data records in child node c, |D g | represents the number of data records of attribute g in data set D; H c (D) represents the information entropy of data set D about attribute c;
[0166] Children(g) represents the child nodes after data partitioning of attribute g; D g =∑ c∈Children(g) D c ; Calculate the utility score at each generalization operation and select the node to generalize downward based on the exponential mechanism.
[0167] After being processed by the TCS algorithm, the original data records are divided into different groups. The categorical attributes are further generalized. Since the categorical attributes have a predefined classification tree and a fixed division strategy, the global sensitivity of the categorical attributes is 1. Compared with the categorical attributes, the global sensitivity of the set-valued attributes d s The difference in generalization processing lies in the combination of child nodes. Assuming that p set-valued attribute nodes are selected, and the corresponding classification tree has t child nodes, generalization will produce (2 t -1) groups. Since differential privacy requires uncertainty, the threshold of the noise size of each group is calculated to determine whether the group is retained. Therefore, in order to improve the efficiency of CluDP, groups with noise sizes smaller than the threshold should be pruned. For numerical attributes d n , by dynamically generating or expanding the corresponding classification tree to probabilistically process numerical attributes, calculate the utility score of each attribute value within the numerical partition range, and use the exponential mechanism to select an attribute value g as the probability of the numerical partition s:
[0168]
[0169] Where I(s) represents the set of attribute values within the partition p; dividevalue represents the selected partition attribute value; ε Denotes the privacy budget parameter for differential privacy.
[0170] The usability analysis of the above privacy-preserving cluster analysis calculation results includes:
[0171] The usability analysis step is used to perform usability analysis on the above privacy-preserving cluster analysis calculation results, and send the results of the usability analysis to data users.
[0172] Analyze the clustering accuracy, inter-cluster distance and cluster quantity indicators of the numerical data clustering results generated by the CluDP algorithm;
[0173] For the classified data protected by the TCS algorithm, the degree of information loss and data query availability indicators are evaluated.
[0174] The present invention also discloses an intelligent processing device for test data based on an improved cluster analysis and understanding algorithm, comprising:
[0175] The test data intelligent processing module is used to clean, convert and reduce the test data, remove non-critical data and reduce the data volume;
[0176] The test data analysis and understanding module is used to perform data preprocessing and privacy-preserving cluster analysis and calculation on the test data in sequence; wherein the privacy-preserving cluster analysis and calculation step comprises: performing privacy-preserving cluster analysis and calculation using the CluDP algorithm based on a decision tree or the TCS algorithm according to different data types;
[0177] The usability analysis module is used to perform usability analysis on the above privacy-preserving cluster analysis calculation results and send the results of the usability analysis to data users.
[0178] It corresponds one to one with the steps in the method embodiment, so it will not be repeated here.
[0179] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
[0180] It is obvious to those skilled in the art that the embodiments of the present invention are not limited to the details of the above exemplary embodiments, and that the embodiments of the present invention can be implemented in other specific forms without departing from the spirit or basic features of the embodiments of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the embodiments of the present invention is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the embodiments of the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units, modules or devices stated in the system, device or terminal claims can also be implemented by the same unit, module or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.
[0181] Finally, it should be noted that the above implementation modes are only used to illustrate the technical solutions of the embodiments of the present invention and are not intended to limit them. Although the embodiments of the present invention have been described in detail with reference to the above preferred implementation modes, those skilled in the art should understand that the technical solutions of the embodiments of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A data intelligent processing method based on an improved cluster analysis understanding algorithm, the method comprising the following steps: The test data intelligent processing steps are used to clean, convert and reduce the test data, remove non-critical data and reduce the data volume; The test data analysis and understanding step is used to perform data preprocessing and privacy-preserving cluster analysis and calculation on the test data in sequence; wherein the privacy-preserving cluster analysis and calculation step includes: According to different data types, the CluDP algorithm based on decision tree or the TCS algorithm is used to perform privacy-preserving clustering analysis and calculation; The usability analysis step is used to perform usability analysis on the above privacy-preserving cluster analysis calculation results, and send the results of the usability analysis to data users.
2. The data intelligent processing method based on the improved cluster analysis and understanding algorithm according to claim 1, characterized in that: The data reduction includes at least one of: merging attributes, stepwise forward selection, stepwise backward deletion, decision tree induction and principal component analysis.
3. The data intelligent processing method based on the improved cluster analysis and understanding algorithm according to claim 2, characterized in that: The data preprocessing of the test data in sequence specifically includes: first, the data user sends a clustering analysis request to the data owner; then, according to the data user's request, the data owner selects a clustering algorithm to process the original data set D and obtains a labeled data set D * , D * The attributes in the data record are represented by A * ={A1, ..., A d , Class}, where Class represents the attributes in each record, A1,…,A d Indicates that D * The class labels in , can be used to help identify the clustering structure of the original data set D during the anonymization process.
4. The data intelligent processing method based on the improved cluster analysis and understanding algorithm according to claim 3, characterized in that: According to different data types, the privacy protection calculations are performed using the decision tree-based CluDP algorithm or the TCS algorithm, including: Performing privacy protection calculation on the preprocessed continuous test data using the CluDP algorithm based on the decision tree; For the classified data in the test data formed by the continuous data discretization step during the above data conversion, or the existing classified data in the test data, the TCS algorithm is used to perform privacy protection calculations.
5. The data intelligent processing method based on the improved cluster analysis and understanding algorithm according to claim 4, characterized in that: The decision tree-based CluDP algorithm includes: First, initialize the root node of the classification tree and set the number of numeric attributes to d n ; Initialize the division value Divide0, which represents the initial division value set for the numeric attribute during initialization; Then, the utility score of each candidate attribute is calculated and a split value Divide is selected that is proportional to its probability. i , Divide i Indicates the i-th partition value selected for a numeric attribute during the partition process; if Divide i If is a numeric attribute, a partition value is selected for it; a privacy budget of ε / 2 is used in the entire partition process, and the rest is used to partition leaf nodes; therefore, the privacy budget ε / 2 is evenly distributed to all steps of the partition process, and the privacy budgets allocated to these steps are set to ε′ represents the privacy budget allocated to each step of the partitioning process, and h represents the maximum height of the decision tree.
6. The data intelligent processing method based on the improved cluster analysis and understanding algorithm according to claim 4, characterized in that: The TCS algorithm includes: First, through the label dataset D * The attributes in are generalized according to the classification tree, all attributes are generalized to the initial root node, and then the child nodes are generalized separately; Then, the information gain between any attribute and the class label is calculated: Among them, H g (D) represents the information entropy of the dataset D about the attribute g, and Ω(Class) represents all possible class values |D g | represents all the data about attribute g in the dataset D, Indicates the number of data records in dataset D whose attribute g takes the value of cls; calculates the available score of generalizing any attribute to its child nodes: Where: q(g) represents the information gain value when attribute g is used as the partition attribute, H g (D) represents the information entropy of the data set D about the attribute g, which measures the degree of disorder of the attribute g; |D c | represents the number of data records in child node c, |D g | represents the number of data records of attribute g in data set D; H c (D) represents the information entropy of data set D about attribute c; Children(g) represents the child nodes after data partitioning of attribute g; D g =∑ c∈Children(g) D c ; Calculate the utility score at each generalization operation and select the node to generalize downward based on the exponential mechanism.
7. The data intelligent processing method based on the improved cluster analysis and understanding algorithm according to claim 6, characterized in that: After being processed by the TCS algorithm, the original data records are divided into different groups. The threshold of the noise size of each group is calculated to determine whether the group is retained. For the numerical attribute d n , by dynamically generating or expanding the corresponding classification tree to probabilistically process numerical attributes, calculate the utility score of each attribute value within the numerical partition range, and use the exponential mechanism to select an attribute value g as the probability of the numerical partition s: Where I(s) represents the set of attribute values within the partition p; dividevalue represents the selected partition attribute value; ε Denotes the privacy budget parameter for differential privacy.
8. The data intelligent processing method based on the improved cluster analysis and understanding algorithm according to claim 6, characterized in that: The usability analysis of the above privacy-preserving cluster analysis calculation results includes: Analyze the clustering accuracy, inter-cluster distance and cluster quantity indicators of the numerical data clustering results generated by the CluDP algorithm; For the classified data protected by the TCS algorithm, the degree of information loss and data query availability indicators are evaluated.
9. An intelligent processing device for test data based on an improved cluster analysis and understanding algorithm, characterized in that: include: The test data intelligent processing module is used to clean, convert and reduce the test data, remove non-critical data and reduce the data volume; The test data analysis and understanding module is used to perform data preprocessing and privacy-preserving cluster analysis and calculation on the test data in sequence; wherein the privacy-preserving cluster analysis and calculation step comprises: performing privacy-preserving cluster analysis and calculation using the CluDP algorithm based on a decision tree or the TCS algorithm according to different data types; The usability analysis module is used to perform usability analysis on the above privacy-preserving cluster analysis calculation results and send the results of the usability analysis to data users.