Table data generalization method and device

Abstract

The present application discloses a table data generalization method and device, which are used to improve the efficiency of table data generalization. The method includes: determining various types of sub-data respectively included in multiple table data in the table to be generalized, wherein the types of sub-data include numerical type, first text type containing semantics, and second text type not containing semantics; determining the distance between the sub-data of various types included in any table data and each center point according to the distance parameter; wherein the similarity between the numerical sub-data and the numerical variable of each center point is its distance parameter; the similarity between the first text sub-data and the first text variable of each center point is its distance parameter; the distance parameter of the second text sub-data indicates whether it is the same as the second text variable of each center point; according to the distance between each table data and each center point, the multiple table data are divided into at least one category cluster; and at least one category cluster is respectively generalized using a set data generalization algorithm.

Description

Technical Field

[0001] The present application relates to the field of data security technology, and in particular to a table data generalization method and device. Background Art

[0002] In the related art, in order to ensure the security of data, data out of the production environment will be desensitized. For example, the data can be generalized, and the quasi-identifiers in the data can be generalized, that is, the content included in the data that is neither unique nor sensitive information is generalized, so as to achieve the purpose of protecting sensitive information in the data. At present, common data generalization algorithms include the k-anonymity algorithm and its improved version of the k-anonymity (K-Anonymity by Clustering in Attribute Hierarchies Algorithm, KACA) algorithm based on local generalization. Among them, the KACA algorithm first divides the data into several clusters based on the idea of ​​clustering, and makes the number of data tuples included in each cluster greater than or equal to the preset K value. Furthermore, different generalization values ​​are used for generalization of each cluster to achieve data generalization. Although the KACA algorithm can generalize data well and protect the privacy information in the data, when the data processed is relatively large, the efficiency of data generalization using the KACA algorithm will be reduced. Summary of the invention

[0003] In an exemplary embodiment of the present application, a table data generalization method and device are provided, which proposes clustering the table data before generalizing the table data, and taking into account the scenarios containing semantics during clustering, thereby improving the accuracy of clustering, thereby improving the efficiency of table data generalization and the availability of the generalized table data.

[0004] In a first aspect, an embodiment of the present application provides a table data generalization method, including:

[0005] Splitting the table to be generalized into multiple table data, and determining each type of sub-data included in each table data, wherein the types of the sub-data include a numerical type, a first text type containing semantics, and a second text type not containing semantics;

[0006] For any table data, the distance between the any table data and each center point is determined according to the distance parameters of each type of sub-data included in the table data, wherein the distance parameter of the numerical sub-data is the similarity between the numerical sub-data and the numerical variable included in each center point; the distance parameter of the first text sub-data is the similarity between the first text sub-data and the first text variable included in each center point; the distance parameter of the second text sub-data is used to indicate whether the second text sub-data is the same as the second text variable included in each center point;

[0007] Dividing the plurality of table data into at least one category cluster according to the determined distances between each table data and each center point;

[0008] For any category cluster, a set data generalization algorithm is used to generalize the any category cluster.

[0009] Based on the above scheme, the present application proposes clustering the table data before generalization, and then generalizing the data for each clustering result. This solves the problem of low generalization efficiency in existing generalization algorithms. In addition, when clustering, the scheme of the present application also takes into account the scenario where semantics exists. When clustering, the text data containing semantics and the text data not containing semantics are distinguished, and the distances are calculated separately, thereby improving the accuracy of the clustering results.

[0010] In some embodiments, after determining each type of sub-data included in each table of data, the method further includes:

[0011] The first text-type sub-data included in each table data is input into a pre-trained language model to obtain a semantic vector corresponding to the first text-type sub-data.

[0012] In some embodiments, for any table data, determining the distance parameters of each type of sub-data included in the table data specifically includes:

[0013] The Euclidean distance between the numerical sub-data and the numerical variables included in each center point is used as the distance parameter of the numerical sub-data;

[0014] Taking the Euclidean distance between the semantic vector and the first text-type variable included in each center point as the distance parameter of the first text-type sub-data;

[0015] The Hamming distance between the second text-type sub-data and the second text-type variables included in each center point is used as a distance parameter of the second text-type sub-data.

[0016] In some embodiments, for any table data, determining the distance between the any table data and each center point according to the distance parameters of each type of sub-data included in the table data includes:

[0017] The weighted sum of the distance parameters of each type of sub-data included in the arbitrary table data is used as the distance between the arbitrary table data and each center point.

[0018] In some embodiments, for any category cluster, generalizing the any category cluster using a set data generalization algorithm includes:

[0019] For any category cluster, according to the quasi-identifiers contained in each table data in the category cluster, the category cluster is divided into at least one equivalence class; wherein the similarity degree of the quasi-identifiers of any two table data included in the equivalence class is less than a set threshold;

[0020] The data included in any equivalence class is generalized using the generalization value corresponding to the equivalence class.

[0021] In some embodiments, according to the quasi-identifiers contained in each table data in any category cluster, the any category cluster is divided into at least one equivalence class, including:

[0022] Dividing any one of the category clusters into a plurality of data sets, wherein the quasi-identifiers of the table data included in each data set are the same;

[0023] Determine whether there is a target data set in the multiple data sets; wherein the target data set is a data set containing table data whose number is less than a preset value;

[0024] If the target data set does not exist in the multiple data sets, each data set is regarded as an equivalence class;

[0025] If the target data set exists in the multiple data sets, any one of the target data sets is selected, the any one of the target data sets and the data set closest to the any one of the target data sets are combined into a new set, and the process returns to the step of determining whether the target data set exists in the multiple data sets.

[0026] In some embodiments, the distance between data sets is determined by:

[0027] Determine the standard deviation corresponding to the first data set according to the sensitivity and privacy protection degree of each table data contained in the first data set, and determine the standard deviation corresponding to the second data set according to the sensitivity and privacy protection degree of each table data contained in the second data set; wherein the sensitivity and privacy protection degree of each table data is used to characterize the importance of the table data; the first data set is any one of the multiple data sets, and the second data set is any one of the multiple data sets except the first data set;

[0028] The difference between the standard deviation corresponding to the first data set and the standard deviation corresponding to the second data set is used as the distance between the first data set and the second data set.

[0029] In some embodiments, generalizing the data included in any equivalence class using the generalization value corresponding to the equivalence class includes:

[0030] The quasi-identifiers in each table data included in any equivalence class are replaced with the generalized values ​​corresponding to any equivalence class.

[0031] In a second aspect, an embodiment of the present application provides a generalization device for table data, including:

[0032] The clustering module is configured to perform:

[0033] Splitting the table to be generalized into multiple table data, and determining each type of sub-data included in each table data, wherein the types of the sub-data include a numerical type, a first text type containing semantics, and a second text type not containing semantics;

[0034] For any table data, the distance between the any table data and each center point is determined according to the distance parameters of each type of sub-data included in the table data, wherein the distance parameter of the numerical sub-data is the similarity between the numerical sub-data and the numerical variable included in each center point; the distance parameter of the first text sub-data is the similarity between the first text sub-data and the first text variable included in each center point; the distance parameter of the second text sub-data is used to indicate whether the second text sub-data is the same as the second text variable included in each center point;

[0035] Dividing the plurality of table data into at least one category cluster according to the determined distances between each table data and each center point;

[0036] The generalization module is used to generalize any category cluster by using a set data generalization algorithm.

[0037] In some embodiments, the clustering module is further configured to:

[0038] The first text-type sub-data included in each table data is input into a pre-trained language model to obtain a semantic vector corresponding to the first text-type sub-data.

[0039] In some embodiments, the clustering module is specifically used to:

[0040] The Euclidean distance between the numerical sub-data and the numerical variables included in each center point is used as the distance parameter of the numerical sub-data;

[0041] Taking the Euclidean distance between the semantic vector and the first text-type variable included in each center point as the distance parameter of the first text-type sub-data;

[0042] The Hamming distance between the second text-type sub-data and the second text-type variables included in each center point is used as a distance parameter of the second text-type sub-data.

[0043] In some embodiments, the clustering module is specifically used to:

[0044] The weighted sum of the distance parameters of each type of sub-data included in the arbitrary table data is used as the distance between the arbitrary table data and each center point.

[0045] In some embodiments, the generalization module is specifically used to:

[0046] For any category cluster, according to the quasi-identifiers contained in each table data in the category cluster, the category cluster is divided into at least one equivalence class; wherein the similarity degree of the quasi-identifiers of any two table data included in the equivalence class is less than a set threshold;

[0047] The data included in any equivalence class is generalized using the generalization value corresponding to the equivalence class.

[0048] In some embodiments, the generalization module is specifically used to:

[0049] Dividing any one of the category clusters into a plurality of data sets, wherein the quasi-identifiers of the table data included in each data set are the same;

[0050] Determine whether there is a target data set in the multiple data sets; wherein the target data set is a data set containing table data whose number is less than a preset value;

[0051] If the target data set does not exist in the multiple data sets, each data set is regarded as an equivalence class;

[0052] If the target data set exists in the multiple data sets, any one of the target data sets is selected, the any one of the target data sets and the data set closest to the any one of the target data sets are combined into a new set, and the process returns to the step of determining whether the target data set exists in the multiple data sets.

[0053] In some embodiments, the generalization module is further used to:

[0054] Determine the distance between data sets:

[0055] Determine the standard deviation corresponding to the first data set according to the sensitivity and privacy protection degree of each table data contained in the first data set, and determine the standard deviation corresponding to the second data set according to the sensitivity and privacy protection degree of each table data contained in the second data set; wherein the sensitivity and privacy protection degree of each table data is used to characterize the importance of the table data; the first data set is any one of the multiple data sets, and the second data set is any one of the multiple data sets except the first data set;

[0056] The difference between the standard deviation corresponding to the first data set and the standard deviation corresponding to the second data set is used as the distance between the first data set and the second data set.

[0057] In some embodiments, the generalization module is specifically used to:

[0058] The quasi-identifiers in each table data included in any equivalence class are replaced with the generalized values ​​corresponding to any equivalence class.

[0059] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a controller and a memory. The memory is used to store computer-executable instructions, and the controller executes the computer-executable instructions in the memory to use hardware resources in the controller to perform the operation steps of any possible implementation method of the first aspect.

[0060] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored, and when the computer-readable storage medium is run on a computer, the computer executes the methods in the above aspects.

[0061] In addition, the beneficial effects of the second to fourth aspects can refer to the beneficial effects described in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application.

[0063] Figure 1 A flow chart of a table data generalization method provided in an embodiment of the present application;

[0064] Figure 2 A flow chart of another table data generalization method provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of the structure of a table data generalization device provided in an embodiment of the present application;

[0066] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0068] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0069] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0070] In order to facilitate understanding of the table data generalization solution proposed in the embodiment of the present application, the technical terms involved in the present application are first introduced:

[0071] (1) Table data: data stored in the form of a table in a computer. Optionally, each row in the table can be a table data, or each column in the table can be a table data. As an example, please refer to Table 1 below.

[0072] Table 1

[0073] Name gender age post code Purchase Preferences Alice male 24 100083 Fitness Equipment Bob male 23 100084 clothing Candy female 26 100102 Skin Care Davis female 27 100104 Cooking tools Edith male 36 102208 Fitness Equipment Frank male 36 102201 clothing Grace female 34 102218 books Holley female 33 102219 clothing

[0074] (2) Identifier: A type of data attribute, generally a unique identifier for an individual. For example, the name information in Table 1 is a quasi-identifier of the table data.

[0075] (3) Quasi-identifier: A type of data attribute that is not a unique identifier for an individual, but can help researchers manage related data. For example, the zip code, age, birthday, and other information in Table 1 are quasi-identifiers of the table data.

[0076] (4) Sensitive information: a type of data attribute that involves the privacy of the data. For example, data such as salary. For another example, the purchase preferences in Table 1 can also be regarded as sensitive information contained in the table data.

[0077] (5) Non-sensitive information: a type of data attribute that can be made public directly and does not involve data privacy protection. For example, information such as serial numbers.

[0078] (6) Data desensitization: A common privacy protection technology that refers to transforming sensitive information according to preset rules or transformation algorithms, thereby making personal identity information unrecognizable or directly hiding sensitive information included in the data.

[0079] (7) Data generalization: It is a method of data desensitization. It is an analysis process that abstracts and summarizes a large amount of task-related data in the database from relatively low-level concepts to higher-level concepts. For example, data A is 32 and data B is 37. When generalizing data A and data B, the generalization result can be: (30, 40).

[0080] (8) Information loss: refers to the metric of the quality loss or distortion of the new data obtained after a series of data desensitization operations. There are three granularity metrics for information distortion: information loss of each quasi-identifier, information loss of each table data (or each tuple), and information loss of the entire data table.

[0081] At present, with the development of big data, artificial intelligence and Internet of Things technology, how to effectively protect personal privacy and protect sensitive corporate data from being leaked has become the focus of more and more security vendors. The era of big data is the era of the Internet of Everything. The value of data is reflected at the three levels of the country, government and big data companies. Therefore, data is known as the "oil of the future". No country, enterprise, organization or individual in the world can escape and resist the historical development trend that data is becoming more and more, ubiquitous and flowing everywhere in life. At the same time, with the integration and development of high-tech information technology and the industrial field, the circulation and sharing of industrial data has been promoted, and people can more easily extract unknown, implicit and potentially valuable information from the data. However, industrial data is very sensitive. For example, if enterprise data or application data (Application, APP) is tampered with or stolen, this will cause the problem of privacy information leakage in the data release process, which will seriously threaten user security, enterprise security and even national security. During the data release process, privacy information leakage mainly refers to the leakage of sensitive information in the data. Traditional encryption technology and access restriction technology cannot effectively achieve the purpose of protecting individual information privacy. Therefore, in order to reduce the leakage of sensitive information during the data release process, data desensitization technology came into being. Data desensitization technology is an effective way to solve the above-mentioned security problems and risks.

[0082] Traditional data desensitization algorithms include the k-anonymity algorithm, the L-diversity algorithm improved based on the k-anonymity algorithm, and the KACA algorithm improved based on the k-anonymity algorithm. The basic idea of ​​these algorithms is to generalize the quasi-identifier of a single table data so that the table data is indistinguishable in the entire data set. Among them, the k-anonymity algorithm can reduce the possibility of re-identification attacks (that is, if you have background knowledge or other information sources, you can still reply or identify the table data processed by the k-anonymity algorithm). Based on the defects of the above-mentioned k-anonymity algorithm, the L-diversity algorithm is proposed. The L-diversity algorithm proposes the concept of sensitive information diversity. That is, for the sensitive data of the table data with the same quasi-identifier, diversity is required to ensure that the privacy information cannot be inferred through background knowledge and other methods. Although the L-diversity algorithm has made some contributions to protecting the diversity of sensitive information, it does not consider the semantics of sensitive attributes and the overall distribution of sensitive attributes. Based on this, the KACA algorithm based on clustering ideas is proposed in the relevant technology. The main method is to divide multiple table data in the data table into several clusters, so that the number of table data in each cluster is greater than or equal to the preset value, and then replace the table data contained in any cluster with the generalization value corresponding to the cluster. However, this method is not suitable for larger data tables. For the generalization of larger data tables, the calculation time of the KACA algorithm will increase, making the generalization efficiency lower.

[0083] In view of this, the embodiment of the present application proposes a table data generalization method and device. When generalizing the table data, the table data is first pre-classified using a clustering algorithm, and then the classification results are generalized one by one, so as to improve the efficiency of data generalization. In addition, the semantic scenes are considered during clustering, which also improves the accuracy of clustering.

[0084] To facilitate understanding of the table data generalization solution proposed in this application, see Figure 1 , is a flowchart of a method for generalizing table data provided in an embodiment of the present application. Optionally, the method flow can be executed by a hardware facility with computing functions such as a computer, a chip, a processor or a server. The method flow specifically includes:

[0085] 101 , split the table to be generalized into multiple table data, and determine each type of sub-data included in each table data.

[0086] The types of sub-data include a numerical type, a first text type containing semantics, and a second text type not containing semantics.

[0087] As an example, taking the above Table 1 as an example, Table 1 can be regarded as a table to be generalized, each row of data in Table 1 can be regarded as a table data (except the first row), the numerical sub-data can be the age or zip code in Table 1, the first text sub-data can be the purchase preference in Table 1, and the second text sub-data can be the name and gender in Table 1.

[0088] 102 . For any table data, determine the distance between any table data and each center point according to the distance parameters of each type of sub-data included in the table data.

[0089] Among them, the distance parameter of the numerical sub-data is the similarity between the numerical sub-data and the numerical variable included in each center point, the distance parameter of the first text sub-data is the similarity between the first text sub-data and the first text variable included in each center point, and the distance parameter of the second text sub-data is used to indicate whether the second text sub-data is the same as the second text variable included in each center point. Optionally, each center point can be pre-set or set according to the distribution of each type of sub-data.

[0090] For ease of description, the distance between the first table data and the first center point is taken as an example for introduction (the first table data is any one of the multiple table data contained in the table to be generalized, and the first center point is any one of the multiple center points). In some embodiments, the distance between the numerical sub-data contained in the first table data and the numerical variable contained in the first center point, the distance between the first text sub-data contained in the first table data and the first text variable contained in the first center point, and the distance between the second text sub-data contained in the first table data and the second text variable contained in the first center point can be calculated respectively. The distance between the first table data and the first center point is determined based on the weighted sum of these three distances. For example, the weighted sum can be used as the distance between the first table data and the first center point, or some algorithms can be used to process the weighted sum to obtain the distance between the first table data and the first center point.

[0091] 103 , dividing the plurality of table data into at least one category cluster according to the determined distances between each table data and each center point.

[0092] For example, after calculating the distance between each table data and each center point, it is found that data 1, data 3 and data 6 are closest to center point A, then data 1, data 3 and data 6 can be classified into the category cluster corresponding to center point A.

[0093] 104 , for any category cluster, generalize the category cluster using a set data generalization algorithm.

[0094] Optionally, the data generalization algorithm used in this application can be the k-anonymity algorithm, L-diversity algorithm or KACA algorithm introduced above. A KACA algorithm based on sensitive privacy protection can also be used, that is, a parameter is introduced on the basis of the KACA algorithm: sensitive privacy protection. As an optional method, the sensitive privacy protection can be set according to the importance of the data in each table.

[0095] Based on the above scheme, the present application proposes clustering the table data before generalization, and then generalizing the data for each clustering result. This solves the problem of low generalization efficiency in existing generalization algorithms. In addition, when clustering, the scheme of the present application also takes into account the scenario where semantics exists. When clustering, the text data containing semantics and the text data not containing semantics are distinguished, and the distances are calculated separately, thereby improving the accuracy of the clustering results.

[0096] Below, the process of clustering proposed in the present application is introduced in detail. In order to facilitate understanding of the clustering method proposed in the present application, the clustering process in the prior art is briefly introduced below. The currently commonly used clustering algorithm is the K-Prototypes algorithm, which is a combination of the K-Means algorithm and the K-Modes algorithm, and is an algorithm adopted for data of mixed attributes (that is, data that both contain numerical data and text data). When the algorithm is used, the numerical variables and text variables contained in the data are split apart, the distance between the variables between the data samples is calculated separately, and then the weighted sum of the two is used as the distance between the data sample and the center point, and finally the classification of the data sample is realized according to the distance.

[0097] The clustering method proposed in the embodiment of the present application is an improved algorithm based on the K-Prototypes algorithm, which proposes to obtain the numerical sub-data contained in the table data, the first text-type sub-data containing semantics, and the second text-type sub-data without semantics, and respectively calculate the distance parameters of these three types of sub-data, that is, the distances between these three types of sub-data and the center point. The weighted sum of the three distance parameters calculated is used as the distance between the table data and the center point. In some embodiments, in order to ensure that the distance parameters of each type of sub-data can be accurately calculated, the embodiment of the present application also proposes that after a table data is split into numerical sub-data, the first text-type sub-data and the second text-type sub-data, the first text-type sub-data containing semantics is converted into a semantic vector. As an optional method, the first text-type sub-data can be input into a pre-trained language model to obtain the semantic vector corresponding to the first text-type sub-data. Optionally, a pre-trained language model such as a BERT model or a Roberta model can be used. For example, taking the BERT model as an example, the output of the CLS layer of the model can be used as a semantic vector. Alternatively, the first text-type sub-data may be segmented, and a pre-trained BERT model may be used to obtain a vector of each word (token), and the semantic vector of the entire first text-type sub-data may be calculated by averaging or maximizing. As an example, the following formula (1) or formula (2) may be used to determine the semantic vector of the first text-type sub-data:

[0098]

[0099]

[0100] Among them, e sent～ is the semantic vector of the first text-type sub-data, is the vector of the i-th token.

[0101] Optionally, after the first text-type sub-data is converted into a semantic vector, the distance parameters of the semantic vector, the numerical sub-data, and the second text-type sub-data (that is, the distances of the three from each center point) can be calculated respectively. As a possible implementation method, the Euclidean distance between the numerical sub-data and the numerical variable included in each center point can be used as the distance parameter of the numerical sub-data; the Euclidean distance between the semantic vector and the first text-type variable included in each center point can be used as the distance parameter of the first text-type sub-data; and the Hamming distance between the second text-type sub-data and the second text-type sub-data included in each center point can be used as the distance parameter of the second text-type sub-data.

[0102] For ease of understanding, the clustering process proposed in this application is introduced below in conjunction with specific embodiments:

[0103] Step 1: Randomly select K center points.

[0104] Wherein, K ≥ 1. Optionally, the randomly selected K center points may also be referred to as initial center points. The initial center points may be manually configured or randomly generated by a computer. Optionally, different types of data tables may correspond to different initial center points.

[0105] Step 2: For any table data, calculate the distance between the table data and each center point.

[0106] For ease of description, the first table data and the first center point are used as an example for introduction. Optionally, the Euclidean distance between the numerical sub-data contained in the first table data and the numerical variable included in the first center point can be calculated. As an example, the Euclidean distance between the numerical sub-data contained in the first table data and the numerical variable included in the first center point can be calculated using the following formula (3):

[0107]

[0108] Where x1 is a numeric sub-data, y1 is a numeric variable, d1(x1,y1) is the Euclidean distance between the numeric sub-data and the numeric variable, n is the number of data included in the numeric sub-data (or, it can also be the number of data included in the numeric variable), x 1i is the ith data included in the numeric sub-data, y 1i The i-th data included in the numeric variable.

[0109] Further, the Euclidean distance between the first text sub-data included in the first table data and the first text variable included in the first center point can be calculated. As an example, the Euclidean distance between the numerical sub-data included in the first table data and the numerical variable included in the first center point can be calculated using the following formula (4):

[0110]

[0111] Wherein, x2 is the first text-type sub-data, y2 is the first text-type variable, d2(x2, y2) is the Euclidean distance between the first text-type sub-data and the first text-type variable, n is the number of data included in the first text-type sub-data (or, it can also be the number of data included in the first text-type variable), k is the dimension of the semantic vector corresponding to the first text-type sub-data, x 2ij is the j-th dimension value of the i-th data included in the first text-type sub-data, y 2ij The j-th dimension value of the i-th data included in the first text variable.

[0112] Furthermore, the Hamming distance between the second text-type sub-data included in the second table data and the second text-type variable included in the second center point can be calculated. As an example, the Hamming distance between the numerical sub-data included in the second table data and the numerical variable included in the second center point can be calculated using the following formulas (5) and (6):

[0113]

[0114]

[0115] Wherein, x3 is the second text-type sub-data, y3 is the second text-type variable, d3(x3,y3) is the Hamming distance between the second text-type sub-data and the second text-type variable, m-(n+1) is the number of data included in the second text-type sub-data (or, it can also be the number of data included in the second text-type variable), x 3i The i-th data included in the second text-type sub-data, y 3i The i-th data included in the second text variable.

[0116] Finally, the distance between the first table data and the first center point can be determined based on the three distances calculated above. For example, the following formula (7) can be used:

[0117] d(x,y)=λ1*d1(x1,y1)+λ2*d2(x2,y2)+λ3*d3(x3,y3); Formula (7)

[0118] Among them, d(x,y) is the distance between the first table data and the first center point, λ1 is the weight of the numerical sub-data contained in the first table data, d1(x1,y1) is the Euclidean distance between the numerical sub-data and the numerical variable, λ2 is the weight of the first text sub-data contained in the first table data, d2(x2,y2) is the Euclidean distance between the first text sub-data and the first text variable, λ3 is the weight of the second text sub-data contained in the first table data, and d3(x3,y3) is the Hamming distance between the second text sub-data and the second text variable.

[0119] Step 3: Divide the data in each table into multiple category clusters based on the distance between each table data and each center point.

[0120] Step 4: After the classification clusters are completed, update the center points.

[0121] As an optional method, for any category cluster, the mean of the numerical sub-data in each table data included in the category cluster can be used as the numerical variable of the new center point, the mean of the first text sub-data in each table data included in the category cluster can be used as the first text variable of the new center point, and the mode of the second text sub-data in the table data included in the category cluster can be used as the second text variable of the new center point.

[0122] Step 5: After updating the center point, repeat steps 2 to 4 until the category cluster to which the table data belongs does not change, and the multiple category clusters finally obtained are taken as clustering results.

[0123] The above describes the process of clustering table data. The following describes the process of generalizing data based on the clustering results. In order to facilitate understanding of the data generalization process proposed in the embodiment of the present application, we first briefly introduce the traditional data generalization technology. The commonly used data generalization algorithm is the KACA algorithm, or the KACA algorithm based on sensitive privacy protection. Among them, the specific steps of the KACA algorithm are:

[0124] Step 1: Get the table to be generalized and set the generalization parameter value K.

[0125] Step 2: Divide the table data included in the table to be generalized into multiple data sets, and the quasi-identifiers of the table data included in each data set are the same.

[0126] Step 3: Loop through the following steps (1) to (4) until there is no data set containing table data with a number less than K.

[0127] (1) Randomly select a data set containing table data less than K.

[0128] For the convenience of description, this data set is referred to as data set A.

[0129] (2) Calculate the distance between data set A and all other data sets.

[0130] As an example, the distance between data set A and data set B is calculated as an example. Optionally, the distance between data set A and data set B can be calculated using the following formulas (8) to (10):

[0131]

[0132]

[0133]

[0134] Where WHD(p,q) is the distance from weighted level p to weighted level q (used to characterize the distance between different generalization levels), w is the generalization weight, t is a table data in the data set A, t ～ is a table data in data set B, D(t,t ～ ) is t and t ～ The distance between i is the table data before generalization, is the generalized table data, level(v i ) is v i The level of the generalization domain, for The level of the generalization domain, d is the data set A, d ～ is the data set B, D(d,d ～ ) is the distance between data set A and data set B.

[0135] (3) After calculating the distances between data set A and all other data sets, determine the data set that is closest to data set A.

[0136] For the convenience of description, the data set closest to data set A is referred to as data set C.

[0137] (4) Combine data set A and data set C into a new data set.

[0138] Step 4: Treat each data set as an equivalence class, and use the generalization value corresponding to any equivalence class to perform data generalization processing on the equivalence class.

[0139] Optionally, in order to better protect the sensitive information contained in the table data, in some embodiments, a data generalization method for setting sensitive privacy protection levels is also proposed. Each table data corresponds to a sensitive privacy protection level, and the specific value of the sensitive privacy protection level can be set according to the importance of the sensitive information contained in the table data. In some embodiments, after adding the sensitive privacy protection level, the following method can also be used to determine the distance between two data sets: the standard deviation corresponding to a data set can be calculated based on the sensitive privacy protection level of each table data contained in a data set. The difference in the standard deviations corresponding to any two data sets can be used as the distance between the two data sets. For ease of understanding, the data generalization method proposed in this application is introduced in conjunction with specific embodiments below:

[0140] Step 1: Get the table to be generalized and cluster the data in multiple tables contained therein.

[0141] The specific clustering process can be referred to the introduction in the above embodiment, which will not be described in detail here. After clustering, at least one category cluster can be obtained.

[0142] Step 2: For any category cluster, divide the table data contained in the category cluster into at least one data set.

[0143] The quasi-identifiers of the table data in each data set are the same.

[0144] Step 3: Execute the following loop steps (1) to (3) until there is no data set containing table data with a number less than the preset value.

[0145] (1) Select any data set that contains table data that is less than the preset value.

[0146] For the convenience of description, this data set is referred to as data set A.

[0147] (2) Calculate the distance between data set A and other data sets in the category cluster where data set A is located.

[0148] As an optional method, the distance between the data set A and other data sets may be determined according to the privacy protection sensitivity of the table data included in the data set.

[0149] For example, taking the calculation of the distance between data set A and data set D as an example, the distance between data set A and data set D can be calculated using the following formulas (11) to (13):

[0150]

[0151]

[0152] D i,j =|σ i -σ j |; Formula (13)

[0153] Among them, σ is the standard deviation corresponding to the data set A, N is the number of table data contained in the microdata set A, and x i is the sensitive privacy protection degree of the i-th table data contained in the data set A, μ is the average value of each table data contained in the data set A, D i,j is the distance between data set A and data set D.

[0154] (3) Determine the data set closest to data set A (referred to as data set E), and combine data set E and data set A to form a new data set.

[0155] Step 4: Treat each data set as an equivalence class, and use the generalized value corresponding to any equivalence class to replace the quasi-identifier of each table data contained in the equivalence class.

[0156] In order to further understand the solution proposed in this application, the following is an introduction in conjunction with specific embodiments. Figure 2 , is a flow chart of a table data generalization method provided in an embodiment of the present application, specifically comprising:

[0157] 201 , split each table data in the table to be generalized into numerical sub-data, first text sub-data and second text sub-data.

[0158] For the introduction of each type of sub-data, please refer to the introduction in the above embodiment, which will not be repeated here.

[0159] 202. For any table data, calculate distance parameters of various sub-data included in the table data.

[0160] The distance parameter is the distance between the sub-data and the center point. Specifically, the Euclidean distance between the numerical sub-data and the numerical variable of each center point can be calculated, the Euclidean distance between the first text sub-data and the first text variable of each center point can be calculated, and the Hamming distance between the second text sub-data and the second text variable of each center point can be calculated.

[0161] In some embodiments, before calculating the above distance parameters, the first text-type sub-data containing semantics may be first converted into a semantic vector. The specific conversion method can refer to the introduction in the above embodiments.

[0162] 203 . For any table data, a weighted sum of distance parameters of each type of sub-data included in the table data is used as the distance between the table data and each center point.

[0163] 204 , dividing the table data into at least one category cluster according to the distance between each table data and each center point.

[0164] 205 , for any category cluster, divide the table data included in the category cluster into multiple data sets.

[0165] The quasi-identifiers of the table data contained in each data set are the same.

[0166] 206 , determining whether a target data set exists in the multiple data sets.

[0167] The target data set is a data set containing table data whose quantity is less than a preset value.

[0168] If yes, continue to step 207.

[0169] If not, continue to step 208.

[0170] 207 : For any target data set, the target data set and the data set closest to it are combined into a new data set.

[0171] After forming a new data set, the process returns to step 206 .

[0172] 208 , taking each data set as an equivalence class, and for any equivalence class, generalizing each table data included in the equivalence class using the generalization value corresponding to the equivalence class.

[0173] Specifically, for any equivalence class, the quasi-identifier of each table data included in the equivalence class may be replaced with a generalized value corresponding to the equivalence class.

[0174] In some embodiments, after the generalization process, the information loss of the table data may be calculated to evaluate the availability of the generalized table data. Optionally, the following formulas (14) to (17) may be used to calculate the information loss:

[0175]

[0176]

[0177]

[0178]

[0179] Among them, IL num is the information loss of the numerical sub-data, is the i-th numerical sub-data after generalization, a i is the ith numerical sub-data before generalization, |v i | is the numeric domain of the numeric sub-data, IL cla (x) is the information loss of the i-th text type sub-data (including the first text type and the second text type), v i is the i-th text-type sub-data before generalization, is the i-th text-type sub-data after generalization, For v i and The subtree with the least common ancestor as the root, h represents the height of the tree, IL(t) is the information loss of a table data, and IL(D) is the information loss of a table.

[0180] Based on the same concept as the above method, see Figure 3, is a table data generalization device 300 provided in an embodiment of the present application. The device 300 is used to perform each step in the above method, and in order to avoid repetition, it will not be described again here. The device 300 includes: a clustering module 301 and a generalization module 302.

[0181] The clustering module 301 is configured to perform:

[0182] Splitting the table to be generalized into multiple table data, and determining each type of sub-data included in each table data, wherein the types of the sub-data include a numerical type, a first text type containing semantics, and a second text type not containing semantics;

[0183] For any table data, the distance between the any table data and each center point is determined according to the distance parameters of each type of sub-data included in the table data, wherein the distance parameter of the numerical sub-data is the similarity between the numerical sub-data and the numerical variable included in each center point; the distance parameter of the first text sub-data is the similarity between the first text sub-data and the first text variable included in each center point; the distance parameter of the second text sub-data is used to indicate whether the second text sub-data is the same as the second text variable included in each center point;

[0184] Dividing the plurality of table data into at least one category cluster according to the determined distances between each table data and each center point;

[0185] The generalization module 302 is used to generalize any category cluster using a set data generalization algorithm.

[0186] In some embodiments, the clustering module 301 is further used to:

[0187] The first text-type sub-data included in each table data is input into a pre-trained language model to obtain a semantic vector corresponding to the first text-type sub-data.

[0188] In some embodiments, the clustering module 301 is specifically used to:

[0189] The Euclidean distance between the numerical sub-data and the numerical variables included in each center point is used as the distance parameter of the numerical sub-data;

[0190] Taking the Euclidean distance between the semantic vector and the first text-type variable included in each center point as the distance parameter of the first text-type sub-data;

[0191] The Hamming distance between the second text-type sub-data and the second text-type variables included in each center point is used as a distance parameter of the second text-type sub-data.

[0192] In some embodiments, the clustering module 301 is specifically used to:

[0193] The weighted sum of the distance parameters of each type of sub-data included in the arbitrary table data is used as the distance between the arbitrary table data and each center point.

[0194] In some embodiments, the generalization module 302 is specifically used to:

[0195] For any category cluster, according to the quasi-identifiers contained in each table data in the category cluster, the category cluster is divided into at least one equivalence class; wherein the similarity degree of the quasi-identifiers of any two table data included in the equivalence class is less than a set threshold;

[0196] The data included in any equivalence class is generalized using the generalization value corresponding to the equivalence class.

[0197] In some embodiments, the generalization module 302 is specifically used to:

[0198] Dividing any one of the category clusters into a plurality of data sets, wherein the quasi-identifiers of the table data included in each data set are the same;

[0199] Determine whether there is a target data set in the multiple data sets; wherein the target data set is a data set containing table data whose number is less than a preset value;

[0200] If the target data set does not exist in the multiple data sets, each data set is regarded as an equivalence class;

[0201] If the target data set exists in the multiple data sets, any one of the target data sets is selected, the any one of the target data sets and the data set closest to the any one of the target data sets are combined into a new set, and the process returns to the step of determining whether the target data set exists in the multiple data sets.

[0202] In some embodiments, the generalization module 302 is further used to:

[0203] Determine the distance between data sets:

[0204] Determine the standard deviation corresponding to the first data set according to the sensitivity and privacy protection degree of each table data contained in the first data set, and determine the standard deviation corresponding to the second data set according to the sensitivity and privacy protection degree of each table data contained in the second data set; wherein the sensitivity and privacy protection degree of each table data is used to characterize the importance of the table data; the first data set is any one of the multiple data sets, and the second data set is any one of the multiple data sets except the first data set;

[0205] The difference between the standard deviation corresponding to the first data set and the standard deviation corresponding to the second data set is used as the distance between the first data set and the second data set.

[0206] In some embodiments, the generalization module 302 is specifically used to:

[0207] The quasi-identifiers in each table data included in any equivalence class are replaced with the generalized values ​​corresponding to any equivalence class.

[0208] Figure 4 The electronic device 400 in the embodiment of the present application may further include a communication interface 403, such as a network port, through which the electronic device may transmit data, such as obtaining a table to be generalized from a database through the communication interface 403.

[0209] In the embodiment of the present application, the memory 402 stores instructions that can be executed by at least one controller 401. At least one controller 401 can be used to execute each step in the above method by executing the instructions stored in the memory 402. For example, the controller 401 can implement the above Figure 3 The functions of the clustering module 301 and the generalization module 302 in .

[0210] Among them, the controller 401 is the control center of the electronic device, which can use various interfaces and lines to connect various parts of the entire electronic device, by running or executing instructions stored in the memory 402 and calling data stored in the memory 402. Optionally, the controller 401 may include one or more processing units, and the controller 401 may integrate an application controller and a modem controller, wherein the application controller mainly processes the operating system and application programs, etc., and the modem controller mainly processes wireless communications. It is understandable that the above-mentioned modem controller may not be integrated into the controller 401. In some embodiments, the controller 401 and the memory 402 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0211] The controller 401 may be a general controller, such as a central processing unit (CPU), a digital signal controller, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general controller may be a microcontroller or any conventional controller, etc. The steps performed by the data statistics platform disclosed in the embodiments of the present application may be performed directly by a hardware controller, or performed by a combination of hardware and software modules in the controller.

[0212] The memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 402 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 402 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0213] By designing and programming the controller 401, for example, the code corresponding to the training method of the neural network model introduced in the aforementioned embodiment can be solidified into the chip, so that the chip can execute the steps of the aforementioned neural network model training method during operation. How to design and program the controller 401 is a technology well known to those skilled in the art and will not be elaborated here.

[0214] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0215] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a controller of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the controller of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0216] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0217] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0218] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0219] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A table data generalization method, characterized in that: include: Splitting the table to be generalized into multiple table data, and determining each type of sub-data included in each table data, wherein the types of the sub-data include a numerical type, a first text type containing semantics, and a second text type not containing semantics; For any table data, the distance between the any table data and each center point is determined according to the distance parameters of each type of sub-data included in the table data, wherein the distance parameter of the numerical sub-data is the similarity between the numerical sub-data and the numerical variable included in each center point; the distance parameter of the first text sub-data is the similarity between the first text sub-data and the first text variable included in each center point; the distance parameter of the second text sub-data is used to indicate whether the second text sub-data is the same as the second text variable included in each center point; wherein the center point is a data point set based on a pre-set selection rule and used for cluster analysis; Dividing the plurality of table data into at least one category cluster according to the determined distances between each table data and each center point; For any category cluster, a set data generalization algorithm is used to generalize the any category cluster.

2. The method according to claim 1, characterized in that After determining each type of sub-data included in each table data, the method further includes: The first text-type sub-data included in each table data is input into a pre-trained language model to obtain a semantic vector corresponding to the first text-type sub-data.

3. The method according to claim 2, characterized in that For any table data, determining the distance parameters of each type of sub-data included in the table data specifically includes: The Euclidean distance between the numerical sub-data and the numerical variables included in each center point is used as the distance parameter of the numerical sub-data; Taking the Euclidean distance between the semantic vector and the first text-type variable included in each center point as the distance parameter of the first text-type sub-data; The Hamming distance between the second text-type sub-data and the second text-type variables included in each center point is used as a distance parameter of the second text-type sub-data.

4. The method according to any one of claims 1 to 3, characterized in that: The step of determining, for any table data, the distance between the any table data and each center point according to the distance parameters of each type of sub-data included in the table data, includes: The weighted sum of the distance parameters of each type of sub-data included in the arbitrary table data is used as the distance between the arbitrary table data and each center point.

5. The method according to any one of claims 1 to 3, characterized in that: For any category cluster, generalizing the any category cluster using a set data generalization algorithm includes: For any category cluster, according to the quasi-identifiers contained in each table data in the category cluster, the category cluster is divided into at least one equivalence class; wherein the similarity degree of the quasi-identifiers of any two table data included in the equivalence class is less than a set threshold; The data included in any equivalence class is generalized using the generalization value corresponding to the equivalence class.

6. The method according to claim 5, characterized in that According to the quasi-identifiers contained in each table data in any category cluster, the any category cluster is divided into at least one equivalence class, including: Dividing any one of the category clusters into a plurality of data sets, wherein the quasi-identifiers of the table data included in each data set are the same; Determine whether there is a target data set in the multiple data sets; wherein the target data set is a data set containing table data whose number is less than a preset value; If the target data set does not exist in the multiple data sets, each data set is regarded as an equivalence class; If the target data set exists in the multiple data sets, any one of the target data sets is selected, the any one of the target data sets and the data set closest to the any one of the target data sets are combined into a new set, and the process returns to the step of determining whether the target data set exists in the multiple data sets.

7. The method according to claim 6, characterized in that The distance between data sets is determined by: Determine the standard deviation corresponding to the first data set according to the sensitivity and privacy protection degree of each table data contained in the first data set, and determine the standard deviation corresponding to the second data set according to the sensitivity and privacy protection degree of each table data contained in the second data set; wherein the sensitivity and privacy protection degree of each table data is used to characterize the importance of the table data; the first data set is any one of the multiple data sets, and the second data set is any one of the multiple data sets except the first data set; The difference between the standard deviation corresponding to the first data set and the standard deviation corresponding to the second data set is used as the distance between the first data set and the second data set.

8. The method according to claim 5, characterized in that Using the generalization value corresponding to any equivalence class to generalize the data included in any equivalence class, including: The quasi-identifiers in each table data included in any equivalence class are replaced with the generalized values ​​corresponding to any equivalence class.

9. A table data generalization device, characterized in that: include: The clustering module is configured to perform: Splitting the table to be generalized into multiple table data, and determining each type of sub-data included in each table data, wherein the types of the sub-data include a numerical type, a first text type containing semantics, and a second text type not containing semantics; For any table data, the distance between the any table data and each center point is determined according to the distance parameters of each type of sub-data included in the table data, wherein the distance parameter of the numerical sub-data is the similarity between the numerical sub-data and the numerical variable included in each center point; the distance parameter of the first text sub-data is the similarity between the first text sub-data and the first text variable included in each center point; the distance parameter of the second text sub-data is used to indicate whether the second text sub-data is the same as the second text variable included in each center point; wherein the center point is a data point set based on a pre-set selection rule and used for cluster analysis; Dividing the plurality of table data into at least one category cluster according to the determined distances between each table data and each center point; The generalization module is used to generalize any category cluster by using a set data generalization algorithm.

10. An electronic device, characterized in that: The electronic device comprises a controller and a memory, The memory is used to store computer programs or instructions; The controller is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 8 is performed.

Images