Generalizing data records

Abstract

A mechanism for performing generalization on data records. The data records are clustered to produce one or more clusters of data records. In each cluster, any attribute of an object belonging to any data record in the cluster is identified. An iterative process is performed on a cluster of processing sets each containing identified attributes, where the number of attributes in the set is N for identifying any / all of the sets of attributes for which a data record between 0 to k has a binary value that affiliates all of the attributes of the set to the data record. In the clustering, binary data values for one or more attributes in each identified set of attributes are generalized. The iteration process is repeated for m times, and the value of N is increased by one every time iteration is carried out.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to generalization of data records. Background Art

[0002] As machine learning methods continue to gain popularity, there is an increasing demand for datasets of large amounts of data records used to train machine learning algorithms to perform specific tasks. However, typically each data record is unique or nearly unique to a specific individual or object, which means that data records for a specific individual or object may be identifiable, which violates data protection laws.

[0003] Therefore, methods of anonymizing and / or generalizing data records are needed to at least avoid such violations. It would therefore be advantageous to provide new methods of improved generalization or anonymization of data records. Summary of the invention

[0004] The invention is defined by the claims.

[0005] According to an example according to one aspect of the present invention, a computer-implemented method for generalizing a data record is provided.

[0006] The computer-implemented method includes: obtaining multiple binary data records, each binary data record includes multiple binary values, each binary value is associated with a different attribute of the object of the binary data record, wherein each binary value identifies whether the attribute belongs to the object of the binary data record; performing a clustering process on the multiple binary data records to generate at least one cluster of binary data records; and performing one or more iterations of a generalization process for each cluster of binary data records.

[0007] The generalization process includes identifying as identified attributes any attribute for which any data record in the cluster has a non-zero number of binary values ​​attributing the attribute to their corresponding binary data records; identifying any sets of attributes, wherein each set contains a different combination of N identified attributes; and performing a generalization evaluation process for each set of attributes.

[0008] The generalization evaluation process includes: a first step of identifying the number of binary data records that meet the conditions, wherein each binary data record that meets the conditions contains the following binary value, which attributes each attribute in the set of attributes to its corresponding binary record; and a second step, in response to the number of binary data records that meet the conditions being greater than 0 and less than a first predetermined number, generalizing the binary values ​​associated with the set of attributes in all binary data records in the cluster of binary data records.

[0009] The value of N starts at 1 and increases by 1 for each iteration of the generalization process.

[0010] The present invention provides a method for generating k m A technique for anonymizing a dataset that does not require a predefined generalization hierarchy and has minimal generalizations about the values ​​of the dataset. The proposed method utilizes clustering to group binary data records together, and then processes each binary data record cluster to generalize the binary data values, thereby allowing less than a first predetermined number (e.g., k) of data records to be specifically identified when searching for a particular combination of values ​​for an attribute.

[0011] The proposed method facilitates the generalization of data records without losing important information in the dataset containing the data records.

[0012] In some examples, obtaining the plurality of binary data records includes obtaining a plurality of data records, wherein each data record includes a plurality of non-binary values, each non-binary value identifying a different attribute of an object attributed to the data record; and converting each data record into a binary data record.

[0013] This facilitates the method for generalizing datasets that include data records with non-binary values ​​that identify attributes associated with the objects of the data records. This increases the usability and applicability of the proposed method to a wider range of datasets, particularly datasets containing categorical information.

[0014] In some examples, the step of converting each data record into a binary data record includes: identifying all attributes identified by any non-binary value of any data record in the plurality of data records as binary data record attributes; and for each data record, generating a binary value for each binary data record attribute, each generated binary value identifying whether the data record contains a non-binary value identifying that the object is associated with the binary data record attribute. This provides a technique for reliably and accurately converting non-binary values ​​into binary values.

[0015] In some examples, at least one cluster of binary data records contains fewer than a second predetermined number of clusters. The second predetermined number of clusters may be the total number of binary data records divided by the first predetermined number.

[0016] In some examples, performing the clustering process includes: clustering the plurality of binary data records using a first clustering algorithm; and combining any clusters of binary data records having fewer than a first predetermined number of binary data records with clusters of nearest neighbor binary data records.

[0017] In some examples, performing the clustering process includes: clustering the plurality of binary data records using a second clustering algorithm; and removing any cluster of binary data records having fewer than a first predetermined number of binary data records from at least one cluster of binary data records.

[0018] These methods avoid the number of binary data records in a single cluster of binary data records being less than a first predetermined number, which would otherwise result in all binary values ​​in such a cluster being generalized.

[0019] In some examples, generalizing the binary value includes marking the binary value as having an unknown value. This provides a reliable mechanism for generalizing the binary value.

[0020] In some examples, performing the generalization evaluation process for each set of attributes includes sequentially performing iterations of the generalization evaluation process for each set of attributes in turn.

[0021] In some examples, performing the generalization evaluation process for each set of attributes includes: performing a first step of each generalization evaluation process sequentially; and performing a second step of each generalization evaluation process in parallel with each other.

[0022] In some examples, for each identified cluster, after performing any generalization evaluation process, in response to the number of attributes associated with ungeneralized binary values ​​being less than two, performing one or more iterations of the generalization process is terminated.

[0023] The one or more iterations of the generalization process may include a third predetermined number of iterations. The third predetermined number of iterations may define the required k m - the value of m in the anonymized dataset. Alternatively, the third predetermined number of iterations may be defined as the required k m - The m-value in the anonymized dataset.

[0024] The method may also include outputting the plurality of binary data records after performing one or more iterations of a generalization process for each binary data record cluster.

[0025] The method may include, after performing one or more iterations of a generalization process on each binary data record cluster: for each binary data record, converting each binary data record into a corresponding data record containing one or more non-binary values ​​in the following manner: for each binary value that associates an attribute with an object of the binary data record, defining a non-binary value for the data record, wherein the non-binary value is used to identify the associated attribute with the data record; and for each generalized binary value, defining a non-binary value for the data record, wherein the non-binary value identifies an uncertain relationship between the object of the data record and the attribute associated with the generalized binary value.

[0026] Also proposed is a computer program product comprising computer program code means which, when executed on a computing device having a processing system, causes the processing system to perform all the steps of any method disclosed herein.

[0027] A processing system for generalizing data records is also provided. The processing system is configured to: obtain a plurality of binary data records, each binary data record comprising a plurality of binary values, each binary value being associated with a different attribute of an object of the binary data record, wherein each binary value identifies whether the attribute belongs to the object of the binary data record; perform a clustering process on the plurality of binary data records to generate at least one cluster of binary data records; and perform one or more iterations of a generalization process for each cluster of binary data records.

[0028] The generalization process includes identifying as identified attributes any attribute for which any data record in the cluster has a non-zero number of binary values ​​attributing the attribute to their corresponding binary data records; identifying any sets of attributes, wherein each set contains a different combination of N identified attributes; and performing a generalization evaluation process for each set of attributes.

[0029] The generalization evaluation process includes: a first step of identifying the number of binary data records that meet the conditions, wherein each binary data record that meets the conditions contains the following binary value, which attributes each attribute in the set of attributes to its corresponding binary record; and a second step, in response to the number of binary data records that meet the conditions being greater than 0 and less than a first predetermined number, generalizing the binary values ​​associated with the set of attributes in all binary data records in the cluster of binary data records.

[0030] The value of N starts at 1 and increases by 1 for each iteration of the generalization process.

[0031] A person skilled in the art can readily adapt a processing system to perform the functions of any of the methods disclosed herein.

[0032] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] For a better understanding of the invention, and to show more clearly how it may be put into practice, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0034] Figure 1 The proposed method is illustrated;

[0035] Figure 2A first example of a clustering process is illustrated;

[0036] Figure 3 A second example of a clustering process is illustrated;

[0037] Figure 4 An example method for obtaining a plurality of binary data records is illustrated; and

[0038] Figure 5 Another proposed method is illustrated. DETAILED DESCRIPTION

[0039] The present invention will be described with reference to the accompanying drawings.

[0040] It should be understood that the detailed description and specific examples, although indicating exemplary embodiments of the device, system and method, are intended only for illustrative purposes and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the device, system and method of the present invention will be better understood based on the following description, the appended claims and the accompanying drawings. It should be understood that the drawings are only schematic and are not drawn to scale. It should also be understood that throughout the drawings, the same reference numerals are used to represent the same or similar parts.

[0041] The present invention provides a mechanism for performing generalization on data records. The data records are clustered to produce one or more clusters of data records. In each cluster, any attribute of an object that is attributed to any data record in the cluster is identified. An iterative process is performed on each cluster containing a processing set of identified attributes, wherein the number of attributes in the set is N, and any / all sets of attributes are identified for which data records between 0 and k have the following binary values ​​that attribute all attributes of the set to the data record. In the cluster, the binary data values ​​of one or more attributes in each identified attribute set are generalized. The iterative process is repeated m times, and the value of N is increased by one for each iteration.

[0042] The present disclosure relates to a method of generalizing (binary) data records.

[0043] More generally, the present disclosure provides a method based on k m - Anonymity model. In this model, a dataset D is considered k-thousandth if any attacker knows at most m values ​​of a record t∈D but cannot use this knowledge to identify fewer than k records in D. m -Anonymous. The present disclosure provides a method for implementing k for multiple (binary) data records. m Anonymization techniques, i.e. forming datasets of (binary) data records.

[0044] In the context of the present disclosure, a binary data value is typically a data value that can have one of two possible values. However, it is possible for a binary data value to take a third value to represent "uncertain", "null", "unknown", or "either", for example, if the binary data value has been generalized according to the proposed process. Configurations for such binary data values ​​are well known.

[0045] A binary data value will be associated with a binary variable or attribute. The alternative label for a binary data value is the binary data value, and the alternative label for a binary variable or attribute is the binary variable or attribute.

[0046] Each binary data record is associated with a different object (e.g., a different individual or human being). Each binary data record includes a plurality of binary values. Each binary value is associated with a different (potential) attribute of the binary data record object, indicating whether the attribute is associated with the object.

[0047] Thus, a data set of binary data records may contain Q binary data records. Each binary data record contains a binary value for each of the N attributes. The binary value (e.g., 0 or 1) indicates whether the corresponding (nth) attribute is associated with the object of the binary data record.

[0048] For example, a binary value of 1 may indicate that the corresponding attribute is associated with the object, while a binary value of 0 may indicate that the corresponding attribute is not associated with the object. Of course, this correspondence may be reversed in some variants (e.g., a binary value of 0 indicates that the corresponding attribute is to be associated with the object).

[0049] Table 1 shows a portion of a (general) plurality of binary data records. The binary values ​​shown in Table 1 are merely examples given for illustrative purposes.

[0050]

[0051] Table 1

[0052] In the following description, the proposed (general) method will be explained in the context of an example set of generalized binary data records. This is done to improve the understanding of the context. Table 2 illustrates an example set of (5) binary data records to be used.

[0053] As a working example, each "object" may represent a different student in an educational institution, and each "attribute" may represent a course (e.g., school subject) that the student may take (e.g., Chinese, Math, English, French, Physics, etc.). As another working example, each "object" may represent a shopper in a store, and each "attribute" may represent an item that the shopper purchased during a visit to the store (e.g., "milk", "cheese", "bread", etc.).

[0054] With respect to the example set of binary data records, a binary value of 1 indicates that the corresponding attribute is associated with the object of the data record containing the binary value, while a binary value of 0 indicates that the corresponding attribute is not associated with the object.

[0055]

[0056] Table 2

[0057] Figure 1 A flow chart illustrating a computer-implemented method 100 for generalizing a data record is provided. The computer-implemented method may be implemented by a processing system, suitable embodiments of which will be described below.

[0058] Method 100 includes a step 110 of obtaining a plurality of binary data records. For example, the plurality of binary data records may be obtained, retrieved, or otherwise received from an external processing system or memory, or may be generated by a processing system executing method 100. The content of the plurality of binary data records has been explained previously. Example techniques for generating binary data records will be provided later in this disclosure.

[0059] The plurality of binary data records may effectively form a binary data set.

[0060] The method 100 further includes step 120 of performing a clustering process on the plurality of binary data records to generate at least one binary data record cluster. The clustering process is used to group similar binary data records (eg, binary data records that share similar combinations of binary values ​​for different attributes) together.

[0061] Suitable clustering techniques for use with binary data records will be apparent to those skilled in the art. For example, some clustering techniques are capable of identifying similarities between binary variables, such as the k-modes algorithm (e.g., the kmodes function provided in the klaR package for R programming). The k-modes algorithm studies the asymptotic behavior of correlations, and the patterns give a characteristic description of the clusters.

[0062] Examples of suitable clustering algorithms include algorithms proposed in the following articles: Li, Tao. "A general model for clustering binary data." Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining. 2005; Ordonez, Carlos. "Clustering binary data streams with k-means." Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery. 2003; Li, Tao. "A unified view on clustering binary data." Machine Learning 62(2006): 199-215, etc.

[0063] Further examples of suitable clustering procedures are provided later in this disclosure.

[0064] Table 3 illustrates the results of performing a clustering process on the example set of binary data records listed first in Table 2. Object numbers 1, 3, and 4 are clustered into a first cluster, and object numbers 2 and 5 are clustered into a second cluster.

[0065]

[0066] Table 3

[0067] The method further includes, for each binary data record cluster, performing one or more iterations of the generalization process 130. The number of iterations may be predefined (eg, defined by a user or as a default number).

[0068] Specifically, the number of iterations may define the value of m, which is the maximum number of attributes of a single record for which a potential attacker cannot distinguish between fewer than k records. Thus, the number of iterations may be k m The value of m in the anonymous model. It should be understood that the value of m is predefined, for example, during the setup process or to represent k m The default value for anonymization.

[0069] The generalization process 130 includes a step 131 of identifying as identified attributes any attributes for which a non-zero number of binary values ​​for any data record in the cluster attributes the attributes to their corresponding binary data records.

[0070] Executing step 131 greatly reduces the workload of the subsequent generalization evaluation process.

[0071] For the example binary data record set, performing step 131 on the first cluster (at least during the first iteration of the generalization process) would identify attributes A, B, C, and E (but not D) as recognized attributes. For the example binary data record set, performing step 131 on the second cluster (at least during the first iteration of the generalization process) would identify attributes A, C, D (but not B or E) as recognized attributes.

[0072] The generalization process 130 also includes a step 132 of identifying any sets of attributes. Each set contains a different combination of N identified attributes. The value of N starts at 1 and increases by 1 for each iteration of the generalization process. Therefore, at least for the first iteration of the generalization process, the set of attributes may include or contain a single attribute.

[0073] Thus, for the first iteration of the generalization process, performing step 132 on the first cluster will identify four sets of attributes, each set of attributes containing a different identified attribute. Similarly, for the first iteration of the generalization process, performing step 132 on the second cluster will identify three sets of attributes, each set of attributes containing a single different identified attribute.

[0074] The generalization process 130 then performs a generalization evaluation process 135 on each set of attributes identified in step 132 .

[0075] The generalization evaluation process includes a first step 136 of identifying the number of binary data records that satisfy the condition. A binary data record that satisfies the condition is a record that contains a binary value that attributes each set of attributes to its corresponding binary record.

[0076] Therefore, for the example set of binary data records (as shown in Tables 2 and 3), if the set of attributes identified for the first cluster in step 132 only includes attribute A, then the only binary data record in the first cluster that meets the conditions will be the binary data record of object 1.

[0077] For another example, for the example binary data record set (as shown in Tables 2 and 3), if the set of attributes identified for the first cluster in step 132 only includes attribute B, then the only binary data records that meet the conditions in the first cluster will be the binary data records of object 1 and object 4.

[0078] The generalization evaluation process 135 further includes a second step 137 of generalizing binary values associated with at least one of the sets of attributes among all the binary data records in the binary data record clustering in response to the number of binary data records satisfying the condition being greater than 0 and less than a first predetermined number.

[0079] In the context of the present invention, the step of generalization includes marking the binary data value as unknown (i.e., not 0 or 1), for example, changing the binary data value to an indeterminate value or a null value. For example, this may include simply deleting the binary data value (such that it becomes a "null" value). Other suitable marking methods are well known in the art.

[0080] For example, the first predetermined number may be set or defined by a user or individual and / or set or defined (e.g., predefined) for a particular use case scenario of the proposed method. More specifically, the first predetermined number may be k m The value of k in the anonymization model.

[0081] Step 137 may be subdivided into a determination sub-step 137A and a generalization sub-step 137B. The determination sub-step 137A determines the number N of binary data records satisfying the condition CM falls between 0 and the first predetermined number (k), i.e., 0 < N CM < k. The sub-step 137B may be executed in response to an affirmative determination in the sub-step 137A. The generalization sub-step 137B includes generalizing binary values associated with at least one (e.g., only one) of the sets of attributes of the instance undergoing the generalization evaluation process 135.

[0082] If multiple sets of attributes are identified in step 132, there are multiple ways to perform the generalization evaluation process 135 for each set of attributes identified in step 132.

[0083] In the first method, each iteration of the generalization process 135 is performed sequentially. Thus, the first step 136 and the second step 137 are performed for the first iteration and then repeated for the second iteration.

[0084] In the second method, the iterations of the generalization process are performed in parallel. For example, step 136 and the sub-step 137A may be performed for each set of attributes identified in step 132. Then, the sub-step 137B may be performed by generalizing the binary values associated with the most common attribute among all the sets of attributes associated with the numbers between 0 and the first determined number of binary data records satisfying the condition. If multiple attributes are the most common, any or a random one of the most common attributes may be selected as the most common attribute. Then, the generalization evaluation process may be repeated until no set of attributes is associated with the numbers between 0 and the first determined number of binary data records satisfying the condition.

[0085] As a working example of the second approach, consider a scenario where, during a particular iteration of the generalization process for the first cluster (identified in Table 3), the set of attributes identified in step 132 includes: (B, C); (B; E); and (C; E).

[0086] Iterations of steps 136 and 137A may be performed for each of these sets of attributes. In this example, if the first predetermined number is k=3, all three sets of attributes ((B, C); (B; E); and (C; E)) are identified as having 0 to the first predetermined number of data records that meet the condition.

[0087] Thus, the iteration of step 137B may be performed by generalizing in the first cluster a binary value associated with one of the attributes in the set of attributes having from 0 to a first predetermined number of data records satisfying the condition. The attribute may be the most common attribute in all of the sets of attributes identified as having from 0 to a first predetermined number of data records satisfying the condition. Since all three attributes (B, C, E) are equally common, one of the attributes (e.g., B) is (pseudo) randomly selected as the most common attribute.

[0088] Another iteration of steps 136 and 137A is then performed for each attribute set. In the scenario where attribute B is generalized, step 136 identifies only the attribute set (C, E) as having 0 to a first predetermined number of data records that satisfy the condition. This is because the binary values ​​of the binary data records in the first cluster associated with attribute B have been generalized, so the number of data records that satisfy the condition of attribute B is now 0.

[0089] Then, another iteration of step 137B is performed by generalizing in the first cluster the binary values ​​associated with one of the attributes in the set of attributes having from 0 to a first predetermined number of data records satisfying the condition (e.g., one of the attributes C and E). Since attributes C and E are equally common, one of the attributes (e.g., C) can be (pseudo-)randomly selected as the most common attribute.

[0090] Another iteration of steps 136 and 137A is then performed for each set of attributes. In the scenario where attributes B and C have been generalized (in the previous iteration), then no attribute is identified as having 0 to the first predetermined number of data records that meet the condition. The iterative repetition of the generalization evaluation process can then end.

[0091] To facilitate iterative execution of the generalization evaluation process 135 for each set of N attributes, the generalization evaluation process 135 may also include a step 139A of determining whether all sets of N attributes have been processed. In response to a positive determination, the iterative repetition of the generalization evaluation process 135 ends. In response to a negative determination, the next set of N attributes may be selected in step 139B. Alternatives or variations of these techniques will be apparent to those skilled in the art (e.g., if the second method outlined above is followed).

[0092] In order to facilitate performing one or more iterations (e.g., m iterations) of the generalization process 130, the generalization process may further include (after completing the iterations of the generalization process) determining whether the m iterations of the generalization process 130 have been performed for the cluster 140. In response to a positive determination, the iterative execution of the generalization process for the cluster may end. In response to a negative determination, the generalization process 130 is repeated (i.e., step 131 is repeated).

[0093] To facilitate iteratively performing the generalization process 130 for each cluster, the generalization process may further include (after performing m iterations of the generalization process for the cluster) step 151, determining whether all clusters identified in step 120 have been processed. In response to a positive determination, the iterative repetition of the generalization process ends, for example, at step 152. In response to a negative determination, the next cluster is identified (for example, in step 152), and the iteration of the generalization process 130 is repeated for the next cluster.

[0094] As mentioned before, m represents the maximum number of attributes of a single record for which a potential attacker cannot distinguish between fewer than k records. Therefore, the number of iterations can be k m The value of m in the anonymous model. It should be understood that the value of m is predefined, for example, during the setup process or to represent k m The default value for anonymization.

[0095] To better understand the context, the following provides complete working results of one or more iterations of the generalization process for the example set of binary data records listed in Table 3. In this scenario, the number of iterations of the generalization process 150 is 2, and the first predetermined number ((k)) is 2.

[0096] During the first iteration of the generalization process for the first cluster, attributes A, B, C, and E (but not D) are the identified attributes. Each identified attribute serves as a set of N identified attributes, where, for the first iteration, N=1. For the generalization evaluation process, the only identified attribute is attribute A, for which a binary data value (in the first cluster) between 0 and 2 (i.e., 1) associates the attribute with its corresponding binary data record. Therefore, the binary value associated with attribute A is generalized for each binary data record in the first cluster.

[0097] In the second iteration of the generalization process for the first cluster, attributes B, C, and E (but not A or D, which have been generalized) are the identified attributes. Therefore, the set of attributes includes: (B, C); (B, E); and (C, E) - where, for the second iteration, N = 2. For the generalization evaluation process, the only set of attributes is (B, E) for which between 0 and 2 (i.e., 1) binary data records contain binary data values ​​that attribute the set of attributes to the binary data record. Therefore, the binary value associated with one of the attributes B and E is generalized to each binary data record in the first cluster.

[0098] Thus, the result of performing two iterations of the generalization process on the first cluster is that the binary data values ​​associated with attribute A and associated with one of attributes B and E (in the first cluster) are generalized (eg, attribute B).

[0099] In the first iteration of the generalization process for the second cluster, attributes A, C, and D (but not B and E) are identified attributes. Each identified attribute serves as a set of N identified attributes, where, for the first iteration, N=1. For the generalization evaluation process, the only identified attribute is attribute C, for which a binary data value (in the second cluster) between 0 and 2 (i.e., 1) associates the attribute with its corresponding binary data record. Therefore, the binary value associated with attribute C is generalized to each binary data record in the second cluster.

[0100] In the second iteration of the generalization process for the second cluster, attributes A and D (but not C, B, or E, which have been generalized) are identified attributes. Therefore, the set (one or more) of attributes only includes: (A, D) - where, for the second iteration, N = 2. For the generalization evaluation process, there is no set of attributes for which 0 to 2 (i.e., 1) binary data records contain binary data values ​​that attribute the set of attributes to the binary data record. Therefore, no further generalization is required.

[0101] Therefore, the result of performing two iterations of the generalization process on the second cluster is that the binary data values ​​associated with attribute C (in the second cluster) are generalized.

[0102] Table 4 illustrates the results of the generalization process on a collection of binary data records. The symbol * represents a generalized binary data value, e.g., a binary data value that has been labeled as uncertain. It is clear from Table 4 that all binary data records belonging to the same cluster have the same set of binary values ​​for one or more attributes, e.g., labeled as uncertain.

[0103]

[0104] Table 4

[0105] It has been described above how to perform a clustering process on a plurality of binary data records to generate at least one binary data record cluster.

[0106] Preferably, the clustering process is configured such that at least one cluster of binary data records contains fewer than a second predetermined number of clusters. Specifically, the second predetermined number of clusters may be the total number of binary data records (Q) divided by the first predetermined number (k), i.e., Q / k. More specifically, at least one cluster may be the value of Q / k rounded to the nearest integer.

[0107] In a preferred example, each cluster contains at least a first predetermined number of binary data records. An example clustering method to achieve this goal is described below.

[0108] Figure 2 A first example of a clustering process 200 performed on a plurality of binary data records is illustrated.

[0109] The clustering process 200 includes a step 210 of clustering a plurality of binary data records using a first clustering algorithm. For example, the first clustering algorithm may be a k-modes algorithm. Examples of other suitable clustering algorithms have been described previously.

[0110] The clustering process also includes step 220, which combines any cluster of binary data records having less than a first predetermined number of binary data records with the cluster of the nearest neighbor binary data records. If necessary, step 220 can be iteratively repeated until each cluster of binary data records contains no less than the first predetermined number of binary data records.

[0111] This approach advantageously avoids the situation where a cluster is unlikely to contain less than a first predetermined number of binary data records, which would otherwise result in all binary data values ​​(attributing a particular attribute to a binary data record) being generalized.

[0112] Figure 3 A second example of a clustering process 300 performed on a plurality of binary data records is shown.

[0113] The clustering process 300 includes a step 310 of clustering the plurality of binary data records using a second clustering algorithm. For example, the second clustering algorithm may be a k-modes algorithm. Examples of other suitable clustering algorithms have been described previously.

[0114] The clustering process also includes step 320 of removing any cluster of binary data records having fewer than a first predetermined number of binary data records from at least one cluster of binary data records. This approach advantageously avoids the situation where a cluster cannot contain fewer than a first predetermined number of binary data records, which would otherwise result in all binary data values ​​being generalized.

[0115] Of course, in any of the above clustering methods, the total number of the plurality of binary data records should be greater than the first predetermined number.

[0116] The previously disclosed method provides a technique for generalizing data records. This includes the step of obtaining a plurality of binary data records. In some examples, the step includes generating a plurality of binary data records from a corresponding plurality of data records containing non-binary values.

[0117] Figure 4 A process 400 is shown for obtaining a plurality of binary data records that may be integrated into an embodiment. The process 400 may be performed or carried out by a suitably configured processing system.

[0118] Process 400 includes a step 410 of obtaining a plurality of data records, wherein each data record contains a plurality of non-binary values, each non-binary value identifying a different attribute attributed to an object of the data record.

[0119] For example, the plurality of data records may be obtained, retrieved, or received from an external processing system or memory.

[0120] Specifically, each data record may include one or more text values ​​for identifying attributes associated with the object of the data record. For example, each text value may be a textual representation of an attribute associated with the data record object. As a working example, if each data record is for a student in an educational institution, then each text value may be a textual representation of the school discipline / subject studied by the student (e.g., "Chinese", "Mathematics", "Physics", etc.).

[0121] Process 400 also includes step 420 of converting each data record into a binary data record. Step 420 can be performed by executing sub-step 421, identifying all attributes identified by any non-binary value of any of the plurality of data records as binary data record attributes, and executing sub-step 422, for each data record, generating a binary value for each binary data record attribute, each generated binary value identifying whether the data record contains a non-binary value identifying an object associated with the binary data record attribute.

[0122] Substep 422 effectively creates a binary value vector for each data record. Different positions within the vector will represent different attributes, so that the vector can be converted back into a collection of non-binary values ​​for the data record. The number of binary values ​​in each vector is equal to the number of binary data record attributes.

[0123] For each data record, the technique can efficiently convert a collection of text values ​​or other non-binary values ​​into a string or vector of binary values ​​- thereby forming multiple binary data records.

[0124] Figure 5 A method 500 according to another embodiment is illustrated.

[0125] The method 500 comprises performing the method 100 according to the previously described embodiments.

[0126] The method 500 also includes step 510 of, after performing one or more iterations of the generalization process on the clustering of each binary data record, converting each binary data record into a corresponding data record containing one or more non-binary values, each non-binary value identifying an attribute associated with an object of the data record that is the same as the object of the binary data record from which it is converted.

[0127] Specifically, step 510 may include, for each binary data record, generating non-binary values ​​only for those binary values ​​that associate the corresponding attribute with the object of the binary data record. This may be easily performed by assigning the label of the attribute associated with the appropriate binary value as a non-binary value.

[0128] If the position of each binary value in the binary data record corresponds to a known attribute, step 510 can be easily performed. In particular, non-binary values ​​can be generated by using predetermined (eg, textual) attribute tags.

[0129] Binary values ​​that have been generalized (e.g., marked as uncertain) may also be marked as uncertain after being converted to non-binary values ​​in the data record in step 510. For example, this may be accomplished by inserting an asterisk or other identifier after the non-binary value or using any other suitable marking technique (e.g., a separate marking variable for each non-binary value).

[0130] Thus, step 510 may include, after performing one or more iterations of the generalization process on each binary data record cluster: for each binary data record, converting each binary data record into a corresponding data record containing one or more non-binary values ​​in the following manner: for each binary value that associates an attribute with an object of the binary data record, defining a non-binary value for the data record, the non-binary value being used to identify the associated attribute using the data record; for each generalized binary value, defining a non-binary value for the data record, the non-binary value identifying an uncertain relationship between the object of the data record and the attribute associated with the generalized binary value.

[0131] Table 5 illustrates the set of data records generated by converting each binary data record into a corresponding data record. Each data record contains one or more non-binary values ​​that identify attributes associated with the object of the data record (via the corresponding binary value in the binary data record). Each data record also provides labeled non-binary values ​​for attributes associated with binary values ​​labeled as uncertain in the corresponding binary data record.

[0132] Specifically, Table 5 represents a set of data records generated by converting the binary data records (with uncertain values) in Table 4. In this case, each of the attributes A, B, C, D, and E is associated with a corresponding school subject / discipline, as follows: A = Chinese; B = Mathematics; C = English; D = French, and E = Physics.

[0133]

[0134] Table 5

[0135] When generating the data record, each data record may also optionally include an identifier of the object. This may be omitted in some variations to improve anonymity, for example, to reduce the risk of the object identifier being transmitted to an external party or a third party.

[0136] Method 500 may also include a step 520 of outputting all (converted) data records. It should be understood that, in practice, the output of step 510 need not be in the form of a table, but may be, for example, in the form of a data stream or other methods for defining a data set. As can be seen from Table 5 and the embodiments above, the present invention provides a generalization method without pre-defining a generalization level. The proposed method also maximizes the retention of the original information in the data set by retaining most of the values ​​in the data set with minimal generalization.

[0137] A skilled person will be able to easily develop a processing system for executing any of the methods described herein. Therefore, each step of the flowchart may represent a different action performed by a processing system and may be performed by a corresponding module of the processing system.

[0138] Thus, embodiments may utilize a processing system. A processing system may be implemented in a variety of ways using software and / or hardware to perform the various functions desired. A processor is an example of a processing system that employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform the desired functions. However, a processing system may be implemented with or without a processor, and may also be implemented as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuits) for performing other functions.

[0139] Examples of processing system components that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0140] In various implementations, a processor or processing system may be associated with one or more storage media, such as volatile and non-volatile computer memory, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that perform desired functions when run on one or more processors and / or processing systems. The various storage media may be fixed within the processor or processing system, or may be transportable so that one or more programs stored thereon may be loaded into the processor or processing system.

[0141] It should be understood that the disclosed method is preferably a computer-implemented method. Thus, the concept of a computer program is also proposed, which includes code units for implementing any described method when the program is run on a processing system (e.g., a computer). Therefore, different parts, lines or code blocks of a computer program according to an embodiment can be executed by a processing system or a computer to perform any method described herein.

[0142] Also proposed is a non-transitory storage medium storing or carrying a computer program or computer code which, when executed by a processing system, causes the processing system to perform any of the methods described herein.

[0143] In some alternative embodiments, the functions recorded in the block diagram(s) or flow chart(s) may not occur in the order shown in the drawings. For example, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functions involved.

[0144] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure and the claims.Although certain measures are recited in mutually different dependent claims, this does not indicate that a combination of these measures cannot be used to advantage.

[0145] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. If the term "suitable for" is used in the claims or in the description, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". If the term "arranged" is used in the claims or in the description, it should be noted that the term "arranged" is intended to be equivalent to the term "system" and vice versa.

[0146] A single processor or other unit may implement the functions of several items recited in the claims. If a computer program is described above, it may be stored / distributed on a suitable medium such as an optical storage medium or a solid-state medium provided together with other hardware or as part of other hardware, but may also be distributed in other forms such as via the Internet or other wired or wireless telecommunications systems.

[0147] Any reference signs in the claims should not be construed as limiting the scope.

[0148] Those skilled in the art can understand and implement variations to the disclosed embodiments when practicing the claimed invention by studying the drawings, the disclosure and the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the word "one" or "an" does not exclude a plurality.

[0149] The functions implemented by a processor may be implemented by a single processor or by multiple independent processing units, which may be collectively considered to constitute a “processor.” In some cases, these processing units may be remote from each other and communicate with each other by wire or wirelessly.

[0150] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0151] The computer program may be stored / distributed on suitable media such as optical storage media or solid-state media provided together with or as part of other hardware, but may also be distributed in other forms such as via the Internet or other wired or wireless telecommunication systems.

[0152] If the term "suitable for" is used in the claims or the description, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". If the term "arranged" is used in the claims or the description, it should be noted that the term "arranged" is intended to be equivalent to the term "system" and vice versa.

[0153] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method for generalizing a data record, the computer-implemented method comprising: Obtaining a plurality of binary data records, each binary data record comprising a plurality of binary values, each binary value being associated with a different attribute of an object of the binary data record, wherein each binary value identifies whether the attribute belongs to the object of the binary data record; performing a clustering process on the plurality of binary data records to produce at least one cluster of binary data records; and For each cluster of binary data records, one or more iterations of the generalization process are performed, including: identifying as an identified attribute any attribute for which a non-zero number of binary values ​​for any data record in the cluster attributes the attribute to their corresponding binary data record; any set of identified attributes, where each set contains a different combination of the N identified attributes; and A generalization evaluation process is performed for each set of attributes, the generalization evaluation process comprising: A first step is to identify the number of binary data records that meet the condition, wherein each binary data record that meets the condition contains the following binary value, which attributes each attribute in the set of attributes to its corresponding binary record; and In a second step, in response to the number of binary data records that meet the condition being greater than 0 and less than a first predetermined number, generalize the binary values ​​associated with the set of attributes in all binary data records in the cluster of binary data records, wherein the value of N starts from 1 and increases by 1 for each iteration of the generalization process.

2. The computer-implemented method of claim 1 , wherein: Obtaining the plurality of binary data records includes: obtaining a plurality of data records, wherein each data record comprises a plurality of non-binary values, each non-binary value identifying a different attribute of an object attributed to the data record; and Convert each data record into a binary data record.

3. The computer-implemented method of claim 2, wherein: The steps to convert each data record into a binary data record include: identifying all attributes identified by any non-binary value in any data record in the plurality of data records as binary data record attributes; and For each data record, a binary value is generated for each binary data record attribute, each generated binary value identifying whether the data record contains a non-binary value identifying that the object is associated with the binary data record attribute.

4. A computer-implemented method according to any one of claims 1 to 3, wherein: The at least one cluster of binary data records contains less than a second predetermined number of clusters.

5. The computer-implemented method of claim 4, wherein: The second predetermined number of clusters is the total number of binary data records divided by the first predetermined number.

6. The computer-implemented method of any one of claims 1 to 5, wherein: The clustering process includes: clustering the plurality of binary data records using a first clustering algorithm; and Any cluster of binary data records having fewer than the first predetermined number of binary data records is combined with the cluster of nearest neighbor binary data records.

7. A computer-implemented method according to any one of claims 1 to 5, wherein: The clustering process includes: clustering the plurality of binary data records using a second clustering algorithm; and Any cluster of binary data records having fewer than the first predetermined number of binary data records is deleted from the at least one cluster of binary data records.

8. A computer-implemented method according to any one of claims 1 to 7, wherein: Generalizing a binary value includes marking the binary value as having an unknown value.

9. The computer-implemented method of any one of claims 1 to 8, wherein: Performing the generalization evaluation process for each set of attributes includes sequentially performing iterations of the generalization evaluation process for each set of attributes in turn.

10. The computer-implemented method of any one of claims 1 to 8, wherein: Executing the generalization evaluation process for each set of attributes includes: sequentially executing the first step of each generalization evaluation process; and The second step of each generalization evaluation process is performed in parallel with each other.

11. The computer-implemented method of any one of claims 1 to 10, wherein: For each identified cluster, after performing any generalization evaluation process, responsive to the number of attributes associated with ungeneralized binary values ​​being less than two, terminating the one or more iterations of performing the generalization process.

12. The computer-implemented method of any one of claims 1 to 11, wherein: The one or more iterations of the generalization process include a third predetermined number of iterations.

13. The computer-implemented method of any one of claims 1 to 12, further comprising, after performing the one or more iterations of the generalization process for each cluster of binary data records: Convert each binary data record into a corresponding data record containing one or more non-binary values, for each binary data record: defining, for each binary value associating an attribute with the object of the binary data record, a non-binary value for the data record, the non-binary value identifying the attribute associated with the data record; For each generalized binary value, a non-binary value is defined for the data record, the non-binary value identifying an indeterminate relationship between the object of the data record and an attribute associated with the generalized binary value.

14. A computer program product comprising computer program code means which, when run on a computing device having a processing system, causes the processing system to perform all the steps of the method according to any one of claims 1 to 13.

15. A processing system for generalizing data records, the processing system being configured to: A plurality of binary data records are obtained, each binary data record comprising a plurality of binary values, each binary value being associated with a different attribute of an object of the binary data record, wherein: Each binary value identifies whether the attribute belongs to the object of the binary data record; performing a clustering process on the plurality of binary data records to produce at least one cluster of binary data records; and For each cluster of binary data records, one or more iterations of the generalization process are performed, including: identifying as an identified attribute any attribute for which a non-zero number of binary values ​​for any data record in the cluster attributes the attribute to their corresponding binary data record; any set of identified attributes, where each set contains a different combination of the N identified attributes; and A generalization evaluation process is performed for each set of attributes, the generalization evaluation process comprising: A first step is to identify the number of binary data records that meet the condition, wherein each binary data record that meets the condition contains the following binary value, which attributes each attribute in the set of attributes to its corresponding binary record; and In a second step, in response to the number of binary data records that meet the condition being greater than 0 and less than a first predetermined number, generalize the binary values ​​associated with the set of attributes in all binary data records in the cluster of binary data records, wherein the value of N starts from 1 and increases by 1 for each iteration of the generalization process.

Images