A function dependency determination method and device
By generating a partition set and sampling the tuple set, invalid functional dependencies are generated and inversely derived to obtain valid functional dependencies, thus solving the problem of low efficiency in the existing technology and achieving efficient determination of functional dependencies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-20
- Publication Date
- 2026-03-24
Smart Images
Figure CN115168504B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to one or more embodiments in the field of data processing, and more particularly to a method and apparatus for determining function dependencies. Background Technology
[0002] As data assets increase, data processing workflows become more complex, and manual data processing methods are no longer sufficient to handle the explosive growth of enterprise data in the era of big data. When faced with datasets containing a huge number of fields and diverse data types, it is often necessary to first determine the relationships between the fields before further processing the dataset. Using functional dependencies to represent the relationships between fields is currently the mainstream approach.
[0003] Although some methods for determining functional dependencies based on datasets have been proposed in related technologies, they are often inefficient and increase computational and time overhead. Summary of the Invention
[0004] In view of the above, one or more embodiments of this specification provide a method and apparatus for determining function dependencies, which can solve the shortcomings existing in the related art.
[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:
[0006] According to a first aspect of one or more embodiments of this specification, a method for determining function dependencies is proposed, the method comprising:
[0007] Obtain the original dataset, which contains multiple tuples, each containing field values corresponding to at least two fields;
[0008] For each field, a corresponding partition set is generated. The partition set contains at least one set of tuples. The tuples in each set of tuples have the same field value for the corresponding field in the partition set.
[0009] Sampling is performed on all sets of tuples to obtain tuple pairs, and each tuple pair contains two tuples belonging to the same set of tuples.
[0010] For each tuple pair, generate the corresponding invalid functional dependencies and add them to the invalid functional dependency set to obtain all invalid functional dependencies corresponding to the sampled tuple pairs;
[0011] The set of invalid functional dependencies is inverted to obtain a set of valid functional dependencies, which contains the valid functional dependencies corresponding to the original dataset.
[0012] According to a second aspect of one or more embodiments of this specification, a reasoning method based on a sensitive field of functional dependency is proposed, the method comprising:
[0013] Obtain functional dependencies generated based on the dataset, which are used to characterize the relationships between fields contained in the dataset;
[0014] Identify the sensitive fields that the user has marked in the dataset;
[0015] Determine the target functional dependency from the functional dependencies, wherein the fields contained in the left-hand side set of the target functional dependency belong to the sensitive fields;
[0016] If the fields contained in the right-hand side set of the objective function dependency are different from the sensitive fields, then the fields contained in the right-hand side set of the objective function dependency are determined to be potential sensitive fields.
[0017] According to a third aspect of one or more embodiments of this specification, a function dependency determination apparatus is provided, the apparatus comprising:
[0018] First acquisition unit: Acquire the original dataset, which contains multiple tuples, each tuple containing field values corresponding to at least two fields;
[0019] Partitioning Unit: Generate a corresponding partition set for each field. The partition set contains at least one set of tuples. The tuples in each set of tuples have the same field value for the corresponding field in the partition set.
[0020] Sampling unit: Sample all sets of tuples to obtain tuple pairs, each tuple pair containing two tuples belonging to the same set of tuples;
[0021] Generation unit: For each tuple pair, generate the corresponding invalid functional dependencies and add them to the invalid functional dependency set to obtain all invalid functional dependencies corresponding to the sampled tuple pairs;
[0022] Inversion unit: Invert the invalid functional dependency set to obtain a valid functional dependency set, which contains the valid functional dependencies corresponding to the original dataset.
[0023] According to a fourth aspect of one or more embodiments of this specification, an inference apparatus based on a sensitive field of functional dependency is provided, the apparatus comprising:
[0024] The second acquisition unit: acquires the functional dependencies generated based on the dataset, wherein the functional dependencies are used to characterize the relationships between the fields contained in the dataset;
[0025] Fourth determining unit: Determines the sensitive fields marked by the user in the fields of the dataset;
[0026] Fifth determining unit: Determine the target functional dependency from the functional dependencies, wherein the fields contained in the left-hand side set of the target functional dependency belong to the sensitive fields;
[0027] Determination Unit: If the fields contained in the right-hand side set of the objective function dependency are different from the sensitive fields, then the fields contained in the right-hand side set of the objective function dependency are determined to be potential sensitive fields.
[0028] According to a fifth aspect of one or more embodiments of this specification, an electronic device is provided, comprising:
[0029] processor;
[0030] Memory used to store processor-executable instructions;
[0031] The processor implements the method as described in the first or second aspect by running the executable instructions.
[0032] According to a sixth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first or second aspect.
[0033] As can be seen from the above technical solutions, the method for determining functional dependencies provided in one or more embodiments of this specification first obtains an original dataset containing multiple tuples. Each tuple in the original dataset contains field values corresponding to at least two fields. Then, a partition set containing at least one set of tuples is generated for each field. Next, all tuple sets are sampled to obtain tuple pairs. Each tuple pair contains two tuples belonging to the same tuple set. For each tuple pair, a corresponding invalid functional dependency is generated and added to the invalid functional dependency set, thus obtaining all invalid functional dependencies corresponding to the sampled tuple pairs. Since tuples in the same tuple set have the same field value for a certain field, invalid functional dependencies can definitely be generated based on the sampled tuple pairs. Compared to directly comparing tuples in the original dataset, generating a partition set first reduces the number of tuple pairs, thereby reducing the computational and time overhead of subsequently generating invalid functional dependencies from tuple pairs. The invalid functional dependency set is then inverted to obtain a valid functional dependency set, which contains the valid functional dependencies corresponding to the original dataset. Compared to directly verifying whether functional dependencies hold, this specification generates functional dependencies to be verified by inverting invalid functional dependencies and uses the set of invalid functional dependencies to determine valid functional dependencies. This reduces the number of functional dependencies that need to be verified and simplifies the steps of verifying functional dependencies, thereby improving the efficiency of determining functional dependencies. Attached Figure Description
[0034] Figure 1 This is a system architecture diagram of a method for determining function dependencies provided in an exemplary embodiment.
[0035] Figure 2This is a flowchart illustrating a method for determining function dependencies as provided in an exemplary embodiment.
[0036] Figure 3 This is a schematic diagram of an exemplary embodiment for generating a partition set.
[0037] Figure 4 This is a schematic diagram of sequential sampling provided in an exemplary embodiment.
[0038] Figure 5 This is a schematic diagram illustrating the generation of invalid functional dependencies, provided in an exemplary embodiment.
[0039] Figure 6 This is a schematic diagram of constructing a binary search tree provided in an exemplary embodiment.
[0040] Figure 7 This is a schematic diagram of an exemplary embodiment for generating valid functional dependencies.
[0041] Figure 8 This is a schematic diagram of a sliding serial port sampling method provided in an exemplary embodiment.
[0042] Figure 9 This is a flowchart illustrating an exemplary embodiment of a reasoning method for sensitive fields based on functional dependencies.
[0043] Figure 10 This is a schematic structural diagram of a device provided in an exemplary embodiment.
[0044] Figure 11 This is a block diagram of a function dependency determination apparatus provided in an exemplary embodiment.
[0045] Figure 12 This is a block diagram of an inference apparatus for sensitive fields based on functional dependencies, provided in an exemplary embodiment. Detailed Implementation
[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.
[0047] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.
[0048] To further illustrate one or more embodiments of this specification, the following embodiments are provided:
[0049] Figure 1 This is a system architecture diagram of a method for determining function dependencies provided in an exemplary embodiment, such as... Figure 1 As shown, the system architecture diagram includes: database 11 and server 12; database 11 is used to store and provide datasets, while server 12 is used to process the datasets provided by database 11 to determine functional dependencies.
[0050] Functional dependency refers to a situation where one attribute set can determine another attribute set, and the other attribute set is said to depend on the first attribute set. From a mathematical perspective, functional dependency is as follows: Let R(U) be a relation schema on an attribute set U, and X and Y be subsets of U. If for any two possible relations r1 and r2 of R(U), if r1[x] = r2[x], then r1[y] = r2[y], or if r1[y] is not equal to r2[y], then r1[x] is not equal to r2[x], then X determines Y, or Y depends on X.
[0051] Taking a student table as an example, if a student's student ID can determine their name, and vice versa, this is called a functional dependency: the name attribute depends on the student ID attribute, or vice versa. These dependencies are represented by "{student ID}→{name}" and "{name}→{student ID}". If a student's age cannot determine their name, but their name can, this is called a functional dependency: the name attribute determines the age attribute, or vice versa. Finally, if a student's age cannot determine their school, and their school cannot determine their course, then there is no functional dependency between the age and school attributes.
[0052] It can be seen that the relationship pattern between attribute set X and attribute set Y corresponds to functional dependency: when there is a one-to-one relationship between X and Y, such as the one-to-one relationship between student ID and name, then there is a functional dependency "X→Y" or "Y→X"; when there is a one-to-many relationship between X and Y, such as the one-to-many relationship between age and name, then there is a functional dependency "Y→X"; if there is a many-to-many relationship between X and Y, such as the many-to-many relationship between student and course, then there is no functional dependency between X and Y.
[0053] In the technical solution of this specification, database 11 can be any type of relational database, and the dataset maintained by database 11 is carried in tables; each record in the dataset is recorded as a row of data in the table, that is, a tuple. For example, in the student table scenario mentioned above, each student's name, student ID, age, and other information are recorded in a row in the student table, forming the corresponding tuple. At the same time, the columns of the table are fields, and each row of data contains field values corresponding to each field. For example, each student's information contains field values corresponding to fields such as name, student ID, and age, which are the attributes of the corresponding student in the dimensions represented by these fields.
[0054] Server 12 can be a physical server containing a single host, or a virtual server hosted in a host cluster. During operation, server 12 obtains a dataset from database 11 and processes the dataset using the technical solutions described in this specification to determine functional dependencies. While in Figure 1 In the illustrated embodiment, database 11 is shown to be independent of server 12, but in some cases, database 11 may also be deployed in the local storage space of server 12, and this specification does not limit this.
[0055] In fact, the technical solution in this specification can efficiently and quickly analyze and obtain the corresponding functional dependencies through dataset processing, so as to achieve further processing objectives based on the obtained functional dependencies. For example, in the scenario of sensitive field analysis, by determining the functional dependencies, it can be used to analyze known sensitive fields, thereby inferring potential sensitive fields. The technical solution in this specification will be described below with reference to embodiments.
[0056] Figure 2 This is a flowchart illustrating a method for determining function dependencies as provided in an exemplary embodiment. For example... Figure 2 As shown, the method may include the following steps:
[0057] Step 201: Obtain the original dataset, which contains multiple tuples, each containing field values corresponding to at least two fields.
[0058] A dataset can refer to a collection of data, usually in tabular form. Each column represents a specific field, each row corresponds to a tuple, and each tuple has a corresponding field value. The raw dataset can refer to an unprocessed dataset, as opposed to a replacement dataset. The process of converting the raw dataset into a replacement dataset will be described in detail in subsequent steps and will not be repeated here.
[0059] Tuple number Name gender age blood pressure Medication 1 Xiao Hong female 11 normal Drug X 2 noob male 20 Low Drug B 3 Xiao Hei male 65 normal Drug X 4 Little Blue male 35 high Drug A 5 Little Green female 24 high Drug A 6 Xiao Huang male 24 normal Drug X
[0060] Table 1
[0061] The following section uses Table 1 as an example to describe in detail the method for determining functional dependencies. Table 1 is a dataset provided by an exemplary embodiment. As shown in Table 1, the original dataset includes 6 tuples, corresponding to tuple numbers "1, 2, 3, 4, 5, 6", and 5 different fields: "Name", "Gender", "Age", "Blood Pressure", and "Medication". Each of the 6 tuples has a corresponding field value for these 5 fields. For example, the tuple with tuple number "1" has the field value "Xiao Hong" for the "Name" field, "Female" for the "Gender" field, "11" for the "Age" field, "Normal" for the "Blood Pressure" field, and "Drug X" for the "Medication" field. Different tuples can have the same or different values in the same field. For example, tuple number "3" has the value "Xiao Hei" in the "Name" field and "Normal" in the "Blood Pressure" field. This tuple differs from tuple number "1" in the "Name" field but has the same value in the "Blood Pressure" field. Subsequent steps generate invalid functional dependencies by comparing the differences in the values of each tuple in the same field.
[0062] Step 202: Generate a corresponding partition set for each field. The partition set contains at least one set of tuples, and the tuples in each set have the same field value for the corresponding field in the partition set.
[0063] Figure 3 This is a schematic diagram illustrating an exemplary embodiment for generating a partition set, such as... Figure 3As shown, five partition sets are generated for the five fields in Table 1. Each partition set contains multiple sets of tuples, and the tuples in each set have the same field value for the corresponding field in that partition set. For example, for the "Gender" field, there are only two field values: "Male" and "Female". The tuples with the value "Male" in the "Gender" field have tuple indices "2, 3, 4, 6", and the tuples with the value "Female" have tuple indices "1, 5". Therefore, the tuples corresponding to indices "2, 3, 4, 6" and the tuples corresponding to indices "1, 5" are added to two different tuple sets, and both of these tuple sets are partitioned into the "Gender" partition set. The final result is as follows. Figure 3 As shown, there are 5 partition sets, corresponding to "name, gender, age, blood pressure, and medication". In the "gender" partition set, there are 2 sets of tuples. The first set of tuples is "{2, 3, 4, 6}", which contains 4 tuples. The field value of these 4 tuples in the gender field is "male". The second set of tuples is "{1, 5}", which contains 2 tuples. The field value of these 2 tuples in the gender field is "female".
[0064] This embodiment divides tuples with the same field value into the same tuple set, so that in the subsequent sampling process of the tuple set, only tuples with the same field value on a certain field are sampled, which reduces the number of sampled tuple pairs, improves sampling efficiency, and further reduces the computational overhead of generating functional dependencies.
[0065] Step 203: Sample all tuple sets to obtain tuple pairs, each tuple pair containing two tuples belonging to the same tuple set.
[0066] During the sampling process, the partition set or the set of tuples in the partition set can be sorted and sampled in a certain order, or the set of tuples can be sampled randomly without sorting. This specification does not impose any restrictions on this.
[0067] In one embodiment, each partition set is treated as a queue unit, and the tuple sets in each queue unit are sorted according to their efficiency values. The efficiency value is positively correlated with the number of tuples contained in the corresponding tuple set. When sampling the tuple set in any queue unit, the tuple sets in the queue unit are sampled in descending order of efficiency value to obtain tuple pairs.
[0068] Figure 4 This is a schematic diagram of sequential sampling provided in an exemplary embodiment, such as... Figure 4As shown, in the "Name" partition set, the set of tuples with the field value "Male" contains 4 tuples, while the set of tuples with the field value "Female" contains 2 tuples. Therefore, the efficiency value of the set of tuples with the field value "Male" is higher than that of the other set of tuples, so this set of tuples is sampled first. The sampling result for the set of tuples with the field value "Male" is "{2, 3}, {3, 4}, {4, 6}, {2, 4}, {3, 6}, {4, 6}, and the sampling result for the set of tuples with the field value "Female" is "{1, 5}", resulting in 7 tuple pairs. Similarly, the tuple sets in other partition sets are also sorted and sampled according to their efficiency values. The final sampling result should be "{2,3},{3,4},{4,6},{1,5},{5,6},{1,3},{3,6},{4,5},{2,4},{1,6},{2,6}", a total of 11 tuple pairs.
[0069] This embodiment sorts the tuple sets according to their efficiency values and prioritizes sampling the tuple sets with high efficiency values. This allows most tuple pairs to be extracted in the early stages of sampling, making the growth curve of the entire functional dependency approach a flattening curve earlier, thus ending the sampling process earlier and saving sampling time.
[0070] Step 204: Generate corresponding invalid functional dependencies for each tuple pair and add them to the invalid functional dependency set to obtain all invalid functional dependencies corresponding to the sampled tuple pairs.
[0071] An invalid functional dependency is a dependency that does not exist. The left-hand side of an invalid functional dependency consists of all fields in the two tuples that have the same value, while the right-hand side consists of any fields in the corresponding tuple pair that have different values. In a dataset, fields with the same value cannot determine fields with different values. For example, in the tuple pair "{2, 3}", tuples 2 and 3 only have the same value in the "gender" field; their values in all other fields are different. Clearly, the "gender" field alone cannot determine the values of tuples 2 and 3 in other fields. Therefore, a functional dependency generated in this way is definitely invalid.
[0072] Figure 5 This is a schematic diagram illustrating the generation of invalid functional dependencies, as provided in an exemplary embodiment. Figure 5As shown, invalid functional dependencies are generated based on the differences in field values of the two tuples in a tuple pair across all fields. Taking the tuple pair "{2, 3}" as an example, tuples 2 and 3 only have the same value in the "gender" field; their values differ in all other fields. Using the "gender" field as the left-hand side of the invalid functional dependency and the remaining fields as the right-hand side, four invalid functional dependencies can be generated: "{gender}→{name}", "{gender}→{age}", "{gender}→{blood pressure}", and "{gender}→{medication}". Similarly, invalid functional dependencies are generated by comparing tuples within all tuple pairs. The functional dependencies obtained from the tuple pairs “{2,3}, {3,4}, {4,6}, {1,5}, {2,4}, {2,6}” are all the same: {gender}→{name}, {gender}→{age}, {gender}→{blood pressure}, {gender}→{medication}. The functional dependencies obtained from the tuple pair “{5,6}” are: {age}→{name}, {age}→{gender}, {age}→{blood pressure}, {age}→{medication}. The functional dependencies obtained from the tuple pairs “{1,3}{4,5}{1,6}” are all: {blood pressure,medication}→{name}, {blood pressure,medication}→{gender}, {blood pressure,medication}→{age}. The functional dependencies obtained from the tuple pairs “{3,6}” are: {gender,blood pressure,medication}→{name}, {gender,blood pressure,medication}→{age}. This embodiment compares the field values of tuple pairs across all fields to identify invalid functional dependencies, thus laying the groundwork for generating valid functional dependencies in the future.
[0073] In the invalid functional dependencies generated in this embodiment, the functional dependencies "{gender}→{name}" and "{gender, blood pressure, medication}→{name}" have an inclusion relationship. Given that the right-hand sides are the same, the left-hand side of the first invalid functional dependency belongs to the left-hand side of the second invalid functional dependency. In fact, since the "gender," "blood pressure," and "medication" fields cannot determine the "name" field, the "gender" field certainly cannot determine the "name" field. Therefore, the first invalid functional dependency is unnecessary, and this unnecessary invalid functional dependency will affect the efficiency of generating subsequent valid functional dependencies.
[0074] In one embodiment, unnecessary invalid functional dependencies are identified, the left-hand side of which is included in the left-hand side of other invalid functional dependencies, and the right-hand side of which is the same as the right-hand side of the other functional dependencies; adding a candidate field to the left-hand side of any invalid functional dependency includes adding a candidate field to the left-hand side of any invalid functional dependency that is different from the unnecessary invalid functional dependency.
[0075] Furthermore, by constructing a binary search tree with the left-hand side of invalid functional dependencies as the path, unnecessary functional dependencies can be pruned, thereby improving the efficiency of converting invalid functional dependencies into valid functional dependencies.
[0076] Figure 6 This is a schematic diagram of constructing a binary search tree provided in an exemplary embodiment, such as... Figure 6 As shown, a binary search tree is built for each field. For ease of drawing, the five fields "Name, Gender, Age, Blood Pressure, and Medication" are represented by "A, B, C, D, and E". Taking the "Blood Pressure" field as an example, the "Blood Pressure" field is taken as the root node. From all invalid functional dependencies, invalid functional dependencies whose left-hand sides contain the "Blood Pressure" field are identified. The results are: "{Blood Pressure, Medication}→{Name}, {Blood Pressure, Medication}→{Gender}, {Blood Pressure, Medication}→{Age}, {Gender, Blood Pressure, Medication}→{Name}, {Gender, Blood Pressure, Medication}→{Age}". The right-hand sides of these five invalid functional dependencies contain three fields: "{Name}, {Gender}, and {Age}". Therefore, A, B, and C are drawn next to the root node (in the actual construction process, the right-hand sides of invalid functional dependencies do not appear in the binary tree; this specification shows the right-hand sides next to the nodes for ease of understanding). The left-hand sides of all five invalid functional dependencies contain "{blood pressure, medication}", so they are connected to form a leaf node E, and A, B, and C are drawn on the edges of node E to represent the right-hand sides. Since the left-hand sides of the latter two invalid functional dependencies also contain a gender field, and the right-hand sides of these two invalid functional dependencies are "{name}" and "{age}" respectively, the original node E is split into two: A and C are drawn next to the left node E, and B is drawn next to the right node E. Child node B of the left node E is constructed, and A and C are continued on the edges of child node B as the right-hand sides. As shown in the figure, there are two branches in the constructed binary search tree, representing "DEB→A" and "DEB→C" respectively, i.e., "{gender, blood pressure, medication}→{name}", "{gender, blood pressure, medication}→{age}", and "{blood pressure, medication}→{gender}". It's easy to see that the two invalid functional dependencies, "{blood pressure, medication}→{name}" and "{blood pressure, medication}→{age}", have been pruned. This is understandable, since the fields of gender, blood pressure, and medication cannot determine the name field, then the fields of blood pressure and medication are even less likely to determine it. This implementation removes unnecessary invalid functional dependencies by constructing a binary search tree with the left-hand side of the invalid functional dependencies as the path, thus reducing subsequent computational overhead.
[0077] In a dataset, the data types of field values are not necessarily the same. When comparing field values of two tuples within a tuple pair, a lot of storage space is often occupied due to the data types. Other methods can be used to replace specific field values for comparison.
[0078] In one embodiment, the field values corresponding to the multiple tuples in the original dataset are replaced with: the field value index formed by combining the index of the partition set corresponding to the field to which the corresponding field value belongs and the index of the tuple set to which the field value belongs, so as to generate a replacement dataset;
[0079] The step of generating corresponding invalid functional dependencies for each tuple pair includes: generating the invalid functional dependencies based on the replacement dataset.
[0080] Taking the tuple with index "1" as an example, assuming the five fields "name, gender, age, blood pressure, and medication" correspond to indices "1, 2, 3, 4, and 5" respectively, the tuple's index in the tuple set containing the "name" partition is 1. (For ease of understanding, the indices of the tuple set can be arranged as follows...) Figure 2 The tuple sets are sorted according to the order in which they are written in the partition set. For example, in the "Gender" partition set, the tuple set with the field value "Male" has an index of 1, and the tuple set with the field value "Female" has an index of 2. Therefore, the field value of the "Name" field for this tuple can be replaced with "1-1". Similarly, the corresponding replacements are made for other field values, as shown in Table 2. When comparing the field values of two tuples within a tuple pair later, the difference in index can be directly compared; there is no need to compare the specific field values.
[0081] Tuple number Name gender age blood pressure Medication 1 1-1 2-2 3-2 4-1 5-1 2 1-2 2-1 3-3 4-3 5-3 3 1-3 2-1 3-4 4-1 5-1 4 1-4 2-1 3-5 4-2 5-2 5 1-5 2-2 3-1 4-2 5-2 6 1-6 2-1 3-1 4-1 5-1
[0082] Table 2
[0083] This embodiment replaces the original field values with the same sequence number. On the one hand, it takes into account the characteristics of the computer's underlying storage structure, reducing the time and space overhead required for computation; on the other hand, it transforms the original data without affecting subsequent computation, reducing the risk of privacy leakage.
[0084] Step 205: Invert the invalid functional dependency set to obtain a valid functional dependency set, which contains the valid functional dependencies corresponding to the original dataset.
[0085] Generalization refers to a covering relationship between functional dependencies. The left-hand side of the generalization of an invalid functional dependency is contained within the left-hand side of the invalid functional dependency, and the right-hand side is the same as the right-hand side of the invalid functional dependency. For example, the invalid functional dependency "{gender, blood pressure, medication}→{name}" can completely cover the invalid functional dependency "{blood pressure, medication}→{name}", so "{blood pressure, medication}→{name}" can be called a generalization of "{gender, blood pressure, medication}→{name}".
[0086] In one embodiment, the step of inverting the invalid functional dependency set to obtain a valid functional dependency set includes: extracting any invalid functional dependency from the invalid functional dependency set, wherein the left-hand side of the invalid functional dependency is all fields whose two tuples in the corresponding tuple pair have the same field value, and the right-hand side is any field whose two tuples in the corresponding tuple pair have different field values; adding candidate fields to the left-hand side of the invalid functional dependency to obtain a functional dependency to be verified, wherein the candidate fields are other fields among the at least two fields that are different from the fields contained in the left-hand side and right-hand side of the invalid functional dependency; if the functional dependency to be verified does not belong to the invalid functional dependency set or the generalization of the invalid functional dependencies in the set, then the functional dependency to be verified is determined to be a valid functional dependency, and the valid functional dependency is added to the valid functional dependency set.
[0087] The following collection Figure 7 A detailed introduction to the inversion method is provided. Figure 7 This is a schematic diagram illustrating the generation of valid functional dependencies, as provided in an exemplary embodiment. Figure 7 As shown, candidate fields are added to the left-hand side set of any invalid functional dependency, and it is verified whether they are converted into valid functional dependencies. Taking the invalid functional dependency "{blood pressure, medication}→{age}" as an example, the fields contained in the left-hand and right-hand side sets of this invalid functional dependency are "blood pressure", "age", and "medication". Therefore, the candidate fields can be "gender" and "name". Adding these two fields to the left-hand side set of the invalid functional dependency respectively, we get two functional dependencies to be verified: "{gender, blood pressure, medication}→{age}" and "{name, blood pressure, medication}→{age}". It can be seen that the first functional dependency to be verified belongs to the set of invalid functional dependencies, while the second functional dependency to be verified does not belong to the set of invalid functional dependencies. Therefore, it can be determined that "{name, blood pressure, medication}→{age}" is a valid functional dependency. This embodiment adds candidate fields to invalid functional dependencies to obtain functional dependencies to be verified. It then determines whether a functional dependency to be verified is a valid functional dependency by verifying whether it belongs to the set of invalid functional dependencies. This process converts invalid functional dependencies into valid functional dependencies, eliminating the need to directly verify the validity of functional dependencies. Instead, it utilizes the set of invalid functional dependencies for determination, reducing computational overhead and improving the efficiency of determining functional dependencies.
[0088] Furthermore, if the functional dependency to be verified belongs to the set of invalid functional dependencies or a generalization of invalid functional dependencies in that set, then other candidate fields, distinct from the already added candidate fields, are added to the left-hand side set of the functional dependency to be verified until the functional dependency to be verified is determined to be a valid functional dependency. Candidate fields are then added to the second functional dependency to be verified. Since the previously added candidate field for "{gender, blood pressure, medication}→{age}" is the "gender" field, the next candidate field to be added should be the "name" field, resulting in the functional dependency to be verified "{name, gender, blood pressure, medication}→{age}". This functional dependency does not belong to the set of invalid functional dependencies, therefore, "{name, gender, blood pressure, medication}→{age}" can be determined as a valid functional dependency. This embodiment ensures that invalid functional dependencies can be fully converted into valid functional dependencies, avoiding the omission of determined functional dependencies.
[0089] As mentioned earlier, during the sampling process, the tuple set can be sorted according to its efficiency value. Since a tuple set with a higher efficiency value extracts more tuple pairs, and a tuple set with a lower efficiency value extracts fewer tuple pairs, when dealing with datasets with a large number of tuples and fields, sampling the tuple set with higher efficiency values first can ensure that most tuple pairs are extracted in the early stages of sampling. In practice, to ensure sampling efficiency, sufficiency can be sacrificed to some extent to improve the efficiency of generating functional dependencies.
[0090] In one embodiment, multiple rounds of sampling are performed on all tuple sets to obtain tuple pairs, including: in the first round of sampling, sliding sampling is performed on all tuple sets according to the initial size of the sampling window; or, in non-first rounds of sampling, the sampling window is increased based on the previous round, and a new round of sliding sampling is restarted for all tuple sets; wherein, the tuples at both ends of the sliding window are used to generate unary pairs; if the growth rate of the invalid functional dependency reaches a first preset lower limit threshold after the current round of sliding sampling ends, the sampling ends; otherwise, the next round of sliding sampling begins.
[0091] Figure 8 This is a schematic diagram of a sliding window sampling method provided in an exemplary embodiment, such as... Figure 8As shown, a sliding window is used to sample the tuple set in multiple rounds (tuple sets with only one tuple cannot be sampled to form a pair, so they are not included in the queue). In the first round of sampling, the initial size of the sampling window is 2. Starting from the tuple set at the head of the queue, i.e., the set of tuples with the field value "male" in the "gender" partition set, sliding sampling is performed on all tuple sets, extracting the two tuples at both ends of the sliding window as a pair. The sampling results of the first round are: {2, 3}, {3, 4}, {4, 6}, {1, 5}, {5, 6}, {1, 3}, {3, 6}, {4, 5}. Figure 4 As can be seen, the invalid functional dependencies generated through these tuples encompass all invalid functional dependencies, which will not be explained one by one here. Since this is the first round of sampling, the growth rate of invalid functional dependencies is 100%.
[0092] In the second round of sampling, the sliding window size is increased to 3, and all tuple sets are traversed. The sampling results of the second round are: {2, 4}, {3, 6}, and {1, 6}. Clearly, the invalid functional dependencies generated by these tuples are duplicates of those generated in the first round, therefore the growth rate of invalid functional dependencies in this round is 0. If the first preset lower threshold is 5%, then the growth rate of invalid functional dependencies after the second round has reached the first preset lower threshold, and sampling can end. To verify that the value of the third round of sampling is not significant, the sliding window size is again increased to 4 for a third round of sampling, and the sampling result is: {2, 6}. The invalid functional dependency generated by the tuple "{2, 6}" is also duplicated of those generated in the first round of sampling. Therefore, it can be seen that as the sampling rounds proceed, the sampling efficiency decreases. Setting a first preset lower threshold can reduce time and computational overhead while ensuring accuracy.
[0093] Furthermore, if the growth rate of the effective functional dependency is not found to have reached the second preset lower threshold after the current round of sliding sampling, then the next round of sliding sampling is initiated until the growth rate of the effective functional dependency reaches the preset threshold after any round of sliding sampling.
[0094] After each round of sampling, it is necessary to calculate not only the growth rate of invalid functional dependencies, but also the growth rate of valid functional dependencies. The growth rate of valid functional dependencies here refers to the growth rate of valid functional dependencies generated by newly added invalid functional dependencies after each round of sampling. For example, if after a round of sampling, a new invalid functional dependency "{age}→{name}" is added, and the valid functional dependencies generated by this invalid functional dependency are "{gender, age}→{name}, {blood pressure, age}→{name}, {medication, age}→{name}", if these 3 valid functional dependencies have not been generated in previous rounds, then the growth rate of valid functional dependencies is the ratio of the number of newly added valid functional dependencies (3) to the total number of valid functional dependencies (the number of previously generated valid functional dependencies plus 3).
[0095] In the above embodiment, since all invalid functional dependencies were identified in the first round of sampling, the growth rate of valid functional dependencies is 0 after the second round of sampling, and no further sampling is needed. Besides this situation, there may also be instances where the growth rate of invalid functional dependencies reaches the first preset lower threshold, but the growth rate of valid functional dependencies does not reach the second preset lower threshold. In this case, sampling will not end and needs to continue. Sampling will only end when the growth rate of invalid functional dependencies reaches the first preset lower threshold and the growth rate of valid functional dependencies reaches the second preset lower threshold.
[0096] This embodiment improves sampling efficiency while ensuring sufficient sampling to a certain extent, thereby reducing the possibility of missing functional dependencies.
[0097] In the above embodiments, effective functional dependencies can be generated based on the original dataset. These effective functional dependencies can be used to characterize the relationships between fields in the original dataset. Taking the effective functional dependency "{gender, age} → {name}" as an example, the left-hand side of this functional dependency includes the "gender" and "age" fields, and the right-hand side includes the "name" field. Therefore, this effective functional dependency can represent that the value of a tuple in the "name" field can be determined by the value of that tuple in the "gender" field and the value of that tuple in the "age" field. Due to the special role of functional dependencies, they are often applied in scenarios where field relationships need to be determined.
[0098] In one embodiment, the user needs to determine the relationship between the "Gender" and "Age" fields in the dataset shown in Table 1. The system determines that there are no valid functional dependencies where "{Gender}→{Age}" or "{Age}→{Gender}" exists, and that there are invalid functional dependencies where "{Gender, Blood Pressure, Medication}→{Age}" and "{Age}→{Gender}" exist. It can be seen that if the value of a tuple in the "Gender" field is determined, the value of that tuple in the "Age" field cannot be determined, and vice versa. Therefore, there is no direct relationship between the "Gender" and "Age" fields, and the two fields cannot be mutually derived.
[0099] In another embodiment, the user needs to determine the relationship between the "Name" and "Age" fields in the dataset shown in Table 1. It is determined that "{Name}→{Age}" exists in the valid functional dependencies, but not in the invalid functional dependencies set, and "{Age}→{Name}" exists in the invalid functional dependencies set. It can be seen that determining the value of a tuple in the "Name" field allows determining its value in the "Age" field; however, determining the value of a tuple in the "Age" field does not necessarily determine its value in the "Name" field. Therefore, the value of the "Name" field can be derived from the value of the "Age" field, but not vice versa. The relationship between the "Name" and "Age" fields is that the "Name" field unilaterally determines the "Age" field.
[0100] Specifically, the method for determining functional dependencies can be applied to sensitive field identification scenarios. In one embodiment, sensitive fields marked by the user in the at least two fields are identified; a target functional dependency is determined from the valid functional dependencies, wherein the fields contained in the left-hand side of the target functional dependency belong to the sensitive fields; if the fields contained in the right-hand side of the target functional dependency are different from the sensitive fields, then the fields contained in the right-hand side of the target functional dependency are determined to be potential sensitive fields.
[0101] Sensitive fields refer to data fields that, if leaked, may cause serious harm to society or individuals, including personal privacy information such as name and gender. Users can label sensitive fields either by simply marking them as non-sensitive or non-sensitive, or by classifying fields into five levels. Non-sensitive fields could be set to level 0, and sensitive fields could be assigned higher levels as their sensitivity increases, with level 5 being the most sensitive. This specification does not restrict the specific labeling method.
[0102] If the sensitive fields marked by the user are gender and age, the valid functional dependencies with these two sensitive fields as the left set are: {gender, age} → {name}, {gender, age} → {blood pressure}, and {gender, age} → {medication}. The right sets of these three valid functional dependencies are the "name", "blood pressure", and "medication" fields, respectively. None of these three fields are sensitive fields; therefore, it can be determined that the "name", "blood pressure", and "medication" fields are all potential sensitive fields.
[0103] This embodiment applies the method for determining functional dependencies to the field of sensitive field identification. It determines the relationship between fields based on functional dependencies and infers potential sensitive fields through functional dependencies and sensitive fields, avoiding manual labeling of sensitive fields and reducing time and manpower costs.
[0104] Figure 9 This is a flowchart illustrating an exemplary embodiment of a reasoning method for sensitive fields based on functional dependencies, such as... Figure 9 As shown, the method may include the following steps:
[0105] Step 901: Obtain the functional dependencies generated based on the dataset, which are used to characterize the relationships between fields in the dataset.
[0106] The determination of function dependencies can be achieved through the foregoing embodiments or through other algorithms in related technologies. This specification does not limit this method.
[0107] Step 902: Determine the sensitive fields marked by the user in the fields of the dataset.
[0108] Step 903: Determine the target functional dependency from the functional dependencies, wherein the fields contained in the left-hand side set of the target functional dependency belong to the sensitive fields.
[0109] Step 904: If the fields contained in the right-hand side set of the objective function dependency are different from the sensitive fields, then the fields contained in the right-hand side set of the objective function dependency are determined to be potential sensitive fields.
[0110] Using the dataset in Table 1 as an example, if the sensitive fields marked by the user are "age" and "blood pressure", the functional dependencies with these two fields as the left-hand side include: {blood pressure, age} → {name}, {blood pressure, age} → {gender}, and {blood pressure, age} → {medication}. The right-hand side of these three functional dependencies are the "name", "gender", and "medication" fields, respectively. None of these three fields are sensitive fields; therefore, we can determine that the "name", "gender", and "medication" fields are potential sensitive fields.
[0111] This embodiment uses functional dependencies to characterize the relationships between fields in a dataset, enabling users to infer potential sensitive fields based on functional dependencies and labeled sensitive fields, thereby reducing time and manpower costs and improving the efficiency of sensitive field identification.
[0112] Figure 10 This is a schematic structural diagram of a device provided in an exemplary embodiment. Please refer to... Figure 10 At the hardware level, the device includes a processor 1002, an internal bus 1004, a network interface 1006, memory 1009, and non-volatile memory 1010, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 1002 reads the corresponding computer program from the non-volatile memory 1010 into memory 1008 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0113] Please refer to Figure 11 A deterministic device for functional dependencies can be applied to, for example... Figure 11 The device shown, in order to implement the technical solution of this specification, may include:
[0114] The first acquisition unit 1101 is used to acquire the original dataset, which contains multiple tuples, and each tuple contains field values corresponding to at least two fields.
[0115] The partitioning unit 1102 is used to generate a corresponding partition set for each field. The partition set contains at least one set of tuples, and the tuples in each set of tuples have the same field value on the corresponding field of the partition set.
[0116] Sampling unit 1103 is used to sample all tuple sets to obtain tuple pairs, and each tuple pair contains two tuples belonging to the same tuple set.
[0117] The generation unit 1104 is used to generate corresponding invalid functional dependencies for each tuple pair and add them to the invalid functional dependency set to obtain all invalid functional dependencies corresponding to the sampled tuple pairs.
[0118] Inversion unit 1105 inverts the invalid functional dependency set to obtain a valid functional dependency set, which contains the valid functional dependencies corresponding to the original dataset.
[0119] Optional, also includes:
[0120] The replacement unit 1106 is used to replace the field values corresponding to the multiple tuples in the original dataset with: the field value index formed by combining the index of the partition set corresponding to the field to which the corresponding field value belongs and the index of the tuple set to which the field value belongs, so as to generate a replacement dataset;
[0121] The generation unit 1104 is specifically used to generate the invalid functional dependencies based on the replacement dataset.
[0122] Optionally, the sampling unit 1103 is specifically used to treat each partition set as a queue unit, and sort the tuple set in each queue unit according to the efficiency value, wherein the efficiency value is positively correlated with the number of tuples contained in the corresponding tuple set.
[0123] When sampling the set of tuples in any queue unit, the tuple sets in the queue unit are sampled in descending order of efficiency value to obtain tuple pairs.
[0124] Optionally, the sampling unit 1103 is specifically used to perform sliding sampling on all tuple sets according to the initial size of the sampling window in the first round of sampling; or, in non-first rounds of sampling, to increase the sampling window based on the previous round and restart a round of sliding sampling on all tuple sets; wherein, the tuples at both ends of the sliding window are used to generate one-tuple pairs;
[0125] If the growth rate of the invalid functional dependency reaches a first preset lower limit threshold after the current round of sliding sampling, then sampling ends; otherwise, proceed to the next round of sliding sampling.
[0126] Optionally, the sampling unit 1103 is specifically used to enter the next round of sliding sampling if the growth rate of the effective functional dependency calculated after the current round of sliding sampling does not reach the second preset lower threshold, until the growth rate of the effective functional dependency calculated after any round of sliding sampling reaches the preset threshold.
[0127] Optionally, the inversion unit 1105 is specifically used for:
[0128] Extract any invalid functional dependency from the set of invalid functional dependencies. The left-hand set of the invalid functional dependency is all fields in the corresponding tuple pair that have the same field value, and the right-hand set is any field in the corresponding tuple pair that has different field values.
[0129] Candidate fields are added to the left-hand set of any invalid functional dependency to obtain the functional dependency to be verified. The candidate fields are other fields among the at least two fields that are different from the fields contained in the left-hand and right-hand sets of any invalid functional dependency.
[0130] If the functional dependency to be verified does not belong to the set of invalid functional dependencies or the generalization of invalid functional dependencies in the set, then the functional dependency to be verified is determined to be a valid functional dependency, and the valid functional dependency is added to the set of valid functional dependencies.
[0131] Optional, also includes:
[0132] The first determining unit 1107 is used to determine unnecessary invalid functional dependencies, wherein the left-hand set of the unnecessary invalid functional dependencies is included in the left-hand set of other invalid functional dependencies, and the right-hand set of the unnecessary invalid functional dependencies is the same as the right-hand set of the other functional dependencies.
[0133] The inversion unit 1105 is specifically used to add candidate fields to the left-hand side set of any invalid functional dependency that is distinct from the unnecessary invalid functional dependency.
[0134] Optional, also includes:
[0135] The addition unit 1108 is used to add other candidate fields that are different from the candidate fields that have been added to the left-hand side set of the functional dependency to be verified if the functional dependency to be verified belongs to the set of invalid functional dependencies or a generalization of invalid functional dependencies in the set, until the functional dependency to be verified is determined to be the valid functional dependency.
[0136] Optional, also includes:
[0137] The second determining unit 1109 is used to determine the sensitive fields marked by the user in the at least two fields;
[0138] The third determining unit 1110 is used to determine the target functional dependency from the effective functional dependencies, wherein the fields contained in the left-hand set of the target functional dependency belong to the sensitive fields;
[0139] The first determination unit 1111 is used to determine that the field contained in the right-hand side set of the objective function dependency is a potential sensitive field if the field contained in the right-hand side set is different from the sensitive field.
[0140] Please refer to Figure 12 A reasoning device based on function-dependent sensitive fields can be applied to, for example... Figure 12 The device shown, in order to implement the technical solution of this specification, may include:
[0141] The second acquisition unit is used to acquire functional dependencies generated based on the dataset, wherein the functional dependencies are used to characterize the relationships between the fields contained in the dataset;
[0142] The fourth determining unit is used to determine the sensitive fields marked by the user in the fields of the dataset;
[0143] The fifth determining unit is used to determine the target functional dependency from the functional dependencies, wherein the fields contained in the left-hand side set of the target functional dependency belong to the sensitive fields;
[0144] The second determination unit is used to determine that if the fields contained in the right-hand side set of the objective function dependency are different from the sensitive fields, then the fields contained in the right-hand side set of the objective function dependency are potential sensitive fields.
[0145] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.
[0146] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0147] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0148] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0149] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0150] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0151] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0152] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this specification, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."
[0153] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.
Claims
1. A method for determining functional dependencies, characterized in that, The method includes: Obtain the original dataset, which contains multiple tuples, each containing field values corresponding to at least two fields; For each field, a corresponding partition set is generated. The partition set contains at least one set of tuples. The tuples in each set of tuples have the same field value for the corresponding field in the partition set. Sampling is performed on all sets of tuples to obtain tuple pairs, and each tuple pair contains two tuples belonging to the same set of tuples. For each tuple pair, generate corresponding invalid functional dependencies and add them to the invalid functional dependency set to obtain all invalid functional dependencies corresponding to the sampled tuple pairs; wherein, the left-hand set of the invalid functional dependencies consists of all fields in which two tuples have the same field value, and the right-hand set consists of any field in which the two tuples in the corresponding tuple pair have different field values. The set of invalid functional dependencies is inverted to obtain a set of valid functional dependencies, which contains the valid functional dependencies corresponding to the original dataset. The process of inverting the invalid functional dependency set to obtain the valid functional dependency set includes: Extract any invalid functional dependency from the set of invalid functional dependencies. The left-hand set of the invalid functional dependency is all fields in the corresponding tuple pair that have the same field value, and the right-hand set is any field in the corresponding tuple pair that has different field values. Candidate fields are added to the left-hand set of any invalid functional dependency to obtain the functional dependency to be verified. The candidate fields are other fields among the at least two fields that are different from the fields contained in the left-hand and right-hand sets of any invalid functional dependency. If the functional dependency to be verified does not belong to the set of invalid functional dependencies or the generalizations of invalid functional dependencies in that set, then the functional dependency to be verified is determined to be a valid functional dependency, and the valid functional dependency is added to the set of valid functional dependencies; wherein, if the left-hand side of any functional dependency is included in the left-hand side of any invalid functional dependency, and the right-hand side of any functional dependency is the same as the right-hand side of any invalid functional dependency, then any functional dependency is a generalization of any invalid functional dependency.
2. The method according to claim 1, characterized in that, The method further includes: replacing the field values corresponding to the multiple tuples in the original dataset with: the field value index formed by combining the index of the partition set corresponding to the field to which the corresponding field value belongs and the index of the tuple set to which the field value belongs, so as to generate a replacement dataset; The step of generating corresponding invalid functional dependencies for each tuple pair includes: generating the invalid functional dependencies based on the replacement dataset.
3. The method according to claim 1, characterized in that, The process of sampling all tuple sets to obtain tuple pairs includes: Each partition is treated as a queue unit, and the tuple set in each queue unit is sorted according to the efficiency value. The efficiency value is positively correlated with the number of tuples contained in the corresponding tuple set. When sampling the set of tuples in any queue unit, the tuple sets in the queue unit are sampled in descending order of efficiency value to obtain tuple pairs.
4. The method according to claim 1, characterized in that, The process of sampling all tuple sets to obtain tuple pairs includes: In the first round of sampling, sliding sampling is performed on all tuple sets according to the initial sampling window size; or, in subsequent rounds of sampling, the sampling window is increased based on the previous round, and a new round of sliding sampling is restarted for all tuple sets; wherein, the tuples at both ends of the sliding window are used to generate unary pairs; If the growth rate of the invalid functional dependency reaches a first preset lower limit threshold after the current round of sliding sampling, then sampling ends; otherwise, proceed to the next round of sliding sampling.
5. The method according to claim 4, characterized in that, The process of sampling all tuple sets to obtain tuple pairs includes: If the growth rate of the effective functional dependency is not found to have reached the second preset lower threshold after the current round of sliding sampling, then proceed to the next round of sliding sampling until the growth rate of the effective functional dependency reaches the preset threshold after any round of sliding sampling.
6. The method according to claim 1, characterized in that, The method further includes: identifying unnecessary invalid functional dependencies, wherein the left-hand side of the unnecessary invalid functional dependencies is included in the left-hand side of other invalid functional dependencies, and the right-hand side of the unnecessary invalid functional dependencies is the same as the right-hand side of other functional dependencies; Adding candidate fields to the left-hand side set of any invalid functional dependency includes: adding candidate fields to the left-hand side set of any invalid functional dependency that is distinct from the unnecessary invalid functional dependency.
7. The method according to claim 1, characterized in that, Also includes: If the functional dependency to be verified belongs to the set of invalid functional dependencies or a generalization of invalid functional dependencies in the set, then other candidate fields that are different from the candidate fields that have been added are added to the left-hand side set of the functional dependency to be verified until the functional dependency to be verified is determined to be a valid functional dependency.
8. The method according to claim 1, characterized in that, Also includes: Identify the sensitive fields that the user has marked in the at least two fields; The target functional dependency is determined from the effective functional dependencies, wherein the fields contained in the left-hand side set of the target functional dependency belong to the sensitive fields; If the fields contained in the right-hand side set of the objective function dependency are different from the sensitive fields, then the fields contained in the right-hand side set of the objective function dependency are determined to be potential sensitive fields.
9. A reasoning method based on sensitive fields of functional dependency, characterized in that, The method includes: Obtain functional dependencies generated based on a dataset, the functional dependencies being used to characterize the relationships between fields in the dataset; wherein the functional dependencies are determined by the method according to any one of claims 1-8; Identify the sensitive fields that the user has marked in the dataset; Determine the target functional dependency from the functional dependencies, wherein the fields contained in the left-hand side set of the target functional dependency belong to the sensitive fields; If the fields contained in the right-hand side set of the objective function dependency are different from the sensitive fields, then the fields contained in the right-hand side set of the objective function dependency are determined to be potential sensitive fields.
10. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor implements the method as described in any one of claims 1-9 by executing the executable instructions.
11. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-9.
Citation Information
Patent Citations
Increment computing method for minimal functional dependencies
CN104699761A
Large-scale distributed function dependency discovery method
CN108595624A