A function dependency approximation discovery method suitable for big data

Abstract

The application discloses a function dependency approximation discovery method suitable for big data, and comprises the following steps: clustering and partitioning tuples in original data set according to attributes, replacing all attribute values in the original data set with cluster serial numbers in the partition to obtain a new data set; creating a priority queue for the new data set by taking the clustering cluster as a unit, sampling tuple pairs in the priority queue, generating non-function dependencies by comparing the attribute value differences of the tuple pairs, and adding the non-function dependencies to a non-function dependency set; taking the growth rate of the set as a judgment standard to control the termination of the sampling process; structurally simplifying the non-function dependency set; verifying the legality of the current and subsequent function dependency sets by using the non-function dependency set, deleting the non-existing function dependencies, adding new attributes to the left part of the function dependencies to obtain new candidate function dependencies, and re-adding the new candidate function dependencies to a function dependency result set; and taking the growth rate of the result set as a judgment standard to control the flow direction. The method has small calculation cost.

Description

Technical Field

[0001] This invention belongs to the field of databases, specifically relating to a method for approximate discovery of functional dependencies applicable to big data. Background Technology

[0002] Functional dependencies have very important applications in the database field. Since the concept of functional dependencies was proposed (Codd EF. Further normalization of the database relational model[J].Data basesystems,1972,6:33–64)(Weddell G E. Reasoning about functional dependencies generalized for semantic data models[J].ACM Transactions on Database Systems(TODS),1992,17(1):32–64), the discovery methods of functional dependencies have been a hot research topic.

[0003] Algorithms for discovering functional dependencies can be divided into exact discovery algorithms and approximate discovery algorithms. Current traditional functional dependency discovery algorithms are mainly based on exact discovery algorithms, and much research has been conducted on discovering accurate and complete sets of functional dependencies, such as (Huhtala Y, J, Porkka P, et al. TANE: An efficient algorithm for discovering functional and approximate dependencies [J]. The computer journal, 1999, 42(2): 100-111) (Novelli N, Cicchetti R. Fun: An efficient algorithm for mining functional and embedded dependencies [A]. International Conference on Database Theory [C]. 2001: 189-203) (Yao H, Hamilton H J, Butz CJ. FD / spl I.bar / Mine: discovering functional dependencies in a database using equivalences [A]. 2002 IEEE International Conference on Data Mining, 2002. Proceedings. [C]. 2002: 729-732) (Lopes S, Petit J-M, Lakhal L. Efficient discovery of functional dependencies and armstrong relations [A]. International Conference on Extending Database Technology [C]. 2000: 350-364) and so on. According to the principle of the algorithm, the accurate discovery algorithm is mainly divided into three categories: lattice traversal algorithm, dependency induction algorithm and hybrid algorithm. Among them, the lattice traversal algorithm, such as (Huhtala Y, J, Porkka P, et al. TANE: An efficient algorithm for discovering functional and approximate dependencies [J]. The computer journal, 1999, 42(2): 100-111), the left part of the function dependency is unitized into a lattice, and all lattices constitute the function dependency space, and the operation on the function dependency is abstracted into the traversal, relationship check, deletion and addition of the lattice;

[0004] The inductive algorithm is represented by (Flach P A, Savnik I. Database dependency discovery: a machine learning approach[J]. AI communications, 1999, 12 (3) : 139-160.), which obtains all non-functional dependencies from the data set first, and then derives all functional dependencies from the non-functional dependencies.

[0005] The hybrid algorithm such as (apenbrock T, Naumann F. A hybrid approach to functional dependency discovery[A]. Proceedings of the 2016 International Conference on Management of Data[C]. 2016: 821-833) adopts the method of multiple module combination and loop control to discover accurate functional dependency results.

[0006] The above functional dependency algorithms have the problems of large calculation overhead and low efficiency when processing big data. SUMMARY

[0007] In view of the above, the purpose of the present application is to provide a functional dependency approximation discovery method suitable for big data, which can effectively reduce the calculation overhead of functional dependency and greatly improve the efficiency of functional dependency discovery.

[0008] To achieve the above application purpose, the embodiment also provides a functional dependency approximation discovery method suitable for big data, comprising the following steps:

[0009] The preprocessing stage: according to the attribute, the tuples in the original data set are clustered and partitioned, and the cluster serial number in the partition is used to replace all attribute values in the original data set to obtain a new data set;

[0010] The sampling and non-functional dependency construction stage: a priority queue is created for the new data set in the form of clustering cluster, a sliding window with dynamic increase is used to sample the tuple pairs in the priority queue, and the non-functional dependencies are generated by comparing the differences of attribute values of the tuple pairs and added to the non-functional dependency set;

[0011] The optimization stage: by checking the relationship between the non-functional dependencies, if a certain non-functional dependency is a generalization form of any other non-functional dependency, the certain non-functional dependency is deleted from the non-functional dependency set to realize the structural simplification of the non-functional dependency set;

[0012] The flip construction function dependent set phase: the legality of all function dependencies in the current function dependent set is verified by non-function dependent set, candidate attributes are added in the left part of the function dependency which is not true, and the candidate function dependency is obtained. If the candidate function dependency is not an arbitrary non-function dependency and its generalization, it is determined that the candidate function dependency is true, and it is added to the function dependent set.

[0013] Preferably, the original data set is clustered and partitioned according to the attribute, and the cluster serial number in the partition is used to replace all attribute values in the original data set, including:

[0014] In attribute units, all tuples are clustered based on attribute values, tuples with the same attribute value are clustered into the same cluster, and all cluster clusters belonging to the same attribute are divided into a partition;

[0015] All cluster clusters in the partition are numbered in ascending order to obtain the cluster serial number, and the attribute value of each tuple in the cluster cluster is replaced by the cluster serial number of the cluster cluster.

[0016] Preferably, when creating a priority queue for a new data set, the cluster cluster with a tuple number less than or equal to 1 is deleted from the new data set.

[0017] Preferably, the units stored in the priority queue are arranged in descending order according to the efficiency value, wherein the efficiency value is equal to the number of non-function dependencies generated in a single unit and the comparison times;

[0018] During sampling, the unit with the highest current efficiency value is selected from the head of the priority queue each time, and the tuples obtained by sampling from the unit are compared two by two. The attributes with the same attribute values jointly constitute the left part of the non-function dependency, and the attributes with different attribute values are used as the right part of the non-function dependency. Accordingly, the non-function dependency is generated, and the generated non-function dependency is added to the non-function dependency set.

[0019] Preferably, the priority queue is sampled using a dynamically increasing sliding window, including:

[0020] In the first round of sampling, a sampling window with an initial size of 2 is used to slide and select all tuples in the unit, and the tuples in the window are compared to generate non-function dependencies. In subsequent sampling rounds, the sliding window is increased, and the tuples at both ends of the window are selected to form tuple pairs for comparison to generate new non-function dependencies.

[0021] Whether the sampling is sufficient is determined by comparing the sampling efficiency value with the preset threshold value, and the corresponding process control: if the sampling efficiency value of the current round is higher than the preset threshold value, the next round of sampling will be entered; if the sampling efficiency value of the current round is lower than the preset threshold value, the sampling is ended, and the optimization stage is entered.

[0022] Preferably, the non-functional dependency set is stored in a binary search tree, and when a non-functional dependency is added, it is necessary to check whether it is an existing functional dependency or its generalization form in the current non-functional dependency set. The left part and the inclusion relationship are checked by means of the path structure of the binary search tree and the prefix tree, and after the check is passed, the current non-functional dependency is added to the current non-functional dependency set, and the storage structure is adjusted and optimized.

[0023] Preferably, by checking the relationship between non-functional dependencies, if a certain non-functional dependency is a generalization form of any other non-functional dependency, the certain non-functional dependency is deleted from the non-functional dependency set, comprising:

[0024] Split all non-functional dependencies in the non-functional dependency set, and split the original non-functional dependency into a corresponding number of single-attribute right part non-functional dependencies according to the right part attribute;

[0025] Classify all single-attribute right part non-functional dependencies according to the right part, create a corresponding number of binary search trees in units of attributes, take the left part of the non-functional dependency as the path, and add the non-functional dependency to the corresponding binary search tree;

[0026] In the adding process, it is necessary to check the association relationship between the current non-functional dependency to be added and all existing non-functional dependencies in the tree. If the non-functional dependency to be added is the same as or a generalization form of any existing non-functional dependency, the non-functional dependency to be added does not need to be added.

[0027] Preferably, the flipping and constructing the functional dependency set stage comprises:

[0028] Initialize the functional dependency set, and create a left part as an empty set and a right part as a functional dependency of each individual attribute, and add it to the functional dependency set;

[0029] In the flipping process, each time a non-functional dependency is taken out from the non-functional dependency set, the right part of the taken non-functional dependency is used to perform a legality check on the functional dependencies in the functional dependency set that have the same right part. If the left part of the functional dependency to be verified is the same as or a subset of the left part of the non-functional dependency, the functional dependency to be verified is not established, and it is deleted;

[0030] For the non-established functional dependency, a candidate attribute is added to its left part to make it a candidate functional dependency, and it is added to the functional dependency set, wherein the candidate attribute needs to satisfy: not being a right part attribute and not appearing in the left part attribute set.

[0031] Preferably, the function dependency set is stored in a binary search tree, and when a function dependency is added, it is necessary to check whether it is an existing function dependency or a generalization form of the existing function dependency in the current function dependency set, and the left part and the inclusion relationship are checked by means of the path structure of the binary search tree and the prefix tree, and after the check is passed, the current function dependency is added to the current function dependency set, and the storage structure is adjusted and optimized.

[0032] Preferably, the method further comprises:

[0033] Whether the function dependency set is complete is judged by comparison between the defined set growth rate and the preset threshold, and the corresponding flow control is performed: if the set growth rate of the current round is higher than the preset threshold, the previous sampling and non-function dependency generation phase is returned to, and the sampling and non-function dependency generation are continued; if the set growth rate of the current round is lower than the preset threshold, the whole flow is ended, and the current function dependency combination is returned as a result.

[0034] The technical concept of the application is that: the original data set is preprocessed first, converted into a partition set with clusters as units, and the cluster serial number is used to replace the original attribute value, thereby reducing the required storage space while keeping the information integrity, and reducing the calculation amount for the subsequent steps. A priority queue is constructed, a sliding window is used to sample tuples, a corresponding non-function dependency is generated for each tuple, and is added to the non-function dependency set, and the sampling units are arranged in descending order according to the efficiency value, so that the tuples with higher efficiency values can be preferentially sampled. When the non-function dependency set is constructed, all the non-function dependencies obtained by sampling are split into non-function dependencies with the same left part and a single attribute right part according to the right part attribute, and then all the non-function dependencies are classified according to the right part attribute, and the non-function dependencies with the same right part are added to the same binary search tree, thereby providing a basis for parallelization in the conversion stage. When the function dependency set is converted and constructed, the initial function dependencies are verified and refined in parallel according to the differences in the right part attribute. First, the non-function dependencies are taken out from the non-function dependency set corresponding to the same right part attribute, and if the function dependency is the same as the non-function dependency or a generalization form of the non-function dependency, the function dependency is not established, and a candidate attribute that appears in both the left part and the right part is added to the left part, so that the candidate function dependency that is not established becomes a candidate function dependency that may be established. After the conversion of all attributes is completed, the growth rate of the function dependency set in the current round is calculated, and if it is lower than the preset threshold, it represents that the current function dependency set tends to be complete, and the whole algorithm flow is terminated, and the current function dependency set is output as a result set; if it is higher than the preset threshold, the sampling stage is returned to, and the algorithm flow is continued. Through the loop control, the accuracy and integrity of the algorithm result are guaranteed.

[0035] Compared with the prior art, the application has at least the following beneficial effects:

[0036] By combining the partition set and the dependent induction method, the computing cost of the function dependency can be effectively reduced, and the efficiency of the function dependency discovery algorithm can be greatly improved. The double-loop control method based on the multi-priority queue is first proposed in the application. The number of tuple pairs required by the traditional dependent induction method is greatly reduced, while the accuracy and completeness of the results are guaranteed. The effect is better than that of the existing method, and the application has the advantages of strong universality, simple use and the like. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0038] Figure 1 is a system architecture diagram of the approximate discovery method of function dependency in the method;

[0039] Figure 2 is a flowchart of the approximate discovery method of function dependency in the method;

[0040] Figure 3 is a schematic diagram of generating a partition set in the method;

[0041] Figure 4 is a schematic diagram of constructing a sampling priority queue in the method;

[0042] Figure 5 is a schematic diagram of constructing a non-function dependency corresponding binary search tree in the method;

[0043] Figure 6 is a schematic diagram of constructing a candidate function dependency corresponding binary search tree in the method. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the protection scope of the present application.

[0045] Figure 1 is a system architecture diagram of the approximate discovery method of function dependency provided by the exemplary embodiments, as shown in Figure 1 , the system architecture diagram comprises a database 1, a database 2 and a server 3; wherein the database 1 is used to store and provide the original data set, the server 3 is used to process the original data set provided by the database 1, and the function dependency set is inferred, and the database 2 is used to store the function dependency set processed by the server 3.

[0046] In an embodiment, functional dependency refers to that one attribute set can uniquely determine another attribute set, or in other words, the other attribute set depends on the attribute set. In the field of database, the definition of functional dependency is as follows: in an instance r of schema R, when and A∈R, functional dependency X→A means that in the instance r, if t1[X]=t2[X], then t1[A]=t2[A] for any two tuples t1, t2.

[0047] Taking the scenario of a course table as an example, the course name can be determined by the course number, and according to the definition of functional dependency, the course number can be said to determine the course name, or the course name depends on the course number, which can be represented as "{course number}->{course name}". However, the course number cannot be determined by the course name, because there can be multiple identical courses, one course name can correspond to multiple course numbers, and the course name cannot determine the course number, i.e., the course number does not depend on the course name, so "{course name}->{course number}" is not true, which can also be called non-functional dependency.

[0048] In an embodiment, database 1 and database 2 can be any type of relational database, and the data set has several attributes and tuples, and all different attribute values under each attribute constitute the value domain space corresponding to the attribute, and each tuple is composed of attribute values under all attributes. The data maintained in database 1 exists in the form of a table, and each record in the data set is a row of data or a tuple in the table. For example, the storage form of the course table in database 1, the course number, course name and course time and other information of each course in the course table are a specific tuple, and the columns of the table are the attribute information corresponding to each attribute in the data set. Database 2 also maintains a table for storing the functional dependency set, and the columns are the functional dependency left part, the functional dependency right part, the generation time, the source data set, the running time and the like.

[0049] Server 3 is a physical server containing an independent host or a virtual server carried by a host cluster. During operation, server 3 obtains the original data set from database 1, calculates the original data set by the functional dependency approximation discovery method provided by the present application, derives the functional dependency set, and stores it in database 2.

[0050] The embodiment also provides a data set as shown in Table 1 for detailed introduction of the functional dependency approximation discovery method provided by the present application.

[0051] Table 1

[0052] Tuple number Name Gender Age Blood pressure Medicine 1 Li 1 Female 60 High Medicine A 2 Wang 2 Male 32 Low Medicine C 3 Zhang 3 Female 28 Normal Medicine X 4 Zhao 4 Female 49 Low Medicine Y 5 Zhou 5 Female 32 Normal Medicine X 6 Huang 6 Female 49 Normal Medicine X 7 Chen 7 Female 32 Low Medicine C

[0053] The server 3 reads the original data from the database 1 and stores it in a table form locally. The read-in table is shown in Table 1. The data set has 7 records, i.e. Table 1 contains 7 tuples, which correspond to tuples 1, 2, 3, 4, 5, 6 and 7 respectively. The table also contains 5 different attributes, which are name, gender, age, blood pressure and medicine. The respective attribute values of the 7 tuples on the 5 attributes are as follows: the attribute value of tuple 1 on the name attribute is Li Yi, the attribute value of tuple 1 on the gender attribute is female, the attribute value of tuple 1 on the age attribute is 60, the attribute value of tuple 1 on the blood pressure attribute is high, and the attribute value of tuple 1 on the medicine attribute is medicine A. The attribute values of different tuples on the same attribute can be the same, such as the attribute values of tuple 1 and tuple 3 on the gender attribute are both female. The attribute values of different tuples on the same attribute can also be different, such as the attribute value of tuple 2 on the age attribute is 32, and the attribute value of tuple 4 on the age attribute is 49.

[0054] Figure 2 is a flowchart of a function dependency approximation discovery method suitable for big data provided by an exemplary embodiment, as shown in Figure 2 The function dependency approximation discovery method includes the following steps, as shown in Table 1.

[0055] S110, preprocessing stage: clustering and partitioning the tuples in the original data set according to the attributes, and replacing all attribute values in the original data set with cluster serial numbers in the partition to obtain a new data set.

[0056] Specifically, the original data set is clustered and partitioned according to the attributes, and all attribute values in the original data set are replaced with cluster serial numbers in the partition, including: clustering all tuples based on attribute values in units of attributes, tuples with the same attribute value are clustered into the same cluster, and all cluster clusters belonging to the same attribute are divided into a partition, i.e. each attribute corresponds to a respective partition, and all partitions constitute a partition set; incrementally numbering all cluster clusters in the partition to obtain cluster serial numbers, and replacing the attribute values of the tuples in each cluster cluster with the cluster serial numbers of the cluster cluster to which they belong to complete preprocessing.

[0057] Taking the data in Table 1 as an example, there are 5 different attributes in the table, which correspond to 5 partitions respectively, and each partition has multiple clusters, i.e., tuple sets. All tuples in the same cluster have the same attribute value on the attribute, and the attribute values of tuples in different clusters on the attribute are certainly different. Taking the attribute age as an example, the attribute value domain size on the attribute is 4, i.e., there are 4 different attribute values: 60, 32, 28, and 49. The tuple with the value 60 on the age attribute is tuple 1, the tuple with the value 32 on the age attribute is tuple 2, 4, and 7, the tuple with the value 28 on the age attribute is tuple 3, and the tuple with the value 49 on the age attribute is tuple 4 and 6. Therefore, there are four clusters on the age attribute: {1}, {2, 5, 7}, {3}, and {4, 6}, and the corresponding 1 partition is {{1}, {2, 5, 7}, {3}, {4, 6}}. Through the above partition operation, it can be ensured that in the next stage of the sampling process, the attribute values of two tuples compared at least on one attribute are the same, and a non-functional dependency can be generated, thereby improving the sampling efficiency. The partition set of Table 1 is shown in Table 2. Figure 3

[0058] After obtaining the partition, the clusters under the partition are numbered, and the first cluster of each partition is numbered 0, and the cluster sequence numbers of subsequent clusters are incremented. The cluster sequence number of a tuple under the attribute partition is used to replace the attribute value of the tuple on the attribute, and after all the replacement is completed, a new data set is obtained, as shown in Table 2.

[0059] Table 2

[0060]

[0061]

[0062] S120, a sampling non-functional dependency construction stage: creating a priority queue for the new data set in units of attributes, sampling tuple pairs in the priority queue using a dynamically increasing sliding window, generating a non-functional dependency by comparing the differences in attribute values of the tuple pairs, and adding the non-functional dependency to a non-functional dependency set.

[0063] In the embodiment, it needs to be clear that the process of generating a non-functional dependency from two tuples in a tuple pair is: comparing whether the attribute values of the two tuples on all attributes are the same, and all the same attributes constitute a consistent set, and different attributes constitute a difference set. Each time, an attribute is taken out from the difference set as the right part, and all attributes of the consistent set together constitute the left part, and then a non-functional dependency is obtained. As can be seen from the above, if the comparison of the two tuples generates a non-functional dependency, the attribute values of the two tuples on at least one attribute are the same.

[0064] ​In the embodiment, since two tuples are needed to generate a non-functional dependency, when creating the priority queue for the new data set, the cluster whose number of tuples is less than or equal to 1 is deleted from the new data set. Specifically, when constructing the priority queue, the cluster in the partition set is taken as a unit, and is added to the priority queue. The tuple pair is taken out in a sliding window manner. It should be noted that the size of the sliding window is at least 2, so the cluster with a size of 1 in the partition set needs to be excluded. As shown in FIG. 9, the size of all clusters in the name is 1, so all clusters corresponding to the name cannot be added to the sampling queue; in the partition corresponding to the gender, the cluster {2} does not meet the requirement and cannot be added to the queue; in the partition corresponding to the age, the cluster {1} and the cluster {3} do not meet the requirement and cannot be added to the queue; in the partition corresponding to the blood pressure, the cluster {1} does not meet the requirement and cannot be added to the queue; in the partition corresponding to the medicine, the cluster {1} and the cluster {4} do not meet the requirement and cannot be added to the queue. After the above screening is completed, the obtained priority queue is as shown in FIG. 10. Figure 3 Figure 4

[0065] In the embodiment, the units stored in the priority queue are arranged in descending order according to the efficiency value, where the efficiency value is equal to the ratio of the number of non-functional dependencies generated in a single unit to the comparison times, and the efficiency value is updated after each round of sampling. When sampling, the unit with the highest current efficiency value is selected from the head of the priority queue each time, and the tuple pairs sampled from the unit are compared two by two. The attributes with the same attribute values jointly constitute the left part of the non-functional dependency, and the attributes with different attribute values jointly constitute the right part of the non-functional dependency. According to this, the non-functional dependency is generated, and the generated non-functional dependency is added to the non-functional dependency set.

[0066] In the embodiment, a dynamically increasing sliding window is used to sample the tuple pairs in the priority queue, including: in the first round of sampling, a sampling window with an initial size of 2 is used to slide and select all tuples in the unit, and the tuple pairs in the window are compared to generate a non-functional dependency; in subsequent sampling rounds, the sliding window is increased on the basis of the last round, and the tuples at both ends of the window are selected to form a tuple pair for comparison when sampling, a new non-functional dependency is generated and added to the non-functional dependency set, and if the non-functional dependency is not any non-functional dependency or a generalized form of the non-functional dependency already existing in the combination, the non-functional dependency is valid; whether the sampling is sufficient and the corresponding flow control are determined by comparing the sampling efficiency value in the current round with a preset threshold value: if the sampling efficiency value in the current round is higher than the preset threshold value, the next round of sampling is entered; if the sampling efficiency value in the current round is lower than the preset threshold value, the sampling is ended, and a refinement optimization phase is entered.

[0067] ​​Specifically, in the first round of sampling, the clusters are traversed and sampled in the default order in the priority queue. First, the size of the sliding window is initialized to 2, and the cluster {1, 3, 4, 5, 6, 7} in the gender is taken out, and the sliding window moves from the left to the right, and each time the tuple at the two ends of the window is sampled. The first sampling gets the tuple pair {1, 3}, at this time the window moves one grid to the right, and the second sampling gets {3, 4}; the window moves one grid to the right again, and the sampling gets {4, 5}; the window moves one grid to the right again, and the sampling gets {5, 6}; the window moves one grid to the right again, and the sampling gets {6, 7}. At this time, the right end of the window has reached the right end of the cluster, and the sampling of this cluster in this round is completed, and the next cluster {2, 5, 7} in the priority queue is taken out for the same sampling operation, and after the sampling is completed, the next cluster {4, 6} is taken out. When the traversal and sampling of all cluster units in the queue are completed, the final sampling result {1, 3}, {3, 4}, {4, 5}, {5, 6}, {6, 7}, {2, 5}, {5, 7}, {4, 6}, {2, 4}, {4, 7}, {3, 5}, {2, 7} is obtained, a total of 12 tuple pairs.

[0068] After obtaining the tuple pair, the attribute values of the two tuples in the tuple pair can be compared according to the above non-functional dependency generation method to generate non-functional dependencies. Taking the first tuple pair {1, 3} as an example, the attribute values of tuple 1 and tuple 3 in the gender attribute are the same, both being "female", and the attribute values in the other four attributes are different. Therefore, the consistent set of tuple 1 and tuple 3 is {gender}, and the difference set is {name, age, blood pressure, medicine}. Taking the first attribute "name" from the difference set as the right part and the consistent set {gender} as the left part, a non-functional dependency {gender}→{name} is formed. Similarly, the attributes in the remaining three difference sets are constructed, and a total of four non-functional dependencies {gender}→{name}, {age}→{name}, {blood pressure}→{name}, and {medicine}→{name} are generated. The consistent set and difference set of the second tuple pair {3, 4} are the same as those of {1, 3}, that is, the non-functional dependency set is the same as that generated by {1, 3}, and the repeated non-functional dependencies do not need to be added again. The third tuple pair {4, 5} is the same. The consistent set of the fourth tuple pair {5, 6} is {gender, blood pressure, medicine}, and the difference set is {name, age}, and two non-functional dependencies {gender, blood pressure, medicine}→{name} and {gender, blood pressure, medicine}→{age} can be generated. The fifth tuple pair {6, 7} is the same as {1, 3}.

[0069] After generating all the non-functional dependencies, the efficiency value corresponding to each cluster in this round needs to be calculated, and the efficiency value is defined as the ratio of the generated non-functional dependencies and the number of tuple comparisons. Taking the cluster corresponding to the partition of the gender attribute as an example, the tuple pair {1, 3} generates 3 non-functional dependencies, the tuple pair {3, 4}, the tuple pair {4, 5}, the tuple pair {6, 7} generates 0 non-functional dependencies; the tuple pair {5, 6} generates 2 non-functional dependencies. The cluster {1, 3, 4, 5, 6, 7} performs 5 tuple pair comparisons and generates 5 different non-functional dependencies, and the efficiency value is 1. Similarly, the efficiency values of all other clusters are calculated, and the clusters in the priority queue are arranged in descending order according to the efficiency values, so that the cluster with the highest efficiency value can be preferentially sampled, that is, the non-functional dependencies are generated as quickly as possible.

[0070] In S130, a simplified optimization phase: by checking the relationship between non-functional dependencies, if a certain non-functional dependency is a generalization form of any other non-functional dependency, the certain non-functional dependency is deleted from the non-functional dependency set to realize the structural simplification of the non-functional dependency set.

[0071] In the embodiment, the non-functional dependency set is stored in a binary search tree, and when adding a non-functional dependency, it needs to be checked whether it is an existing functional dependency or a generalization form in the current non-functional dependency set. The left part and the inclusion relationship are checked by means of the path structure of the binary search tree and the prefix tree, and after the check is passed, the current non-functional dependency is added to the current non-functional dependency set, and the optimized storage structure is adjusted.

[0072] When checking the association relationship between non-functional dependencies, it is necessary to first determine that the right parts of the two non-functional dependencies to be compared are the same, and if the left part of the former is a subset of the left part of the latter, the former is called the generalization form of the latter, and the latter is called the refinement form of the former.

[0073] In the embodiment, the specific process of structurally simplifying the non-functional dependency set is as follows: all non-functional dependencies in the non-functional dependency set are split, and the original non-functional dependency is split into non-functional dependencies with a single attribute right part according to the right part attribute; all single attribute right part non-functional dependencies are classified according to the right part, and a corresponding number of binary search trees are created according to the attribute, and the non-functional dependency is added to the corresponding binary search tree with the left part of the non-functional dependency as the path; during the adding process, the association relationship between the current non-functional dependency to be added and all non-functional dependencies in the tree needs to be checked, and if the non-functional dependency to be added is the same as or a generalization form of any non-functional dependency, the non-functional dependency to be added does not need to be added.

[0074] Based on the above structured reduction process, taking the non-functional dependency set generated in the first round of sampling as an example, {gender}→{name} is a generalization of {gender, blood pressure, drug}→{name}, and can also be called that {gender, blood pressure, drug}→{name} contains {gender}→{name}. In the target non-functional dependency set to be constructed, only the largest non-functional dependency needs to be retained, so {gender}→{name} is an unnecessary non-functional dependency, and {gender, blood pressure, drug}→{name} can be added to the set. Similarly, {blood pressure}→{name}, {drug}→{name} are also generalizations of {gender, blood pressure, drug}→{name}, that is, they are contained by {gender, blood pressure, drug}→{name}. To determine whether there is a containing relationship between non-functional dependencies, two conditions need to be determined: whether the right part is the same and whether the left part contains the relationship.

[0075] In order to facilitate the inspection of the containing relationship between such non-functional dependencies, a binary search tree is used to store the non-functional dependencies, and the five attributes {name, gender, age, blood pressure, drug} are simplified as {A, B, C, D, E}. In the construction process, a binary search tree is established for each attribute. Taking attribute A as an example, the construction process is as follows Figure 5{C, D, E}→ {A}, {B, D, E}→ {A}, {B, C}→ {A}, {C, E}→ {A}. First, {C, D, E}→ {A} is added to the binary search tree, and since the current root node is empty, the root node is initialized to the first attribute C of the left part, the left child of the node must not contain the attribute represented by the node, and the right child of the node must contain the attribute represented by the node. CDE is added to the right child node of the root node, and the left node is initialized to an empty set node. When {B, D, E}→ {A} is added to the binary search tree, the first passing root node is checked, and since the left part BDE does not contain the attribute C represented by the current node, the left child node of the current node is reached. Since the left child node is empty, BDE is filled into the node, and the parent node, i.e., the root node, is returned. The attribute intersection DE between the left child and the right child is checked, and the intersection DE is stored in the C node to facilitate subsequent path judgment. When {B, C}→ {A} is added to the binary search tree, the first passing root node is checked. BC contains the attribute C represented by the current node, and the right child of the current node is traversed. Since the next node is the leaf node CDE, the current attribute set BC to be added cannot be added to the leaf node, so a new non-leaf node representing the attribute E is created. BC does not contain the attribute E, which is added to the current node as the left child. CDE contains the attribute E, which is added to the current node as the right child. The left and right child nodes are checked, and both contain the attribute C, so C is stored as the intersection information on the node E.

[0076] In the phase of reversing the construction of the function dependency set, the legality of all function dependencies in the current function dependency set is verified using the non-function dependency set. Candidate attributes are added in the left part of the function dependency that does not exist, to obtain a candidate function dependency to be verified. If the candidate function dependency is not an arbitrary non-function dependency and its generalization form, it is determined that the candidate function dependency is established, and is added to the function dependency set.

[0077] In the embodiment, the phase of reversing the construction of the function dependency set specifically includes:

[0078] The initialization function depends on the set, and creates the left part as an empty set and the right part as each individual attribute function dependency, which is added to the function dependency set; in the flipping process, each time a non-function dependency is taken out from the non-function dependency set, the function dependency in the function dependency set with the same right part is verified for legality according to the right part of the taken-out non-function dependency, if the left part of the function dependency to be verified is the same as or a subset of the left part of the non-function dependency, the function dependency to be verified is not established, and the remaining function dependency passes the verification and is identified as legal in the current stage; for the non-established function dependency, a candidate attribute is added to the left part to make it a candidate function dependency, and is added to the function dependency set, wherein the candidate attribute needs to meet: not being a right part attribute and not appearing in the left part attribute set.

[0079] In the embodiment, before the candidate function dependency is added to the function dependency set, it also needs to be judged, and the specific judgment can be: if the candidate function dependency is a refinement form of the known established function dependency, the candidate function dependency is established; if the candidate function dependency is a generalization form of the known non-function dependency, the candidate function dependency is not established.

[0080] In the embodiment, the function dependency set is stored by using a binary search tree, and when adding a function dependency, it is necessary to check whether it is an existing function dependency or a generalization form of the function dependency in the current function dependency set, and the left part and the containing relationship are checked by means of the path structure of the binary search tree and the prefix tree, and after the check passes, the current function dependency is added to the current function dependency set, and the storage structure is adjusted and optimized.

[0081] When detecting the relationship between the function dependencies, it is necessary to first determine that the right parts of the two function dependencies to be compared are the same, if the left part of the former is a subset of the left part of the latter, the former is called the generalization form of the latter, and the latter is called the refinement form of the former.

[0082] In the embodiment, the set growth rate and the preset threshold are also defined, and the completeness of the function dependency set is judged by comparing the set growth rate and the preset threshold, and the corresponding flow control is: if the set growth rate of the current round is higher than the preset threshold, return to the last sampling and build non-function dependency stage, continue to sample and generate non-function dependencies; if the set growth rate of the current round is lower than the preset threshold, end the whole process and return the current function dependency combination as the result.

[0083] Specifically, taking the non-function dependency set corresponding to the attribute "name", that is, the non-function dependency set corresponding to the attribute A as an example, it is converted to Figure 5 It can be seen that the non-function dependency set contains three non-function dependencies {B, D, E}→{A}, {B, C}→{A}, {C, D, E}→{A}. Before conversion, a function dependency set is first initialized, which only contains one function dependency and a corresponding binary search tree is created. During the conversion stage, the first non-functional dependency {B, D, E}→{A} is taken out from the non-functional dependency set, and the relationship between the non-functional dependency {B, D, E}→{A} and all functional dependencies in the functional dependency set is checked, and it is found that {B, D, E}→{A} is contained in the non-functional dependency {B, D, E}→{A}, and it is determined that {B, D, E}→{A} is not true, and it is deleted from the candidate set, and candidate attributes B, C, D, and E are added, wherein {B}→{A}, {D}→{A}, and {E}→{A} are all contained in the non-functional dependency {B, D, E}→{A}, so it can be determined that {B}→{A}, {D}→{A}, and {E}→{A} are not true, and they are deleted, and the functional dependency set only has the functional dependency {C}→{A}; the second non-functional dependency {B, C}→{A} is taken out from the non-functional dependency set, and it is checked with the {C}→{A} in the functional dependency set, and it is found that {C}→{A} is contained in {B, C}→{A}, and {C}→{A} is not true, and candidate attributes D and E are added to generate new candidate functional dependencies {C, D}→{A} and {C, E}→{A}, which are added to the functional dependency set after verification. Finally, the non-functional dependency {C, D, E}→{A} is taken out, and it is checked with all functional dependencies in the functional dependency set, and it is found that {C, D}→{A} and {C, E}→{A} are both contained in the non-functional dependency {C, D, E}→{A}, and it is determined that they are not true, and they are deleted from the functional dependency set, and candidate attributes are added to generate new candidate functional dependencies {B, D, E}→{A} and {B, C, E}→{A} which are added to the functional dependency set, and the complete process is shown in Figure 6 .

[0084] The functional dependency approximate discovery method provided in the above embodiment is suitable for data cleaning, data integration, database query optimization, and database analysis and design.

[0085] The specific embodiments described above have described the technical solutions and beneficial effects of the present application in detail, and it should be understood that the above description is only the most preferred embodiment of the present application, and is not used to limit the present application, and any modification, supplement, and equivalent replacement made within the principle range of the present application should be included in the protection range of the present application.

Claims

1. A method for approximate discovery of functional dependencies suitable for big data, characterized in that, The functional dependency refers to the functional dependency relationship in the database, and includes the following steps: Preprocessing stage: Cluster and partition the tuples in the original dataset according to the attributes, and replace all attribute values ​​in the original dataset with the cluster index in the partition to obtain a new dataset; The sampling and construction of non-functional dependencies stage: a priority queue is created for the new dataset with clusters as the unit. A dynamically increasing sliding window is used to sample tuple pairs in the priority queue. Non-functional dependencies are generated by comparing the differences in attribute values ​​of tuple pairs and added to the non-functional dependency set. Simplification and optimization phase: By examining the relationships between non-functional dependencies, if a certain non-functional dependency is a generalized form of any other non-functional dependency, then that non-functional dependency is removed from the set of non-functional dependencies, so as to achieve structured simplification of the set of non-functional dependencies. The reverse construction phase of the functional dependency set: use the non-functional dependency set to verify the validity of all functional dependencies in the current functional dependency set, add candidate attributes to the left-hand side of the invalid functional dependencies to obtain candidate functional dependencies to be verified. If the candidate functional dependency is not any non-functional dependency or its generalization, then the candidate functional dependency is considered valid and added to the functional dependency set. The sampling process involves using a dynamically increasing sliding window to sample tuple pairs in the priority queue. In the first round of sampling, a sampling window of initial size 2 is used to slide and select all tuples in the cell, comparing the tuple pairs within the window to generate non-functional dependencies. In subsequent sampling rounds, the sliding window is increased, and tuples at both ends of the window are selected during sampling to form tuple pairs for comparison, generating new non-functional dependencies. The sufficiency of sampling is determined by comparing the sampling efficiency value with a preset threshold, and corresponding process control is implemented: if the sampling efficiency value of the current round is higher than the preset threshold, the next round of sampling begins; if the sampling efficiency value of the current round is lower than the preset threshold, sampling ends, and the process enters the simplification and optimization phase.

2. The method for approximate discovery of functional dependencies for big data according to claim 1, characterized in that, The process of clustering and partitioning the original dataset based on attributes, and replacing all attribute values ​​in the original dataset with the cluster indices in the partitions, includes: Based on attribute values, all tuples are clustered, with tuples of the same attribute value clustered into the same cluster, and all clusters belonging to the same attribute are divided into one partition. The cluster numbers are obtained by incrementing the number of all clusters in the partition, and the attribute values ​​of the tuples in each cluster are replaced with the cluster number of the cluster to which they belong.

3. The method for approximate discovery of functional dependencies for big data according to claim 1, characterized in that, When creating a priority queue for a new dataset, clusters with fewer than 1 tuples are removed from the new dataset first.

4. The method for approximate discovery of functional dependencies for big data according to claim 1, characterized in that, The cells stored in the priority queue are sorted in descending order according to their efficiency values, where the efficiency value is equal to the ratio of the number of non-functional dependencies generated in a single cell to the number of comparisons. During sampling, the unit with the highest current efficiency value is selected from the priority queue each time. The tuple pairs sampled from the unit are compared pairwise. Attributes with the same attribute value together constitute the left part of the non-functional dependency, and attributes with different attribute values ​​are used as the right part of the non-functional dependency. Based on this, non-functional dependencies are generated and added to the non-functional dependency set.

5. The method for approximate discovery of functional dependencies for big data according to claim 1, characterized in that, The non-functional dependency set is stored using a binary search tree. When adding a non-functional dependency, it is necessary to check whether it is an existing functional dependency or a generalized form of it in the current non-functional dependency set. The left-hand side and containment relationship are checked by using the path structure of the binary search tree and prefix tree. After the check is passed, the current non-functional dependency is added to the current non-functional dependency set, and the storage structure is adjusted and optimized.

6. The method for approximate discovery of functional dependencies for big data according to claim 5, characterized in that, The step of examining the relationships between non-functional dependencies and removing a non-functional dependency from the set of non-functional dependencies if it is a generalization of any other non-functional dependency includes: Split all non-functional dependencies in the non-functional dependency set, and split the original non-functional dependencies into a corresponding number of non-functional dependencies with single attribute right-hand side based on the right-hand side attribute. Classify all non-functional dependencies on the right-hand side of a single attribute according to the right-hand side, create a corresponding number of binary search trees for each attribute, and add the non-functional dependencies to the corresponding binary search trees using the left-hand side of the non-functional dependency as the path. During the addition process, it is necessary to check the relationship between the non-functional dependency to be added and all existing non-functional dependencies in the tree. If the non-functional dependency to be added is the same as any existing non-functional dependency or its generalized form, then it is not necessary to add the non-functional dependency to be added.

7. The method for approximate discovery of functional dependencies for big data according to claim 1, characterized in that, The flipped constructor dependency collection phase includes: Initialize the function dependency set and create function dependencies with an empty left side and each individual attribute on the right side, then add them to the function dependency set; During the flipping process, each time a non-functional dependency is taken from the set of non-functional dependencies, the validity of functional dependencies in the set of functional dependencies with the same right-hand side is checked based on the right-hand side of the taken non-functional dependency. If the left-hand side of the functional dependency to be verified is the same as or a subset of the left-hand side of the non-functional dependency, then the functional dependency to be verified is invalid and is deleted. For a non-valid functional dependency, add a candidate attribute to its left side to make it a candidate functional dependency, and add it to the functional dependency set. The candidate attribute must satisfy the following conditions: it is not a right-hand attribute and does not appear in the left-hand attribute set.

8. The method for approximate discovery of functional dependencies for big data according to claim 1, characterized in that, The function dependency set is stored using a binary search tree. When adding a function dependency, it is necessary to check whether it is an existing function dependency or a generalized form of the current function dependency set. The left-hand side and containment relationship are checked by using the path structure of the binary search tree and prefix tree. After the check is passed, the current function dependency is added to the current function dependency set, and the storage structure is adjusted and optimized.

9. The method for approximate discovery of functional dependencies for big data according to claim 1, characterized in that, Also includes: The completeness of the functional dependency set is determined by comparing the growth rate of the defined set with a preset threshold, and the corresponding process control is implemented: if the growth rate of the set in the current round is higher than the preset threshold, the process returns to the previous sampling stage to build non-functional dependencies and continues to sample and generate non-functional dependencies; if the growth rate of the set in the current round is lower than the preset threshold, the entire process ends and the current functional dependency combination is returned as the result.

Images