A function dependency parallel mining method based on Bayesian network

Abstract

The application discloses a function dependency parallel mining method based on a Bayesian network in the technical field of data mining. The function dependency parallel mining method based on the Bayesian network comprises the following steps: function dependency modeling under the perspective of the Bayesian network; hierarchical sampling and search space reduction; parallel dependency discovery and result merging. The function dependency parallel mining method based on the Bayesian network solves the technical problems of high computational complexity, missed key dependencies and insufficient robustness of existing methods under large-scale and noisy data, reduces the computational and space complexity of function dependency mining under large-scale data, realizes effective identification of low-support high-confidence dependencies, improves the robustness under noisy data, and provides statistical theory support for function dependency mining.

Description

Technical Field

[0001] This invention relates to the field of database mining and database technology, and more specifically, to a parallel function dependency mining method based on Bayesian networks. Background Technology

[0002] Functional dependency (FD) is a core concept in relational database theory. It plays a key role in tasks such as data quality management, schema design, data cleaning, anomaly detection, and query optimization. Its mining goal is to automatically discover deterministic constraint relationships between attributes in a dataset, providing a basis for data modeling and knowledge extraction.

[0003] Existing rule-finding (FD) mining methods mainly include layer-by-layer search methods (such as TANE, FUN, FD_Mine, and DepMiner), depth-first search methods (such as DFD and FastFDs), hybrid methods (such as HyFD), approximate functional dependency discovery methods (such as CORDS), learning-based FD discovery methods (such as FDX and FDMc), Bayesian network-based rule discovery methods, and parallel rule mining methods. However, these methods generally suffer from the following drawbacks: high computational complexity and poor scalability on high-dimensional or large-scale datasets; unacceptable runtime and memory consumption when the number of attributes exceeds 60; support-based pruning strategies easily miss key dependencies with low support but high confidence; sensitivity to data noise, insufficient robustness, distortion of structure learning, and increased false detection rate; lack of a unified statistical modeling framework, making it difficult to characterize causal and conditional independence relationships between attributes; and imperfect parallelization design, failing to fully utilize multi-core or distributed resources. Among them, the HyFD and FDX methods, which are closest to this invention, suffer from the problems of missing rare important dependencies and decreased accuracy under high-dimensional noisy data, respectively.

[0004] Therefore, there is an urgent need in this field for a new method that can efficiently and accurately discover functional dependencies, especially low-support, high-confidence dependencies, in large-scale, noisy data environments. Summary of the Invention

[0005] The purpose of this invention is to provide a parallel functional dependency mining method based on Bayesian networks, which overcomes the technical problems of high computational complexity, missed key dependencies, and insufficient robustness of existing methods under large-scale, noisy data. It reduces the computational and space complexity of functional dependency mining under large-scale data, achieves effective identification of low-support, high-confidence dependencies, improves robustness in noisy data environments, and provides statistical theoretical support for functional dependency mining.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A parallel function dependency mining method based on Bayesian networks includes the following steps: S1. Functional Dependency Modeling from the Perspective of Bayesian Networks: Establishing a mathematical correspondence between Bayesian network structure learning and functional dependency mining, using conditional entropy as a quantitative indicator of attribute dependency, and determining that the functional dependency of attribute X on attribute Y has a high confidence level when the conditional entropy H(Y|X) of attribute X with respect to attribute Y approaches zero. S2. Hierarchical Sampling and Search Space Reduction: Obtain the original relation table, obtain the sample dataset from the original relation table based on the hierarchical sampling strategy, construct a star Bayesian network based on the sample dataset and extract the relevant attribute set of the target attribute, and then vertically divide the original relation table into multiple sub-tables; S3. Parallel Dependency Discovery and Result Merging: The parallel functional dependency mining algorithm PFMiner is adopted to execute the traditional sequential algorithm on multiple sub-tables simultaneously, and the mining results are merged and deduplicated to generate the minimum functional dependency set.

[0007] Furthermore, S1 includes the following steps: S11. Given a set of attributes, take any two attributes X and Y. If there is a directed edge in the Bayesian network from X to Y, it means that attribute X has a significant conditional constraint effect on the value of Y; if the edge does not exist, it means that the two are statistically independent. S12. Bayesian network-based learning methods all aim to train with the conditional entropy approaching 0, that is, when the conditional entropy... The Bayesian network structure learning method will only establish a directed edge between the two attributes X and Y when the value approaches 0. S13. Using conditional entropy as a quantitative indicator of attribute dependencies, automatic inference of functional derivation (FD) is achieved by judging the convergence of conditional entropy:

[0008] Therefore, when conditional entropy When X approaches zero, it means that X almost uniquely determines Y, and at this point, functional dependency exists. Established.

[0009] Furthermore, the Bayesian network structure learning method in S12 typically employs two types of implementation methods as follows: S121. Scoring-based learning method: This method involves defining a scoring function for the network structure and using a search algorithm to find the optimal structure. S122. Constraint-based learning method: Determine the causal direction relationship between nodes through conditional independence test.

[0010] Furthermore, the step S2, which involves obtaining the sample dataset from the original relation table based on a hierarchical sampling strategy, specifically includes the following steps:

[0011] Select the attribute with the most value categories from the original relation table as the representative attribute; The original relation table is divided into multiple layers based on the value range of the representative attribute; Random sampling is performed in each layer at a fixed ratio, and the sampling results of each layer are combined to form the sample dataset. The empirical estimate of the sample distribution is consistent with the overall distribution.

[0012] Furthermore, the construction of the star-shaped Bayesian network in S2 specifically includes: for each attribute A, during the Bayesian network structure learning process, defining a constraint set to prohibit the establishment of edge connections between attributes other than A, thereby forming a star-shaped Bayesian network centered on A.

[0013] Furthermore, the extraction of the relevant attribute set of the target attribute in S2 specifically includes: extracting the Markov blanket of the target attribute A from the star-shaped Bayesian network, and using the attributes in the Markov blanket as the relevant attribute set of the target attribute A.

[0014] Furthermore, S3 includes the following steps: S31. Distribute all sub-tables to multiple processing threads; S32. Each thread independently runs a function dependency mining algorithm from TANE or DFD on its assigned sub-table; S33. After all threads have completed, the system automatically merges the output of each thread, removes duplicate and redundant rules, and generates the final minimal set of functional dependencies.

[0015] Furthermore, the system's automatic merging of outputs from various threads and removal of duplicate and redundant rules in S33 specifically includes the following steps: S331. After all threads have completed their calculations, the system enters the result integration phase, where the result sets output by all threads are merged into a global set; S332. Perform deduplication on duplicate or mutually inclusive dependencies, retaining only the minimal set of functional dependencies; S333. Rules for removing duplicates and redundancies include: If two functional dependency rules exist, FD1: X1→Y and FD2: X2→Y, and X1 is a subset of X2, then FD2 is deleted. By adopting the above technical solution, the present invention has the following advantages: 1. This invention provides a parallel functional dependency mining method based on Bayesian networks, which has strong theoretical innovation. It establishes the numerical equivalence between functional dependencies and Bayesian networks for the first time, and uses the conditional entropy approaching 0 as the high-confidence dependency criterion. This fills the technical gap in mining low-support, high-confidence rules and improves the theoretical interpretability of dependency detection.

[0016] 2. This invention provides a parallel function dependency mining method based on Bayesian networks, which significantly improves search efficiency. It reduces the data size through hierarchical sampling, combines star Bayesian networks and Markov blankets to extract relevant attributes, vertically divides the original relation table into sub-tables, and transforms the global exponential search space into multiple local subspaces, greatly reducing computational complexity and achieving high computational efficiency.

[0017] 3. This invention provides a parallel mining method for function dependencies based on Bayesian networks, which has outstanding parallel acceleration effect. By designing the PFMiner parallel mining framework, parallel processing of multiple sub-tables can be achieved in a multi-core or distributed environment, with an overall speedup of tens to hundreds of times, while ensuring result consistency and minimization. It has good scalability and engineering practical value.

[0018] 4. This invention provides a parallel functional dependency mining method based on Bayesian networks, which is more robust. Hierarchical sampling ensures the statistical characteristics of the data, and Bayesian network modeling reduces the impact of noisy data on dependency detection, effectively improving the confidence and accuracy of dependency discovery in noisy environments. Attached Figure Description

[0019] Figure 1 This is a general flowchart of an embodiment of the present invention; Figure 2 This is a schematic diagram of functional dependency modeling from the perspective of Bayesian networks in an embodiment of the present invention. Detailed Implementation

[0020] The technical solution of the present invention will be specifically described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0021] A parallel functional dependency mining method based on Bayesian networks aims to efficiently discover low-support but high-confidence functional dependencies in large-scale, noisy data. By establishing a numerical equivalence relationship between functional dependencies and parent-child node relationships in Bayesian networks, this invention utilizes the conditional entropy minimization criterion to achieve statistical modeling of dependency rules, and significantly reduces the search space through hierarchical sampling and parallel partitioning, thereby improving the overall mining efficiency and accuracy.

[0022] Its overall process is as follows Figure 1 As shown, it includes the following steps: S1. Functional Dependency Modeling from the Perspective of Bayesian Networks: Establishing a mathematical correspondence between Bayesian network structure learning and functional dependency mining, using conditional entropy as a quantitative indicator of attribute dependency, determining that attribute X has a functional dependency on attribute Y when the conditional entropy H(Y|X) of attribute X with respect to attribute Y approaches zero; This invention first establishes a mathematical correspondence between Bayesian Network Structure Learning (BN) and Functional Dependency Discovery (FD) to achieve statistical modeling of functional dependencies, specifically as follows: Figure 2 As shown.

[0023] S1 includes the following steps: S11. Given a set of attributes If we take any two attributes X and Y, and there exists a directed edge in the Bayesian network from X to Y, it means that attribute X has a significant conditional constraint effect on the value of Y; if the edge does not exist, it means that the two are statistically independent. S12. Bayesian network-based learning methods all aim to train with the conditional entropy approaching 0, that is, when the conditional entropy... The Bayesian network structure learning method will only establish a directed edge between the two attributes X and Y when the value approaches 0. Bayesian network structure learning can typically be implemented using two types of methods: S121. Score-based learning: This method defines a score function for the network structure (such as BIC, AIC, or log-likelihood) and uses a search algorithm (such as hill climbing or genetic algorithm) to find the optimal structure. S122. Constraint-based Learning: Determines the causal relationship between nodes through a conditional independence test.

[0024] This invention demonstrates that both structure learning methods are trained with the conditional entropy approaching 0 as the objective, meaning that when the conditional entropy... The Bayesian network structure learning method only becomes effective when the value approaches 0. Figure 2 Establish a directed edge between these two attributes.

[0025] S13. Using conditional entropy as a quantitative indicator of attribute dependencies, automatic inference of functional derivation (FD) is achieved by judging the convergence of conditional entropy:

[0026] Therefore, when conditional entropy When X approaches zero, it means that X almost uniquely determines Y, and at this point, functional dependency exists. Established.

[0027] This theory establishes numerical equivalence between functional dependency mining (FD) and batch normalization (BN), allowing the FD discovery process to be transformed into a BN structure learning problem. Through this process, functional dependency mining, which originally relied on enumeration or pruning, is transformed into an optimization problem centered on minimizing conditional entropy, thus laying the mathematical foundation for subsequent search space reduction and parallel mining.

[0028] S2. Hierarchical Sampling and Search Space Reduction: Obtain the original relation table, obtain the sample dataset from the original relation table based on the hierarchical sampling strategy, construct a star Bayesian network based on the sample dataset and extract the relevant attribute set of the target attribute, and then vertically divide the original relation table into multiple sub-tables; Before performing Bayesian network structure learning, a representative subset of samples is first obtained through hierarchical sampling to ensure the reliability of the structure under large-scale data. Then, based on the sampled data, a set of relevant attributes is constructed, the key dependency set of each target attribute is identified, and finally the search space is constructed and sub-tables are partitioned to achieve an exponential reduction in the overall search space.

[0029] (a) Stratified Sampling

[0030] 1. Sampling purpose: Due to the original relation table The dataset may contain millions of tuples, and directly learning the Bayesian network structure on the entire dataset would impose significant computational and storage demands. This invention reduces modeling complexity while maintaining the statistical properties of the data through hierarchical sampling, and provides a theoretical guarantee of structural consistency.

[0031] 2. Sampling Method: A sample dataset is obtained from the original relation table based on a hierarchical sampling strategy, specifically including the following steps:

[0032] Select the attribute with the most value categories from the original relation table as the representative attribute; The original relational table is divided into multiple layers based on the value range of the representative attributes; Random sampling is performed in each layer at a fixed ratio, and the sampling results of each layer are combined to form a sample dataset. The empirical estimate of the sample distribution is consistent with the overall distribution.

[0033] In a specific embodiment, the sampling method is as follows: (1) Select the attribute with the most possible value categories as the representative attribute. For example, “Occupation”.

[0034] (2) According to The range of values ​​is divided into Each strata has several layers. .

[0035] (3) In each layer, according to a fixed proportion Random sampling is performed to obtain samples. ,satisfy: .

[0036] (4) Final sample set As input for Bayesian structure learning.

[0037] (5) Based on the finite population theory, this invention proves the empirical estimation of the sample distribution when the sampling ratio between layers is consistent. Compared with the overall distribution Maintain consistency:

[0038] Therefore, structural learning after sampling is statistically equivalent to full learning.

[0039] If the attribute Occupation has 6 possible values, then the original relation table is divided into 6 levels, for example, level 1. Includes a sample of all "Craft-repair" professions. Proportionally... After sampling from each layer, all samples are combined to form the final sample set. This serves as the input for subsequent Bayesian network structure learning.

[0040] (ii) Construction of Correlated Attributes Discovery

[0041] Based on sampling data This invention directly extracts the set of relevant attributes for each target attribute (RHS) from the learned Bayesian network (BN) topology, rather than through additional correlation coefficients or statistical analysis methods. This process is supported by Markov Blanket theory, which identifies the minimum set of attributes that determines the value of the target attribute.

[0042] The construction of a star-shaped Bayesian network specifically includes: for each attribute A, during the Bayesian network structure learning process, defining a set of constraints to prohibit the establishment of edge connections between attributes other than A, thereby forming a star-shaped Bayesian network centered on A.

[0043] In a specific implementation, for each attribute Define a constraint set (blacklist):

[0044] This constraint prohibits edge connections between attributes other than A, ensuring that only edges directly connected to A are retained during the learning process, thus resulting in a star-shaped Bayesian network centered on A. When conditional entropy At that time, and Establish connecting edges between them to form related sets: This set represents all attributes that have a strong dependency on A.

[0045] Extracting the relevant attribute set of the target attribute specifically includes: extracting the Markov blanket of the target attribute A from the star-shaped Bayesian network, and using the attributes in the Markov blanket as the relevant attribute set of the target attribute A.

[0046] In a specific embodiment, attribute set extraction and Markov blanket identification: for each target attribute From its star network Extracting Markov blankets And use it as the final set of relevant attributes: . The parent node represents the attribute that may determine the value of A, and the child nodes represent the attributes that are affected by A. This set is used to define the subsequent candidate space for functional dependencies.

[0047] (III) Search Space Construction

[0048] 1. Sub-table generation mechanism: based on the relevant collection of each attribute. This invention will transform the original relation table The search space is vertically divided into multiple sub-tables, each containing only one target attribute and its related attributes. In this way, each sub-table is an independent search subspace, and the number of columns in each sub-table is much smaller than that in the original table. This effectively avoids the exponential explosion of global attribute combinations and achieves an exponential reduction in the search space.

[0049] 2. SubLearner Algorithm: To automate the above process, this invention designs a SubLearner module. Its main process involves applying a set of constraints to each attribute in the relation table and learning a star Bayesian network structure. Then, it extracts a set of associated attributes and vertically partitions the large relation table according to the attributes in the associated attribute set, ensuring that each sub-table contains only column data corresponding to the attributes in the associated attribute set.

[0050] S3. Parallel Dependency Discovery and Result Merging: The parallel functional dependency mining algorithm PFMiner is adopted to execute the traditional sequential algorithm on multiple sub-tables simultaneously, and the mining results are merged and deduplicated to generate the minimum functional dependency set.

[0051] After completing hierarchical sampling and search space reduction, this invention enters the parallel dependency discovery stage. This stage employs a parallel functional dependency mining algorithm, PFMiner (Parallel FD Miner), which supports multi-core or distributed environments. It executes traditional sequential algorithms, such as TANE or DFD, simultaneously on multiple sub-tables to significantly improve overall mining efficiency and ensure rule consistency and minimization during result merging.

[0052] The goal of this step is to achieve parallel functional dependency mining across multiple sub-tables without compromising discovery quality. Since each sub-table contains only the target attribute and its set of strongly correlated attributes, it is small in size and has a clear structure. Therefore, a lower support threshold σ can be set, while the confidence threshold δ remains high to ensure stable and reliable mining results.

[0053] S3 includes the following steps: S31. Distribute all sub-tables to multiple processing threads; S32. Each thread independently runs a function dependency mining algorithm from TANE or DFD on its assigned sub-table; S33. After all threads have completed, the system automatically merges the output of each thread, removes duplicate and redundant rules, and generates the final minimal set of functional dependencies.

[0054] The S33 system automatically merges the output of each thread and removes duplicates and redundancies. The specific steps involved are as follows: S331. After all threads have completed their calculations, the system enters the result integration phase, where the result sets output by all threads are merged into a global set; S332. Perform deduplication on duplicate or mutually inclusive dependencies, retaining only the minimal set of functional dependencies; S333. Rules for removing duplicates and redundancies include: If there are two functional dependency rules FD1: X1→Y and FD2: X2→Y, and X1 is a subset of X2, then delete FD2; if the confidence and support of the two functional dependency rules are approximately the same, then keep only one of them.

[0055] In a specific implementation, this step is carried out using the parallel functional dependency mining algorithm PFMiner (Parallel FD Miner), as follows: 1. Input: Support threshold (Support Threshold): Allows for lower values ​​to capture rare but important dependencies.

[0056] Confidence threshold (Confidence Threshold): The lower bound of the confidence level of a control rule.

[0057] Sub-table set : Generated by the SubLearner in step S2.

[0058] 2. Thread allocation and algorithm initialization

[0059] The system depends on the number of available processors Initialize the number of threads for the sub-table collection.

[0060] Each thread corresponds to a sub-table and calls a sequential functional dependency discovery algorithm within it. (such as TANE or DFD).

[0061] Each algorithm instance is configured with the same threshold parameter. To maintain consistency in results.

[0062] 3. Parallel Function Dependency Mining

[0063] Each thread performs the following operations independently: Pair list Perform dependency detection and output all high-confidence dependencies that meet the conditions: ; Each result set It contains the minimum dependency cover set within the current sub-table, which includes all dependency rules that cannot be simplified further.

[0064] 4. Result merging and deduplication

[0065] Once all threads have completed their calculations, the system enters the result integration phase.

[0066] The result sets of all threads' outputs are merged into a global set: .

[0067] Perform deduplication on duplicate or mutually inclusive dependencies, retaining only the minimal set of functional dependencies. .

[0068] Deduplication rules include: If there exist two functional dependency rules FD1: X1→Y and FD2: X2→Y, and Then delete FD2; If the confidence and support of two dependency rules are approximately the same, then only one rule is retained.

[0069] 5. Output and Performance

[0070] The output is a set of minimum high-confidence functional dependencies that satisfy a user-defined threshold. .

[0071] Because each sub-table is independent, the algorithm can scale linearly in multi-core or distributed environments, with an overall speedup of tens to hundreds of times.

[0072] Finally, it should be noted that although the present invention has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Various equivalent changes or substitutions can be made without departing from the concept of the present invention. Therefore, any changes or modifications to the above embodiments within the essential spirit of the present invention will fall within the scope of the claims of the present invention.

Claims

1. A parallel function dependency mining method based on Bayesian networks, characterized in that, Includes the following steps: S1. Functional Dependency Modeling from the Perspective of Bayesian Networks: Establishing a mathematical correspondence between Bayesian network structure learning and functional dependency mining, using conditional entropy as a quantitative indicator of attribute dependency. When the conditional entropy H(Y|X) of attribute X with respect to attribute Y approaches zero, it is determined that the functional dependency of attribute X on attribute Y has a high confidence level. S2. Hierarchical Sampling and Search Space Reduction: Obtain the original relation table, obtain the sample dataset from the original relation table based on the hierarchical sampling strategy, construct a star Bayesian network based on the sample dataset and extract the relevant attribute set of the target attribute, and then vertically divide the original relation table into multiple sub-tables; S3. Parallel Dependency Discovery and Result Merging: The parallel functional dependency mining algorithm PFMiner is adopted to execute the traditional sequential algorithm on multiple sub-tables simultaneously, and the mining results are merged and deduplicated to generate the minimum functional dependency set.

2. The parallel function dependency mining method based on Bayesian networks according to claim 1, characterized in that, S1 includes the following steps: S11. Given a set of attributes, take any two attributes X and Y. If there is a directed edge in the Bayesian network from X to Y, it means that attribute X has a significant conditional constraint effect on the value of Y; if the edge does not exist, it means that the two are statistically independent. S12. Bayesian network-based learning methods all aim to train with the conditional entropy approaching 0, that is, when the conditional entropy... The Bayesian network structure learning method will only establish a directed edge between the two attributes X and Y when the value approaches 0. S13. Using conditional entropy as a quantitative indicator of attribute dependencies, automatic inference of functional derivation (FD) is achieved by judging the convergence of conditional entropy: Therefore, when conditional entropy When X approaches zero, it means that X almost uniquely determines Y, and at this point, functional dependency exists. Established.

3. The parallel function dependency mining method based on Bayesian networks according to claim 2, characterized in that, The Bayesian network structure learning method in S12 typically employs two types of implementation methods, as follows: S121. Scoring-based learning method: This method involves defining a scoring function for the network structure and using a search algorithm to find the optimal structure. S122. Constraint-based learning method: Determine the causal direction relationship between nodes through conditional independence test.

4. The parallel function dependency mining method based on Bayesian networks according to claim 1, characterized in that, The step S2, which involves obtaining the sample dataset from the original relation table based on a hierarchical sampling strategy, specifically includes the following steps: Select the attribute with the most value categories from the original relation table as the representative attribute; The original relation table is divided into multiple layers based on the value range of the representative attribute; Random sampling is performed in each layer at a fixed ratio, and the sampling results of each layer are combined to form the sample dataset. The empirical estimate of the sample distribution is consistent with the overall distribution.

5. The parallel function dependency mining method based on Bayesian networks according to claim 1, characterized in that, The construction of the star-shaped Bayesian network in S2 specifically includes: for each attribute A, during the Bayesian network structure learning process, defining a set of constraints to prohibit the establishment of edge connections between attributes other than A, thereby forming a star-shaped Bayesian network centered on A.

6. The parallel function dependency mining method based on Bayesian networks according to claim 1, characterized in that, The extraction of the relevant attribute set of the target attribute in S2 specifically includes: extracting the Markov blanket of the target attribute A from the star-shaped Bayesian network, and using the attributes in the Markov blanket as the relevant attribute set of the target attribute A.

7. The parallel function dependency mining method based on Bayesian networks according to claim 1, characterized in that, S3 includes the following steps: S31. Distribute all sub-tables to multiple processing threads; S32. Each thread independently runs a function dependency mining algorithm from TANE or DFD on its assigned sub-table; S33. After all threads have completed, the system automatically merges the output of each thread, removes duplicate and redundant rules, and generates the final minimal set of functional dependencies.

8. The parallel function dependency mining method based on Bayesian networks according to claim 7, characterized in that, The system's automatic merging of outputs from various threads and removal of duplicates and redundancies in S33 specifically includes the following steps: S331. After all threads have completed their calculations, the system enters the result integration phase, where the result sets output by all threads are merged into a global set; S332. Perform deduplication on duplicate or mutually inclusive dependencies, retaining only the minimal set of functional dependencies; S333. Rules for removing duplicates and redundancies include: If there are two functional dependency rules FD1: X1→Y and FD2: X2→Y, and X1 is a subset of X2, then delete FD2.

Images