Hadoop-based big data association rule mining method

Abstract

The invention discloses a Hadoop-based big data association rule mining method, which comprises the following steps of: firstly, inputting a big data set to be mined, and partitioning the big data set; then, using a two-stage MapReduce process to complete a mining task of association rules in the big data set; and finally, evaluating the frequent item set by using the kulczynski measurement and the imbalance ratio, and eliminating the frequent item set which does not meet the threshold requirements of the kulczynski measurement and the imbalance ratio, so as to ensure that the mined frequent pattern has positive correlation. According to the method disclosed by the invention, great memory and I / O consumption in a one-stage MapReduce process and low efficiency in a multi-stage MapReduce process are avoided. According to the method, the number of candidate item sets is reduced. The support degree of the candidate item sets is quickly obtained by utilizing the characteristics of the crosslinked list. A database does not need to be scanned for multiple times. The positive correlation of the mined item sets can be ensured. The method can be applied to actual decisions.

Description

technical field [0001] The invention belongs to the technical field of large-scale data mining, and in particular relates to a method for mining association rules of big data based on Hadoop. Background technique [0002] Traditional association rule mining algorithms can be mainly divided into three categories: the first category is the "generation-test" method, which generates candidate frequent itemsets through iteration and counts them separately, and counts the frequent itemsets; the second category is "pattern growth" method, it does not need to generate candidate itemsets, but compresses all frequent items into a special data structure (usually a tree structure), and directly generates frequent itemsets by traversing the data structure. The third category is the "vertical format" method, which converts the data set in the horizontal format into a vertical format, and obtains frequent itemsets through the intersection operation. [0003] With the rapid development of ...

AI Technical Summary

Problems solved by technology

[0003] With the rapid development of information technology, the amount of data that needs to be stored and analyzed is increasing explosively. Human beings have entered the era of big data. The traditional association rule mining algorithm can no longer meet the requirements of big data mining. The main difficulty is: a single computer cannot Store all the data that needs to be mined and the intermediate results generated during the mining process; the memory required for the mining process far exceeds the storage capacity of a single machine, and the calculation time is too long to bear, etc.

Images