MapReduce-based FP-Growth load balance parallel computing method

Abstract

The invention relates to a MapReduce-based FP-Growth load balance parallel computing method. The method comprises the steps that 1, a database transaction set D is divided into different continuous partitions, and a sub-transaction set are stored on multiple nodes; 2, parallel computing is conducted on support counts to obtain all the frequent one-item sets FList; 3, items of the frequent one-item sets are divided into M groups according to a load balancing method to obtain a new list GList; 4, the database transaction set D is also divided into M groups according to the new list GList, a local FP-Tree of each transaction set DB is created when the division of the database transaction set D is finished, and a corresponding GList[gidi] is mined according to each local FP-Tree to obtain the frequent patterns of all the items in the frequent one-item set; 5, the frequent patterns of all the items in the frequent one-item set obtained on each node are aggregately output. The MapReduce-based FP-Growth load balance parallel computing method has good load balancing capacity and execution efficiency.

Description

technical field [0001] The invention relates to a load balancing parallel computing method, in particular to a MapReduce-based FP-Grwoth load balancing parallel computing method, which belongs to the technical field of data mining. Background technique [0002] Association rule mining reflects the interdependence and correlation between a thing and other things, which is an important topic in data mining technology. Association rule mining needs to go through two steps, that is, the generation of frequent itemsets and the generation of association rules. The overall performance of association rule mining is mainly determined by the first stage. Classic association rule mining algorithms mainly include Apriori algorithm, FP-Growth algorithm and Eclat algorithm. The former two adopt horizontal data format for mining, while the latter adopt vertical data format for mining. Compared with the Apriori algorithm, the FP-Growth algorithm uses a divide-and-conquer strategy to mine t...

AI Technical Summary

Problems solved by technology

[0004] With the progress of society and the development of science and technology, data is growing explosively. The FP-Growth algorithm for mining association rules in the form of a single machine is far from meeting the storage and mining of massive data. Some existing FP-Growth algorithms The parallel algorithm solves the two problems of database division and subsequent parallel computing, but there are obvious differences and deficiencies in the algorithm in terms of parallel computing efficiency, memory consumption, communication consumption, and performance differences caused by differences in FP-Tree sparsity. Both have a lot to do with the lack of load balancing considerations when dividing database transaction sets

Images