A frequent item set mining method based on a Spark platform

Abstract

The invention relates to a frequent item set mining method based on a Spark platform, and belongs to the technical field of data mining. Based on a Spark big data processing framework, the invention provides a novel BitMapFIM-Miner algorithm. A data set is processed in parallel, the data set does not need to be scanned many times, based on the thought of divide-and-conquer, transactions with the transaction lengths larger than a certain threshold value are segmented, then a bit operation method is used for calculating and generating frequent item sets, and finally the frequent item sets obtained by all parts are summarized and combined. By using a bit operation method, the mining speed of the frequent item set is accelerated, and the execution efficiency of the algorithm is greatly improved; Through theoretical analysis and experimental verification, it is found that an overlong transaction is segmented, a frequent item set can be efficiently obtained while it is guaranteed that the result is accurate, and a new idea is provided for a frequent item set mining method.

Description

technical field [0001] The invention relates to a frequent item set mining method based on a Spark platform, belonging to the technical field of data mining. Background technique [0002] How to improve the mining performance of frequent itemsets in massive data sets is a current research hotspot. The traditional Apriori algorithm, FP-Growth algorithm and its improved algorithm all show certain advantages when dealing with small-scale data. With the advent of the big data era, the stand-alone algorithm can no longer meet the time and space requirements of data processing. Distributed algorithms based on Hadoop and Spark cloud computing platforms continue to appear, but they cannot completely remove the Apriori algorithm and FP-Growth algorithm. However, the Apriori algorithm has the problem of scanning the database multiple times and generating too many candidate item sets; when the amount of data is too large, the FP-Growth algorithm will also spend a lot of time recursive...

AI Technical Summary

Problems solved by technology

With the advent of the big data era, the stand-alone algorithm can no longer meet the time and space requirements of data processing. Distributed algorithms based on Hadoop and Spark cloud computing platforms continue to appear, but they cannot completely remove the Apriori algorithm and FP-Growth algorithm. However, the Apriori algorithm has the problem of scanning the database multiple times and generating too many candidate item sets; when the amount of data is too large, the FP-Growth algorithm will also spend a lot of time recursively extracting the conditional pattern tree

For this reason, many algorithms ported to the Hadoop platform continue to emerge. Compared with Spark, Hadoop has an I / O bottleneck problem, and frequent reading and reading of data sets will consume a lot of unnecessary time.

In summary, whether it is an algorithm based on Apriori, an algorithm based on FP-Growth, or an algorithm transplanted to the Hadoop platform, in the face of massive data, there are problems such as frequent itemsets mining time is too long and memory consumption is too large, and cannot Meet the performance requirements for fast and efficient mining

Images