Frequent item set mining method and device based on MapReduce and array

Abstract

The embodiment of the invention discloses a frequent item set mining method based on MapReduce and an array. The frequent item set mining method comprises the following steps: converting a data set into a two-dimensional array; decomposing the two-dimensional array into a plurality of two-dimensional sub-arrays; allocating the plurality of two-dimensional sub-arrays to at least two nodes which execute frequent item set mining tasks in parallel, wherein each node mines a sub-frequent item set corresponding to the two-dimensional sub-array and retains a non-frequent item set of each node; and counting and summarizing the sub-frequent item sets and combining the non-frequent item sets to obtain a frequent item set of the data set. According to the method, the database is only scanned once andconverted into the two-dimensional array, scanning of the database is reduced, and the I/O time is shortened; meanwhile, the array is creatively decomposed into more sub-arrays through a horizontal division method; a MapReduce programming model of a Hadoop platform is used, and a method of processing two-dimensional subarrays in parallel is adopted, that is, frequent item set mining is performedon the subarrays in parallel at a plurality of nodes, so that the method has relatively good acceleration ratio and expandability, and is suitable for mining frequent item sets for big data sets.

Description

technical field [0001] Embodiments of the present invention relate to the technical fields of data mining and big data, and in particular to a frequent itemset mining method, device, electronic device and storage medium based on MapReduce and arrays. Background technique [0002] In the field of data mining, R. Agrawa and R. Srikant proposed the classic Aprior algorithm, and many literatures have proposed many improved frequent itemset mining algorithms. Compared with the Aprior algorithm, these algorithms shorten the I / O time to a certain extent, and appropriately improve the efficiency of finding frequent itemsets, but they still cannot solve the bottleneck problem of the algorithm well. [0003] At present, with the rapid development of information, the massive data that needs to be analyzed is also increasing day by day, which causes the current serial frequent itemset mining algorithm to face two difficult problems: one is limited by the memory of a single machine, It ...

AI Technical Summary

Problems solved by technology

[0004] To this end, the embodiment of the present invention provides a frequent itemset mining method, device, electronic equipment and storage medium based on MapReduce and arrays, to solve the frequent itemsets caused by the serial frequent itemsets mining algorithm in the prior art Mining inefficiencies

Images