Probability frequent item set excavating method based on MapReduce

Abstract

The invention discloses a probability frequent item set excavating method based on MapReduce. The excavating method includes the following steps that (1), an uncertain data set T1 is read in; (2), all transactions in the uncertain data set T1 are sequentially processed at the Map end, and each item in one transactions and the probability value of the item are mapped into a key value pair (key, value); (3), output of the Map end is received at the Reduce end, and a probability frequent one-item set is generated by using a normal approximation method; (4), a list F-list is generated for the probability frequent one-item set output in the step (3); (5), the uncertain data set T1 stored in a distributed file system HDFS is read in and processed according to the list F-list to generate an uncertain data set T2; (6), a UApriori method based on the MapReduce is operated to continuously process the uncertain data set T2 obtained in the step (5) to generate a candidate item set, and then a probability frequent item set is generated from the candidate item set through the normal approximation method until all probability frequent item sets are generated.

Description

technical field [0001] The invention relates to a computer data mining method, in particular to an approximate mining method for large-scale uncertain data probability frequent itemsets based on MapReduce. Background technique [0002] In recent years, due to some new applications, such as: detection of sensor networks, search of moving objects, network analysis of protein-protein interactions, data integration and data cleaning, etc., the mining of uncertain data has become the focus of data mining. A new hot research topic. Uncertain data mining mainly includes clustering, classification, mining of association rules, outlier detection, etc. Among them, mining of frequent itemsets is a basis in the field of data mining. Therefore, the mining of probabilistic frequent itemsets in uncertain data has become a research hotspot. For example, the currently popular wireless sensor network collects a large amount of data. However, the collected data is usually imprecise due to t...

AI Technical Summary

Problems solved by technology

However, the problem it focuses on is to determine the frequent itemsets in the data. The frequent itemsets to be solved are also based on the support of the itemsets in all transactions, rather than the frequent probability of the itemsets. Determine the data

Another patent "A method and system for mining association rules" (101799810B) is also aimed at mining frequent itemsets in certain data, but on the one hand, this method cannot handle large data because there is no parallelization based on MapReduce; Unable to deal with uncertain data, the currently known patents on frequent itemset mining are not suitable for the mining of probabilistic frequent itemsets in complex uncertain data

Images