Stochastic distributed data stream frequent item set mining system and method thereof

Abstract

The invention discloses a stochastic distributed data stream frequent item set mining system. In the system, a new distributed data stream mining mode for frequent item transactions is adopted, a stochastic frequent item mixer is introduced to improve the accuracy of mining, new transactions, when accumulated to the same number as basic windows, in a data steam are send to n frequent item set mining devices by a data item dividing method according to the different items in the transactions respectively, transaction numbers to which different data items belong are attached to the different data items, and in the frequent item set mining devices, frequent item sets are mined by running and operating different frequent item transactions and the mined frequent item sets are stored in a frequent item set storage. Finally, the data items are fed back to data item dividers through the stochastic frequent item mixer for deep mining. Compared with other methods, the method has the advantages of small memory storage space, high response speed and the like. Meanwhile, the coverage of mode mining can be ensured by increasing the number of the frequent item set mining devices or feedback times.

Description

technical field [0001] The invention belongs to data stream processing technology, in particular, it is a random distributed data stream frequent item set mining system and method thereof. Background technique [0002] Data flow is a data sequence composed of a series of high-speed, real-time, unlimited, and orderly arriving data. Data flow data widely exists in many fields of daily life, such as network traffic monitoring, meteorological monitoring, and sensor network data management. , Web log analysis, etc. In these fields, it is of great significance to discover frequent patterns of transaction data flow. For example, in network traffic monitoring, frequent patterns may indicate network congestion, and network congestion may be a symptom of network attacks. When a large number of IP data packets with the same address appear, a denial of service attack may have occurred; Important meteorological information such as the distribution of precipitation; in the sensor networ...

AI Technical Summary

Problems solved by technology

[0003] In data mining, frequent pattern mining has been extensively studied both in theory and in application and has achieved a lot of results. Many classic algorithms have emerged, but these algorithms are difficult to update incrementally and are not suitable for data stream mining. Because mining frequent patterns is a collection of continuous operations, the computation of any itemset cannot be completely completed before seeing all past and future data, making it difficult to mine and update frequent patterns in a data streaming environment; and Compared with the mining of static data sets, data streams have more information to track and more complex situations to deal with. Frequent item sets will change over time, and infrequent items may become frequent items later. The storage structure requires Dynamically adjust to reflect changes in frequent itemsets over time

[0004] At present, the traditional data flow frequent itemset mining methods are based on transactional data items, adopting a centralized mining mode, the time efficiency and space efficiency are relatively low, and cannot meet the user's real-time requirements for a large amount of data

Images