Multi-layer frequent pattern discovery algorithm with high space extensibility and high time efficiency for mining mass data

Abstract

The invention discloses a multi-layer frequent pattern discovery algorithm with high space extensibility and high time efficiency for mining mass data, relating to the field of intelligent information processing, which has a wide application prospect in mass data mining, particularly in network information search and knowledge discovery. Aiming at the problem that time and space expense bottleneck exists during mass data mining due to the present multi-layer algorithm for simply extending a single-layer algorithm, the invention provides three new technologies: the first one is a hierarchical labeling technology capable of integrating hierarchical structure information in a plurality of data expression methods by the least additional expense to solve the space expense bottleneck; the second one is an extensive virtual projection method for avoiding the repeat generation of a pattern support set and having a higher space utilization rate; the third one is an inverted set enumeration tree for organizing multilayer patterns and a cutting technology thereof, and the inverted set enumeration tree greatly reduces the search space of a frequent pattern, thereby solving an operation time bottleneck. The time and efficiency of the algorithm disclosed by the invention are about 5 times and 1-3 orders of magnitude higher than those of two reference algorithms and the space expense is the least. Various applications such as mass Web mining, multimedia mining and text mining become possible due to the high performance of the algorithm disclosed by the invention.

Description

technical field [0001] The invention relates to the field of intelligent information processing. The present invention designs an algorithm capable of discovering frequent patterns spanning multiple concept levels with high spatial scalability and high time efficiency. It is used in massive data mining, especially network information search and knowledge discovery, including Web mining, text mining, multimedia In mining, it has broad application prospects. Background technique [0002] mining in the web [8] , text mining, multimedia mining [3] Frequent pattern mining is the most basic task in various data mining applications. When mining massive data, it is difficult to extract frequent patterns at the lower or original conceptual level due to the sparsity of the data, and the frequent patterns at the highest conceptual level are often just commonsense knowledge. Therefore, it is valuable to discover frequent patterns across multiple conceptual levels. However, the resu...

AI Technical Summary

Problems solved by technology

FPM-cross [4] Inherited FPGrowth [6] The disadvantage is that a large number of pattern support sub-libraries need to be generated during the mining process, and maintaining these pattern support sub-libraries requires a lot of computing time and storage space

In addition, FPM-cross [4] It is also necessary to insert all the ancestors of each original item at the conceptual level into the corresponding original schema support sub-library, and the space overhead will rapidly expand

Therefore, FPM-cross [4] Also facing the challenges of efficiency and scalability

Images