MPP (massively parallel processing)-based parallel data mining framework and MPP-based parallel data mining method

Abstract

The invention relates to an MPP (massively parallel processing)-based parallel data mining framework and an MPP-based parallel data mining method. The MPP-based parallel data mining framework is mainly and technically characterized in that the mining framework comprises a mining engine node and a plurality of distributed mining agent nodes. The method comprises the steps of assigning the current data mining task to the mining agent node with less data mining task load by the mining engine node, and taking the mining agent node with less data mining task load as a Master mining agent node of the data mining task; assigning mining tasks to the corresponding mining agent nodes by the Master mining agent node which adopts the data-distributed load balancing and nearby mining strategy; enabling all the mining agent nodes to respectively execute a Slaver operator according to an allocated subtask, wherein each Slaver operator is only used for processing an allocated data block. According to the framework and the method, an MPP method is adopted, and the characteristics of data mining are combined, so that mass data can be effectively processed at high speed, the problems that the traditional data mining software is small in data processing capacity and slow in running speed can be solved, and the mass data processing efficiency and the data bearing capacity of a data mining algorithm can be greatly improved.

Description

technical field [0001] The invention belongs to the technical field of data mining, in particular to an MPP-based parallel data mining framework and a method thereof. Background technique [0002] With the rapid development of computer technology, especially the continuous application of Internet technology, people's ability to use network information technology to generate and collect data has been greatly improved, and the data has shown a rapid growth trend. How to obtain the required information from massive data has become an urgent research problem. Faced with such a challenge, data mining (Data Mining) technology emerged as the times require, using data mining technology to obtain hidden useful information from these massive data. However, due to the explosive growth of data, how to use data mining technology to quickly and effectively obtain hidden useful information from massive data is becoming more and more important. [0003] Distributed storage system is to di...

AI Technical Summary

Problems solved by technology

When the amount of data is particularly large, this mode will be extremely slow, and even show incapacity, that is, data mining tasks cannot be performed

Images