Cluster implementation method and system

Abstract

The invention discloses a cluster implementation method and system, wherein the method comprises the following steps: carrying out sharding on candidate samples in a candidate queue by a master control node; and respectively determining whether each sample in allocated samples subject to sharding is a core sample parallelly according to a preset epsilon neighborhood and the minimum density by at least two computing nodes, thus due to the parallel processing of the computing nodes, the marking speed of a cluster to which each sample in a sample database belongs is quickened. The invention also discloses another cluster implementation method and system, and the cluster implementation method comprises the following steps: carrying out sharding on samples which are not marked currently in a sample database by a master control node; allocating and issuing the samples subject to sharding to at least two computing nodes; carrying out parallel processing on candidate samples in a candidate queue by the computing nodes; and combining the obtained processing results of the computing nodes by merge nodes. Because each computing node only processes part of samples, the problem that mass data can not be processed by one computer is solved, and because the mass data can be subject to parallel processing by a plurality of the computing nodes and a plurality of the merge nodes, the processing efficiency is greatly improved.

Description

technical field [0001] The invention relates to the field of data mining, in particular to a method for realizing clustering of massive sample data and a corresponding system. Background technique [0002] In the current field of data mining, existing clustering algorithms can be divided into several categories, including partition-based methods, hierarchical-based methods, density-based methods, grid-based methods, and model-based methods. [0003] When performing data mining, it is necessary to calculate and analyze all the data one by one, and the time complexity of the algorithm is high. Massive data is a challenge to various clustering algorithms. Most of the existing clustering algorithms are still in the laboratory stage. For massive data, some algorithms either cannot be processed effectively, or the processing efficiency is very low. [0004] The DBSCAN algorithm is a clustering algorithm based on spatial density. The algorithm divides regions with sufficiently h...

AI Technical Summary

Problems solved by technology

[0009] The embodiment of the present invention provides a clustering implementation method and a clustering implementation system, and solves the problem of inability to implement clustering processing and low processing efficiency for massive data in the prior art by using multiple nodes for parallel processing

Images