Parallel clustering method based on memory calculation

Abstract

The invention provides a parallel clustering method based on memory calculation, and aims at improving the efficiency in processing mass data of a clustering algorithm DBSCAN. The method comprises the steps that S1) data is divided on the basis of simple random sampling, <ID,Raw_data> serves as input of the stage, simple random sampling is carried out on original data to divide the data, and divided results are stored in different RDD; S2) a memory calculation model is used to execute the DBSCAN algorithm in different calculating nodes in parallel, and the original data in the different RDD is clustered to generate local class clusters; and S3) all the local class clusters are merged based on the center of gravity, a memory calculating model is used to merge the local class clusters, and a global clustering result is generated. According to the invention, the original data is divided via the simple data division manner on the basis of the memory calculating module, and the processing efficiency of the algorithm is improved greatly; the local class clusters are merged based on the gravity center distance to construct the global class cluster, and user requirement for processing large-scale data is met.

Description

technical field [0001] The invention relates to the field of parallelization of data mining algorithms, in particular to a parallel clustering method based on memory calculation. Background technique [0002] Today, with the continuous innovation of information technology, data is growing at an explosive rate. How to effectively process large-scale data has become a serious challenge. [0003] In order to dig out regular information from massive data and find out the differences and connections between data, data mining, as a new discipline, appears in people's sight and plays an important role in various industries. [0004] Cluster analysis occupies a pivotal position in data mining and has received widespread attention. Clustering is usually based on a certain similarity measurement method, so that a group of data with high similarity is clustered together. [0005] The DBSCAN algorithm is a density clustering method based on high-density connected regions proposed by ...

AI Technical Summary

Problems solved by technology

If the data set is not loaded into memory, frequent I/O operations will cause the algorithm to be inefficient

Therefore, the traditional DBSCAN algorithm cannot be applied to cluster analysis of large-scale data sets

[0007] When the existing parallel DBSCAN algorithm performs data partitioning, it usually divides the original database into several disjoint partitions, and ensures the load b

Images