Big-data-oriented distributed density clustering method

Abstract

A big-data-oriented distributed density clustering method comprises the following steps that firstly, environment virtualization is performed, and a Hadoop platform is established; secondly, data are pre-processed and loaded, wherein an original data table is extracted from a database, a needed field is intercepted through a sqoop-query command, and the pre-processed data are directly extracted to an Hdfs; thirdly, a distance matrix is calculated; fourthly, a cut-off distance and dot density are calculated; fifthly, the minimum distance between a dot and a higher-density dot is calculated; sixthly, the critical distance of a critical density point and a critical density point are determined; seventhly, dot clustering is performed, so a final clustering result is obtained; eighthly, off-group points are removed. The big-data-oriented distributed density clustering method is fast and effective when a big data set is processed, and has the effect that input parameters have good robustness on the clustering result.

Description

technical field [0001] The invention relates to the field of big data processing, and relates to a distributed density clustering method. Background technique [0002] Density-based clustering methods view clusters as regions of high-density objects in data space separated by regions of low density. It has the characteristics of discovering clusters of arbitrary shapes, identifying noise points in data sets, insensitive to the order of input objects, and good scalability, so it has important applications in cluster analysis. However, most density-based clustering algorithms cannot find clusters in non-uniform density data sets and have shortcomings such as sensitivity to input parameters and complex iteration of large-scale data. The application of density-based algorithms is limited to a certain extent. [0003] DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a representative density-based clustering algorithm. The result of DBSCAN clustering is a...

AI Technical Summary

Problems solved by technology

[0007] In order to solve the problem of low algorithmic efficiency of existing density clustering methods when processing large data sets, the present invention proposes a distributed density clustering method for large data, which uses distributed methods to calculate point density and point-to-minimum density Clustering the points with equal thresholds is fast and effective when processing large data sets, and has a good effect of the robustness of the input parameters on the clustering results

Images