Large-scale data analysis method based on DBSCAN algorithm

Abstract

The invention discloses a large-scale data analysis method based on a DBSCAN algorithm. The large-scale data analysis method comprises the following steps that 1, an adjacency list is calculated through annular communication; 2, locally and preliminarily determining a temporary cluster number of a core point; step 3, broadcasting synchronous core point cluster numbers in turn; and step 4, updating the boundary point cluster number through reverse annular communication. According to the method, the efficiency problem of the DBSCAN algorithm in large-scale data clustering analysis can be well solved, the problem of dependency of calculation steps in the recursive calculation process of the DBSCAN algorithm is ingeniously solved, the method is made to be suitable for a distributed execution model, and workloads of different nodes in a cluster are made to be balanced as much as possible. The communication traffic in distributed calculation is greatly reduced, calculation and communication are overlapped in a non-blocking communication mode, and the idle waiting time is shortened. And the memory of the calculation node is fully utilized, so that the clustering analysis of mass data on the cluster is realized.

Description

technical field [0001] The invention relates to a big data clustering algorithm and a computer distributed computing technology, in particular to a large-scale data analysis method based on the DBSCAN algorithm. Background technique [0002] With the continuous innovation of information technology, people's demand for extracting effective information from massive data is more and more urgent. However, how to process the massive data is still a serious challenge. Data mining technology to mine effective information from data has become more and more important with the advent of the era of big data. Cluster analysis in the field of data mining is widely used in practice because of its versatility. In conclusion, how to apply cluster analysis techniques to large-scale systems and massive data is one of the important challenges in big data processing. [0003] The DBSCAN algorithm is a density clustering method proposed by Ester Martin et al. in 1996. It takes the data object...

AI Technical Summary

Problems solved by technology

However, as the dimensionality of data increases, the distribution of data often becomes extremely complex, and the effect and performance of simple partition methods will become very unstable; complex partition methods also have the problem that the time consumed by the partition process is too large, and many scenarios Only one-time cluster analysis is needed, and the time is not worth the loss

Images