OPTICS point sorting clustering method based on Spark memory computing big data platform

Abstract

The invention provides an OPTICS clustering algorithm based on a Spark big data platform, and relates to a computer information obtaining and processing technology. Parallel data is structurally partitioned, the optimal data set partitioning is obtained, a corresponding RDD is generated, neighbor sample numbers and core distances are calculated in parallel, partitions are subjected to parallel execution of the OPTICS algorithm to obtain a cluster sequence of the partitions, and the cluster sequence is obtained persistently; clusters are given to the partitions according to the cluster sequence, and samples can obtain global cluster numbers by combining the partitions. By means of the Spark distributed parallel technology, the optimal partitioning structure is found, and the cluster sequence of the partitions is obtained through parallel calculation. According to the OPTICS cluster sequence, a user can observe the inherent clustering structure of a data set from different levels of structures, the method can process a large data set which cannot be processed by a serial algorithm, and the time for obtaining the clustering result is greatly shortened.

Description

technical field [0001] The invention relates to the technical fields of computer data mining and computer information processing. Background technique [0002] With the rapid development of the field of computer information, a large amount of data is collected from all aspects of life, and the scale of various information on the Internet is also increasing geometrically. Rapid analysis from massive data can extract information hidden in The information in the data is becoming more and more important. [0003] Clustering analysis is a main method of data analysis. Clustering is the process of classifying data objects, so that objects in the same cluster have a high degree of similarity, while objects in different clusters are highly different. Different from the classification process, the clustering does not rely on the pre-defined classes and class labels, and the classification criteria and the number of types in the clustering process are unknown. Cluster analysis metho...

AI Technical Summary

Problems solved by technology

[0007] The technical problem to be solved by the present invention is, in view of the disadvantages of the prior art, such as easy memory overflow, too long time and inability to run downtime when performing clustering when processing large batches of data, the present invention proposes a method based on Spark (based on memory computing platform) The OPTICS (Ordering Points To Identify The Clustering Structure) algorithm of the big data platform can handle large batches of data sets, and can get the cluster sorting in a very short time

Images