Large-scale data clustering method and device and computer readable storage medium

Abstract

The invention relates to an artificial intelligence technology. The invention discloses a large-scale data clustering method. The method comprises the following steps: receiving a data sample set input by a user, calculating an average contour coefficient of a cluster center number K value according to the data sample set, selecting the K value with the maximum average contour coefficient, randomly determining K cluster centers, and storing the K cluster centers and the data sample set into a database according to a row priority storage form; calculating loss values of the K cluster centers and the data sample set according to a minimum square error algorithm, and judging a size relationship between the loss values and a preset threshold value; and when the loss value is smaller than the threshold value, outputting the K cluster centers to complete a clustering result. The invention further provides a large-scale data clustering device and a computer readable storage medium. Accordingto the invention, an accurate large-scale data clustering function can be realized.

Description

technical field [0001] The present invention relates to the technical field of artificial intelligence, in particular to a method, device and computer-readable storage medium for intelligently performing large-scale data clustering based on big data input. Background technique [0002] Clustering as a typical data classification method, its core is to find similar categories from large-scale data sets, and divide samples into multiple non-overlapping subsets. The K-means clustering algorithm is one of the most widely divided clustering methods, which uses the quality centers of various samples to represent the class for iteration, and clusters by dynamically adjusting various centers. However, the K-means algorithm is very dependent on the initial k centers. Improper selection of the initial centers may easily lead to local optimal solutions, increase the number of iterations, and reduce execution efficiency. In addition, during the K-means clustering process, data points ne...

AI Technical Summary

Problems solved by technology

However, the K-means algorithm is very dependent on the initial k centers. Improper selection of the initial centers may easily lead to local optimal solutions, increase the number of iterations, and reduce execution efficiency. In addition, during the K-means clustering process, data points need to be calculated The Euclidean distance with the class center point, and to calculate the Euclidean distance, it is necessary to calculate the dot product of the data point and the class center

In the case of massive data participating in clustering, there are a lot of dot products that need to be calculated, which takes a long time and is inefficient

Therefore, traditional clustering algorithms cannot provide a good solution when dealing with large-scale data, no matter from the perspective of system resources or real-time efficiency.

Images