High-dimensional sparse text data clustering method based on Spark

Abstract

The invention requests to protect a high-dimensional sparse text data clustering method based on Spark. The method comprises the following steps of: 1) a RDD (Resilient Distributed Dataset) is used for reading a dataset; 2) a RDD interface is used for designing a distributed spare vector set; 3) a similarity between the distributed spare vector set and an integral dataset of a node where the distributed spare vector set is positioned is calculated, similar matrixes are obtained by abstraction according to numbers, the stored similar matrixes are subjected to symmetrization, and the normalized forms and the Laplace matrix forms of the similar matrixes are obtained; 4) SVD (Singular Value Decomposition) is used for decomposing the normalized Laplace matrix in the 3); 5) a new matrix constructed in the 4) is used into a K-means model as a sample to be trained; and 6) an established model is used for clustering a test set. By use of the method, the operation performance of a traditional clustering algorithm under a big dataset is improved.

Description

technical field [0001] The invention relates to the fields of text data clustering, machine learning and distributed computing, in particular to a Spark-based high-dimensional sparse text data clustering method. . Background technique [0002] With the advent of the big data era, the Internet has accumulated more and more network data. These accumulated data have reached the limit that ordinary computers can handle. In order to deal with increasingly difficult data processing problems, all walks of life have turned their attention to the Spark-based distributed processing platform and parallel sparse data set storage technology. [0003] Spark is a distributed programming framework for big data similar to Hadoop, but there are some useful differences between the two that make Spark superior for certain workloads, in other words, Spark enables in-memory distributed datasets, which can optimize iterative workloads in addition to being able to provide interactive queries. T...

AI Technical Summary

Problems solved by technology

For example, when calculating the similarity matrix M(n*n) between samples (n is very large) or storing a similar size similarity matrix, there may be inoperable problems

At the same time, the traditional spectral clustering algorithm also needs a lot of storage space and computing time when calculating the K eigenvectors of the Laplace matrix.

These outstanding problems make spectral clustering more and more unsatisfactory to meet the computing requirements proposed by the current situation of rapid increase in data volume.

Images