Spark-based multi-feature combined efficient Chinese text clustering method

A clustering method and multi-feature technology, applied in the field of machine learning, can solve the problems of not considering semantic similarity, increase of computational complexity and time complexity, loss of semantic information, etc., achieve good text clustering effect and reduce computing cost and time cost, the effect of reducing complexity

CN107590218AActive Publication Date: 2018-01-16NANJING UNIV OF SCI & TECH
2 Cites 15 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2018-01-16

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a Spark-based multi-feature combined efficient Chinese text clustering method. the method comprises the following steps of: uploading mass data sets into an HDFS file system byutilizing high fault tolerance and high data access throughput of the HDFS file system, carrying out data preprocessing and submitting the data sets to a Spark cluster; and after text set preprocessing is completed, respectively calculating a semantic similarity and a word frequency statistics-based cosine similarity of a dimensionality-reduced text, combining the two similarities to obtain a final text similarity, and carrying out text clustering by utilizing the obtained text similarity and combining a maximum distance method. According to the method, semantic information and word frequencystatistics information are combined to ensure that the text similarity calculation is more correct and the number of iterations is greatly decreased at the same time.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention belongs to the field of machine learning, in particular to a Spark-based multi-feature combined Chinese text efficient clustering method. Background technique

[0002] Clustering technology is a kind of machine learning. It mainly divides the original sample data set into several different data categories based on the differences between sample data and different parameters. Therefore, the ultimate goal of clustering is to make the difference between different samples divided into the same data cluster smaller, while the difference between samples divided into different data clusters is larger.

[0003] Text clustering technology is a kind of clustering, which is mainly based on the following principle: the differences between texts belonging to the same cluster are small, while the differences between texts belonging to different clusters are relatively large. Different from classification, clustering technology belongs to a class of un...

Examples

Embodiment

[0085] combine figure 1 , a Spark-based multi-feature combined Chinese text efficient clustering method, the specific implementation steps include:

[0086] Step 1: Build the Spark platform and HDFS file system on the physical server;

[0087] Step 2: Upload the original text data set to the HDFS file system, use the ICTCLAS Chinese word segmentation system and the Hadoop parallel computing platform to perform parallel word segmentation processing on the original text data set, and re-upload it to the HDFS file system;

[0088] Step 3: The Spark platform reads the word-divided data set from the HDFS file system, converts it into an elastic distributed data set RDD, and starts a certain number of concurrent data sets according to the number of partitions in the RDD set in the user program. The thread reads the data and stores it in system memory;

[0089] Step 4: According to the interdependence between the partitions in the RDD, the Spark job scheduling system splits the wri...