Text data stream clustering algorithm based on affinity propagation

Abstract

The invention discloses a text data stream clustering algorithm based on affinity propagation. The text data stream clustering algorithm is characterized by including the following steps: 1, carrying out dimension reduction processing on a text data set to obtain a corresponding text vector set; 2, obtaining clustering centers of all moments, and completing the clustering algorithm. By means of the text data stream clustering algorithm, the accuracy and the robustness of the algorithm can be improved without assigning the number of clusters in advance, and therefore the requirements for solving practical problems are met.

Description

technical field [0001] The invention relates to a clustering algorithm of text data flow based on neighbor propagation. Background technique [0002] With the advent of the big data era, a large amount of unstructured data has been generated on the network. Faced with these unstructured data, which are generated in real time, have a huge amount of data, and have complex structures, people urgently need to extract valuable information and knowledge from them. Text data stream clustering technology is a common method for analyzing these unstructured data. It has achieved good application results in news filtering, topic detection and tracking (TDT), user feature recommendation, etc., and has quickly become a current research hotspot. Since text data has high-dimensional sparse features, how to improve the efficiency and accuracy of clustering algorithms is very important. In 2005, Shi Zhong proposed the OSKM algorithm, which is an extension of the k-means algorithm. It divid...

AI Technical Summary

Problems solved by technology

[0003] This algorithm also has the following disadvantages: the number of clusters needs to be determined in advance for each clustering, and the number of clusters cannot be changed as the category changes.

The algorithm can achieve better clustering results for spherical clusters, but it is difficult to cluster into clusters of arbitrary shapes.

There are also studies that propose an HPStream algorithm, which uses high-dimensional projection technology to select subspaces for clustering, and uses decay functions to represent evolution information, but the prior parameter—the average clustering dimension is difficult to determine

The above improvement studies have adapted to the problems of flow clustering to a certain extent, but the accuracy and robustness of the clustering results have not been well resolved, and further improvement is needed.

Images