Document clustering method

Abstract

A document clustering method is used for conducting document mining on a document set of a potential Dirichlet distribution model. The document clustering method at least comprises the following steps: conducting training on the potential Dirichlet distribution algorithm in a first document set D1 to obtain parameters beta and phi, wherein the theme number K is preset in the potential Dirichlet distribution algorithm; according to the parameter phi, utilizing the information entropy theory for filtering the first document set D1 to obtain a second document set D2; according to the parameter beta, grouping the second document set D2 to generate a third document set D3 containing grouping information; operating the FG-Kmeans algorithm on the third document set D3 to obtain a finally-clustered clustering center set C and a mark matrix U. According to the document clustering method, documents are grouped according to the potential Dirichlet distribution algorithm, the FG-Kmeans algorithm is utilized for processing the grouped documents, the problem of high-dimensional and sparse data in document mining is well solved, and the concept of feature grouping is introduced into feature space, so that information contained in the feature space is more rich.

Description

technical field [0001] The invention relates to the field of data mining, in particular to a text clustering method. Background technique [0002] With the advent of the era of big data, people are faced with the severe challenge of data clustering on high-dimensional data. Excessively high dimension directly leads to the sparseness of data, which is especially obvious in text mining. Clustering algorithm is an effective method for clustering high-dimensional sparse data. As a clustering algorithm, FG-Kmeans algorithm (Chen, X., Ye, Y., Xu, X., Huang, J.Z.: A feature group weighting method for subspace clustering of high-dimensional data.Pattern Recognition45(1)(2012)) successfully introduced the concept of group into soft clustering. In the FG-Kmeans algorithm, features are divided into several according to similarity The algorithm weights the features and groups at the same time, and finds out the more important feature groups in each cluster and the more important featu...

AI Technical Summary

Problems solved by technology

So far, there is no algorithm that can automatically group the features on the data set, which limits the application of the algorithm to a large extent.

Images