Discriminant text clustering method and system based on minimum normalized information distance

Abstract

The invention discloses a discriminant text clustering method and system based on a minimum normalized information distance, and the method comprises the steps: carrying out the vectorization of a text data set which comprises a plurality of texts, wherein each text comprises a plurality of keywords; aiming at the vectorized text data set, initializing a model parameter set; calculating and updating the parameter set by a gradient descent method through the minimum normalized information distance; setting a termination condition and outputting a final parameter set; and designing a discriminant text clustering algorithm by utilizing the final parameter set to realize text clustering. The invention provides the discriminant text clustering method and system based on the minimum normalized information distance. Aiming at the model selection problem of the existing discriminant clustering algorithm, the invention provides the method for using normalized information measure as a target function, so the algorithm has automatic model selection capability, and the capability of obtaining a better clustering result under the condition that the initial model order of manual selection is unreasonable is improved.

Description

technical field [0001] The invention relates to the fields of natural language processing and text mining, in particular to a discriminative text clustering method and system based on minimum normalized information distance. Background technique [0002] Existing text clustering mostly uses the k-means algorithm, and the method of maximizing mutual information (or its variants) is mostly used in the discriminative clustering algorithm. These methods are likely to cause the model order (number of clusters, such as K) of K-means is always equal to the initial value, which makes this type of algorithm not have the ability of automatic model selection, so the model order of the final clustering result is largely determined by humans. However, it is difficult for people to give the most reasonable model order in text clustering, and a large or small model order can easily lead to poor clustering results. [0003] The existing discriminative clustering algorithm based on maximum ...

AI Technical Summary

Problems solved by technology

[0004] 1. If the selection of the initial model order is unreasonable, it will lead to unsatisfactory clustering results. For example, if the selection is too large, it is easy to produce an over-fitting model, that is, "the examples with high similarity that should belong to a cluster may be further subdivided model to different clusters", an extreme example is to divide a training example into a cluster, such clustering results are meaningless; and if the selection is too small, it is easy to produce an underfitting model, that is, "the similarity is not sufficiently low An example of a separate model"

[0005] 2. For the situation where the number of examples in each potential cluster is very different, the discriminative clustering algorithm based on maximum mutual information is prone to produce poor clustering models, that is, "a model that divides data with high similarity into different clusters"

Images