Professional field-oriented on-line theme detection method

Abstract

The invention discloses a professional field-oriented on-line theme detection method. The method comprises the following steps: obtaining a text vector matrix of a preprocessed text set, and extracting a dictionary from the text set; modeling the text vector matrix; calculating a mixed weight p (thetak|d) from a text d to a theme thetak and a frequency p (w|thetak) that a feature word appears in each theme thetak; obtaining the similarity between two texts di and dj, defining a theme model-based theme distance between the texts into a relative entropy distance of a text vector, and calculating a similarity matrix; compressing the text set, thus obtaining a new text sample sect; calculating a similarity matrix of the new text sample set, and selecting a deviation parameter p according to the similarity matrix; combining clustering results, thus generating a new clustering result; calculating distances between all texts in the original text set and compressed classified texts, and performing classification; outputting a text set theme and a final clustering result. Compared with the prior art, the professional field-oriented on-line theme detection method disclosed by the invention has the advantage that by the adoption of an optimal clustering algorithm, the accuracy and the efficiency of the clustering effect are improved.

Description

technical field [0001] The invention belongs to the technical fields of data mining, natural language processing, information extraction and information retrieval, and in particular relates to a topic detection and tracking technology. Background technique [0002] Currently, in the related technologies of topic detection, the commonly used clustering algorithms mainly include K-means clustering algorithm (K-means) and affine propagation clustering algorithm (AP algorithm). K-means clustering algorithm (K-means) is the most popular and typical distance-based partitioning clustering algorithm. The K-means algorithm uses distance as the evaluation index of similarity, and considers that a cluster is a set composed of a group of objects that are similar to each other, so the final goal is to obtain a compact and independent cluster. The K-means algorithm uses a randomly selected point as the initial center point, and then divides the points in the set into corresponding catego...

AI Technical Summary

Problems solved by technology

[0005] The disadvantage of the AP algorithm is that the AP algorithm needs to occupy two CPU times when calculating the data, and the algorithm complexity is high. Therefore, when the data set size is relatively large (N>3000), the AP clustering algorithm often needs to be calculated for a long time.

In recent years, with the substantial increase in the size of web pages, the AP clustering algorithm has been unable to meet the clustering requirements of large-scale data sets.

Images