Class center compression transformation-based text clustering method in search engine

Abstract

The invention discloses a class center compression transformation-based text clustering method in a search engine. The method comprises the following steps of: by using an improved tf-idf formula, calculating word weight of each file in a text set, calculating an initial class center, mining a synonym word set and a concurrent high-frequency word set, calculating a word center and performing primary classification according to similarity of the initial class center with each file; compressing the center word according to information such as title word, article length, synonyms and concurrent associated words, thereby guaranteeing that the same word only occurs in some class centers with high similarity with the word; clustering the file by using a new cluster center again; calculating core similarity of each class; splitting the biggest class; combining smaller classes to produce a new class; iterating compression, clustering and split operation until the number of the classes converges; and guaranteeing that the similarity of the text in the same class with the cluster center reaches a certain threshold value. The clustering accuracy is obviously higher than those of the conventional methods such as KMeans and DBSCAN (Density-based Spatial Clustering of Applications with Noise).

Description

technical field [0001] The invention belongs to the technical field of text mining and machine learning research, and particularly relates to a text clustering method based on class center compression transformation in a search engine. By combining synonymous phrases, co-occurrence associated phrases, vocabulary centers, class centers, title content, Document length and other factors, repeated clustering and splitting iterative methods for text sets to improve clustering accuracy. The method is suitable for search engines and information retrieval systems. Background technique [0002] In the real world, text is the most important carrier of information, in fact, research shows that 80% of information is contained in text documents. Especially on the Internet, text data widely exists in various forms, such as news reports, e-books, research papers, digital libraries, web pages, emails, and so on. Text clustering technology can be applied to information filtering and person...

AI Technical Summary

Problems solved by technology

Therefore, the calculation of document similarity is

Images