Short text clustering and hotspot theme extraction method based on TF-IDF characteristics

Abstract

The invention discloses a short text clustering and hotspot theme extraction method based on TF-IDF characteristics. The method includes the following steps of firstly, conducting Chinese word segmentation on short text samples, and screening out high-frequency vocabularies; secondly, automatically conducting TF-IDF characteristic extraction and generation on each short text sample on the basis of the screened-out high-frequency vocabularies, and establishing a whole sample characteristic vector spatial model; thirdly, reducing spatial dimensions of the samples through singular value decomposition (SVD); fourthly, clustering the short text samples through the combination of the cosine law and the k-means method, and finding potential hotspot themes in each cluster through a visual analysis means. By means of the method, the characteristic selection problem, the sample control dimension reduction problem and the clustering problem of short texts can be well solved; meanwhile, visual analysis on the clustering result can be achieved by means of the visual technology; finally, extraction and analysis are conducted on hotspot themes.

Description

technical field [0001] The invention relates to digital text mining technology, in particular to text clustering and a method for extracting corresponding hot topics. Background technique [0002] Text clustering has been one of the hot issues that researchers have devoted themselves to researching, exploring and solving for many years. Today, there are still many problems that need to be solved urgently. For example, when clustering, the samples are not balanced and the dimension of sample features is too high. , the complexity of the clustering algorithm is too large and so on have brought great challenges. At the same time, with the rapid development of computers, massive amounts of text data are generated every day. With the surge of data, we have entered the era of big data, and more and more complex and difficult problems come with it. Contents of the invention [0003] In order to overcome the problems existing in the above-mentioned prior art, the present inventio...

AI Technical Summary

Problems solved by technology

[0002] Text clustering has been one of the hot issues that researchers have devoted themselves to researching, exploring and solving for many years. Today, there are still many problems that need to be solved urgently. For example, when clustering, the samples are not balanced and the dimension of sample features is too high. , the complexity of the clustering algorithm is too large, etc. have brought great challenges

At the same time, with the rapid development of computers, massive amounts of text data are generated every day. With the surge of data, we have entered the era of big data, and more and more complex and difficult problems to solve

Images