Quick clustering method for massive text data

Abstract

The invention provides a quick clustering method for massive text data. The method comprises the following steps: preprocessing externally input command line parameters and text information read undera specified directory, calling a preset structure body through an internal interface to complete text data clustering under the specified directory, outputting an EXCEL file or a graphical interfaceclustering result under the specified directory, and evaluating a clustering effect. The method specifically comprises the following steps of text data reading, text information preprocessing, text data clustering processing and clustering result output, wherein the text information preprocessing comprises the steps of S1, performing word segmentation on a Chinese document, and performing TOKEN processing on an English document; S2, removing stop words; s3, calculating a simhash code of the document after the stop word is removed; s4, carrying out word embedding in a word2vector vector mode, and calculating a document vector with stop words removed; s5, carrying out word embedding in a bert vector mode to obtain a word vector. According to the method, the optimal clustering algorithm strategy of the clustering algorithm is realized through internal or external evaluation.

Description

technical field [0001] The invention relates to the field of text clustering, in particular to a fast clustering method for massive text data. Background technique [0002] Text clustering is based on the characteristics of high similarity between the same document and small similarity between different documents to aggregate similar documents. At present, most clustering methods focus on improving the accuracy of clustering. In terms of clustering efficiency, It takes a long time to process text data of thousands or tens of thousands of levels, and the accuracy of general clustering algorithms cannot meet the clustering requirements of such a huge amount of text data. The impact of clustering time is not the same. Therefore, in the clustering evaluation of a certain clustering algorithm, the clustering accuracy and clustering time can be used as observation points to effectively judge the clustering effect, so as to provide different clustering algorithms. Efficient cluste...

AI Technical Summary

Problems solved by technology

[0002] Text clustering is based on the characteristics of high similarity between the same document and small similarity between different documents to aggregate similar documents. At present, most clustering methods focus on improving the accuracy of clustering. In terms of clustering efficiency, It takes a long time to process text data of thousands or tens of thousands of levels, and the accuracy of general clustering algorithms cannot meet the clustering requirements of such a huge amount of text data. The impact of clustering time is not the same. Therefore, in the clustering evaluation of a certain clustering algorithm, the clustering accuracy and clustering time can be used as observation points to effectively judge the clustering effect, so as to provide different clustering algorithms. Efficient clustering algorithm selection according to class requirements; currently, there is a lack of clustering processing and clustering effect evaluation of text data with thousands or tens of thousands of levels

Images