Real-time text clustering method based on Jaccard distance

Abstract

The invention relates to a real-time text clustering method based on a Jaccard distance, and the method specifically comprises the following steps: S1, text similarity calculation: selecting two textsfrom to-be-clustered data, extracting keywords from the two texts, calculating the intersection and union set of the keywords corresponding to different texts, and further obtaining the Jaccard distance; s2, setting a hierarchical clustering threshold value; s3, constructing a clustering model, sequentially reading newly loaded data, calculating an average distance between each piece of data andeach class, comparing the average distance with a threshold value, determining whether the classes are clustered or independently classified, and continuously iterating and updating; and S4, writing the clustering result in the S3 into the Hbase and ES database in the form of updating the clustering identifier, wherein the data with the same clustering identifier in the ES database is clustered into one class. According to the text clustering method, real-time analysis of massive text data, similar text clustering and effective deduplication can be realized, the user experience is improved, and meanwhile, the text classification result can be improved.

Description

technical field [0001] The invention relates to the technical fields of natural language processing and big data, in particular to a real-time text clustering method based on Jaccard distance. Background technique [0002] In today's information-explosive society, massive data and information appear every day, and each topic will be mentioned on different platforms or by many people at the same time, so human beings will encounter many repetitions or similar problems while reading information. Data, which is a big obstacle for us to efficiently obtain data information, will waste a lot of time. Therefore, the text clustering method is used to deduplicate massive network text data, and similar data are integrated to form a category, which can be read and processed by classification, thereby greatly improving work efficiency and saving time. [0003] At present, the text similarity distance calculated mainly based on the bag of words model, TF-IDF, and WORD2VEC is used as the...

AI Technical Summary

Problems solved by technology

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the present invention proposes a real-time text clustering method based on Jaccard distance, which solves the technical problems of low real-time clustering accuracy and slow processing speed of existing mass text data

Images