Incremental clustering algorithm based on community detection

Abstract

The invention discloses an incremental clustering algorithm based on community detection. The algorithm is characterized in that the algorithm comprises the steps: employing a Commonity community concept and an Online-Offline two-stage framework, and introducing an IMC concept; obtaining a similarity graph of the target corpus through a similarity calculation method according to the document representation word vector, the representation keyword of the document and the named entity identification prediction word of the document; and finally, processing the similarity graph through a Louvain algorithm to obtain an initialized community result, and employing an incremental clustering algorithm on the basis of the initialized community result to obtain a final clustering result. Compared withthe prior art, the method has the advantages that the calculation time overhead is reduced under the same hardware condition, and the clustering result is quickly generated, so that upstream and downstream services of an application scene can be better served, timely response is achieved, the function of distinguishing hot events from continuously reported events is achieved, and effective clustering and event-level filtering are conducted on news events.

Description

technical field [0001] The invention relates to the technical field of text clustering, in particular to an incremental clustering algorithm based on community detection. Background technique [0002] News is an important source of information, and a news report often contains some specific information, such as a report on a specific company or person. Therefore, many technology companies or researchers are committed to mining valuable information from relevant news reports, so as to serve commercial information analysis or data mining. Clustering is an effective means of gathering related information into topic clusters. With the explosive growth of information, traditional clustering methods will encounter serious performance bottlenecks in the face of large-scale data, and as new data The arrival of will re-cluster the historical data, causing unnecessary performance overhead. Compared with traditional clustering methods, incremental clustering is more suitable for this...

AI Technical Summary

Problems solved by technology

[0004] 1. The time complexity of the incremental clustering algorithm is still relatively large

[0005] DenStream adopts the Online-Offline two-stage clustering framework, which has high computational complexity in the Merging and Pruning stages, which brings huge time overhead; C-DenStream introduces an event-level Must-Link on the basis of DenStream The Cannot-Link restriction improves the clustering results, but there is still the problem of high computational complexity of DenStream; PreDeConStream improves the performance in the Offline stage, but there is still a huge time complexity overhead when searching for the nearest neighbor class

[0006] 2. Lack of ability to distinguish hot events from ongoing reports

[0007] DenStream directly deletes the data in the Outlier-Micro Cluster during the Pruning stage, which means that low-frequency hot events and low-frequency continuous reports are deleted together, causing the risk of information loss; C-DenStream uses a semi-supervised method to classify event-level news However, it is still unable to distinguish hot news and continuous news in the same event category, and it is still unable to distinguish between these two kinds of events; PreDeConStream does not handle this situation, so it lacks the corresponding event distinction ability

[0008] To sum up, the existing incremental clustering algorithms still have high computational time complexity and lack the ability to distinguish hot events from continuous reporting events, and the corresponding incremental text clustering algorithms have not been reported yet.

Images