Distributed index establishment method and system based on text clustering

Abstract

The invention discloses a distributed index establishment method and system based on text clustering.The method includes the steps that unstructured texts are subjected to formatting and word segmentation pretreatment, and the pretreatment result is stored in original distributed nodes; the pretreatment result is subjected to filtering and feature extraction, and processed text lexical feature vectors are obtained; the text lexical feature vectors are clustered through a Canopy-Kmeans clustering algorithm, and K clusters of the text lexical feature vectors are obtained; each cluster of the K clusters is distributed on one or more distributed nodes; the K clusters distributed on the one or more nodes are subjected to full-text index establishment through an index engine, and K full-text indexes are obtained.By means of the embodiment, the method and system are used for establishing a distributed index mode for retrieval, the rapid index mode is provided for a user, and the use experience of the user is improved.

Description

technical field [0001] The invention relates to the technical field of retrieval index construction, in particular to a text clustering-based distributed index construction method and system. Background technique [0002] In traditional structured information management, index technology is usually used to retrieve information. However, in a distributed network environment, the growth rate of knowledge scale is very fast, and the size of index files increases sharply with the growth of scale. The index is stored in a centralized way, and the retrieval efficiency is also seriously affected by the huge index library; in view of this situation, an index method based on document division is proposed, but this index divides the collection in a random way, because the subsets of each division is an equivalent distribution, so all sub-indexes still need to be retrieved during retrieval, resulting in a large retrieval overhead. [0003] Text clustering is based on the clustering hy...

AI Technical Summary

Problems solved by technology

[0002] In traditional structured information management, index technology is usually used to retrieve information. However, in a distributed network environment, the growth rate of knowledge scale is very fast, and the size of index files increases sharply with the growth of scale. The index is stored in a centralized way, and the retrieval efficiency is also seriously affected by the huge index library; in view of this situation, an index method based on document division is proposed, but this index divides the collection in a random way, because each divided subset is an equivalent distribution, so all sub-indexes still need to be retrieved when retrieving, resulting in a large retrieval overhead

Images