Distributed index method based on LSH (Locality Sensitive Hashing)

Abstract

The invention relates to a distributed index method based on LSH (Locality Sensitive Hashing). The distributed index method comprises the following steps: firstly utilizing a clustering algorithm to cluster mass data sets; then mapping clustering centers to different computational nodes; then mapping original mass picture or video characteristic data to the computational nodes corresponding to the type so that each node can process one type; finally utilizing the method based on p-stable distribution LSH to establish data index on different nodes. In order to reduce the merging time of search results on different computational nodes and improve the quality of the search results, the invention provides two methods to select m types recently for subsequent detailed search. The invention provides a guide for automatically mapping the mass data to the different computational nodes; moreover, according to the method, the detailed comparison times during the search period of the LSH further can be effectively reduced, so that the search of the mass data is more accurate and efficient.

Description

technical field [0001] The invention belongs to the technical field of computers, relates to a distributed index and query method in a high-dimensional data space, and relates to an approximate nearest neighbor search method, in particular to a distributed index method based on position-sensitive hashing. Background technique [0002] The traditional nearest neighbor search method plays an important role in information retrieval, data mining, database and other applications. Some classic tree-structure-based search methods usually perform well for low-dimensional data with dimensions less than 10. However, the features used to analyze data are far greater than 10, so these tree-structure-based search methods All degenerate to the efficiency of linear search, this phenomenon is known as the "dimension disaster". [0003] In the last 10 years, some scholars have proposed the use of approximate nearest neighbor search to replace the traditional nearest neighbor search. This me...

AI Technical Summary

Problems solved by technology

[0007] The problem to be solved by the present invention is: in the retrieval and query technology of high-dimensional data, the scalability of traditional position-sensitive hashing is not very good, and the search efficiency is low; In online search, because the calculation method of offline processing takes too long to respond

Images