Peer-to-peer structure based distributed high-dimensional indexing parallel query framework

Abstract

The invention provides a peer-to-peer structure based distributed high-dimensional indexing parallel query framework, which comprises an index creation module, a monitor module, a peer-to-peer site cluster and a soft load balance module. The index creation module is used for creating an index and storing index block files. The monitor module is used for detecting information of available memory of operation sites in the peer-to-peer site cluster and corresponding index block information of each operation site so as to coordinate the operation sites according to detection results. The soft load balance module is used for controlling load balance for the each operation site, and control is regularly synchronized through the monitor module so that the soft load balance module can timely regulate and update a lift of the operation sites currently available. The peer-to-peer structure based distributed high-dimensional indexing parallel query framework has the advantages of capability of reliably searching massive candidates in real time under support of dynamic increase and decrease, and is fast in query speed.

Description

technical field [0001] The invention relates to the technical field of computer information retrieval, in particular to a distributed high-dimensional index parallel query framework based on a peer-to-peer structure. Background technique [0002] The rapid development of content-based image retrieval, video retrieval and other technologies has made high-dimensional indexing technology more and more widely developed and applied. The application of high-dimensional index technology is mainly to speed up the retrieval speed and improve the user's query experience. In most of the current existing applications, the query of high-dimensional index is actually to retrieve the top k nearest neighbors (k Nearest Neighbors, kNN for short) of the query object at the semantic level. [0003] Generally speaking, content-based retrieval technology first needs to convert a large number of candidate objects (images, videos, etc.) into high-dimensional feature vectors, and preserve the char...

AI Technical Summary

Problems solved by technology

The second category is to improve the index creation and query speed by changing the index structure. However, after decades of research, no very efficient index structure that supports hundreds of dimensions or even higher dimensions has been found.

The third category is to improve the architecture of the index system by introducing a parallel index system to increase the speed of creation and query. However, in the current research on real-time reliable parallel systems, due to the existence of technical barriers, related technologies are difficult to be widely used. application

In addition, the traditional master-slave index query architecture lacks self-organization and is prone to performance bottlenecks

Images