Large-scale high-dimensional data approximate neighbor query system and method based on Spark

Abstract

The invention provides a Spark-based large-scale high-dimensional data approximate neighbor query system and method. The system and the method are mainly used for executing approximate neighbor query in a memory. Firstly, clustering partition is carried out according to the similarity of vectors, and each clustering partition corresponds to one partition of a Spark elastic distributed data set. And proportionally sampling the data of each partition, and labeling the partitions. And establishing a global index on the main node by using the sampling data, and establishing a partition index on the corresponding partition. And during query, finding a plurality of corresponding partitions needing to be queried through the global index, and summarizing and sorting results of the partitions to obtain a final result. According to the technical scheme, a highly extensible distributed approximate neighbor query scheme is provided based on the Spark system, and meanwhile the characteristics of low delay and high throughput are achieved.

Description

technical field [0001] The present invention relates to the technical field of computer data management, and more specifically, to a large-scale high-dimensional data fast retrieval method and system. Background technique [0002] Neighbor search is an important operation in many applications, such as image retrieval, recommender system and data mining all need neighbor search. With the rapid development of artificial intelligence related fields, machine learning algorithms have made major breakthroughs in computer vision, speech recognition, natural language processing and other application fields. A large amount of unstructured data (pictures, voice, text) can be converted into vector More efficient data representation has led to the generation of massive vector data. The vector nearest neighbor search algorithm needs to meet the requirements of scalability, high throughput, and low latency in order to efficiently process massive vector data. For example, in Taobao recom...

AI Technical Summary

Problems solved by technology

[0012] 1) It is specially designed for the characteristics of spatial data vectors. It does not consider the needs of non-spatial high-dimensional vectors, and cannot support a large number of existing non-spatial high-dimensional vector data. It is necessary to mine the relationship between vectors and design more effective query methods

[0013] 2) When querying, it often traverses all partitions, then collects the results of each partition, and generates the final result after centralized processing, which requires a large workload and takes a long time

[0014] 3) All are accurate queries, and the accuracy is higher than the actual demand, but the performance such as throughput and delay cannot meet the actual demand

Images