Hybrid granularity distributional memory grid index-based KNN query method

Abstract

The invention discloses a hybrid granularity distributional memory grid index-based KNN query method, and specifically, the method is realized by the following steps: performing a data pre-processing step: based on the grid and density, performing space division of overall data to obtain overview estimation of overall data distribution; performing a data query step: establishing a hybrid granularity distributional memory grid index structure, that is, establishing a non-bisecting coarse-grained grid index and a bisecting fine-grained grid grid index; and and on the basis, by designing a distributional KNN query algorithm, realizing a fast KNN query for mass data. Compared to the prior art, the hybrid granularity distributional memory grid index-based KNN query method, reduces the data skew of the cluster, improves the data index efficiency and supports the distributional algorithm by designing and establishing the hybrid granularity distributional memory grid index, and disperses the bottleneck in single processing performance of the centralized KNN query algorithm, and the real-time degradation problem led by intermediate result rewriting disc of the KNN query algorithem based on the MapReduce construction by using the KNN query algorithm based on the index structure.

Description

technical field [0001] The invention relates to the field of information retrieval, in particular to a highly practical KNN query method based on a mixed granularity distributed memory grid index. Background technique [0002] With the rapid development of technologies such as the Internet, the Internet of Things, and big data, KNN query, as a basic operation, is widely used in various location-based applications. However, as the amount of data continues to grow, the traditional centralized KNN query and the KNN query method based on the MapReduce architecture cannot effectively process massive data quickly. How to expand the traditional centralized KNN query algorithm in the big data environment, combining the characteristics of memory clusters, indexing technology and distributed memory computing technology is the fundamental way to solve this problem. [0003] Indexing technology is a key component of the KNN query algorithm. Its basic idea is to use methods such as divi...

AI Technical Summary

Problems solved by technology

[0004] The efficiency of the KNN query method in the big data environment is relatively low. The lack of an effective index structure and the KNN query algorithm supported by this structure is the key to this problem, which is mainly reflected in: ① The research on the index structure in a centralized environment has been relatively mature, based on These index structures also propose many relatively effective KNN algorithms, but with the explosive growth of data volume, the single-machine processing performance in the centralized environment has become an irreconcilable bottleneck; ②In the distributed environment, the index structure based on the MapReduce architecture and related There are also some studies on the KNN query algorithm, but because the MapReduce architecture is a batch processing model, the intermediate results need to be written back to the disk, which increases I / O, which leads to poor real-time performance of the query, and the existing algorithm does not consider the data distribution. , it is easy to cause the data skew problem of the cluster

[0005] In view of the low efficiency of KNN query in the current big data environment and the shortcomings of existing technologies, we propose a KNN query method based on mixed granularity distributed memory grid index, combined with the characteristics of memory clusters, we estimate the overall data by summarizing , establish a coarse-fine mixed-grained distributed memory grid index structure to reduce data skew and improve data retrieval efficiency; design a non-lost neighbor fine-grained grid search algorithm to ensure fast and accurate positioning of the neighbors of the search object Grid, finally, based on the established index structure and non-lost neighbor fine-grained grid search algorithm, combined with the distributed memory computing model, the traditional centralized KNN algorithm can be distributed and extended to eliminate the single machine of the centralized KNN algorithm Performance bottlenecks and I / O bottlenecks based on the MapReduce architecture KNN algorithm, and then perform fast KNN queries on massive data

Images