High-dimensional vector approximate nearest neighbor retrieval method for persistent storage

Abstract

The invention discloses a high-dimensional vector approximate nearest neighbor retrieval method for persistent storage. The method comprises an index construction stage and a search stage. In the index construction stage, a clustering algorithm is adopted to carry out space division on a data set, a center replacement strategy is introduced to select a representative vector for establishing an entry point diagram, and meanwhile, a second-layer neighbor graph is constructed on the original data set; in the search stage, the query vector is quickly positioned to the closest entry point by using the result of the entry point diagram, so that the search path is shortened. According to the method, the optimal product quantization coding strategy is adopted, so that the SSD data loading overhead is reduced; a multi-beam search strategy is adopted to effectively utilize the page-by-page reading characteristic of a file system, so that the time overhead is reduced, and the data loading amount is increased; and an iterative search strategy is adopted to collect query vector search features, the search process is self-adapted, and the search efficiency is improved. Aiming at mixed architecture challenges of large-scale data, the invention provides an efficient search solution for optimizing approximate nearest neighbor search.

Description

Technical Field

[0001] The present invention relates to the technical field of vector retrieval, and in particular to a high-dimensional vector approximate nearest neighbor retrieval method oriented to persistent storage. Background Art

[0002] With the rapid development of the Internet, unstructured data such as videos and pictures have exploded, bringing challenges to data retrieval. Traditional nearest neighbor search methods are inefficient when processing large-scale high-dimensional data. To improve the search speed, researchers have developed an approximate nearest neighbor search algorithm (ANNS) that reduces computational overhead by sacrificing a certain degree of accuracy. Existing ANNS mainly focuses on the use of volatile storage, which is too costly when facing TB-level data sets. Under the hybrid architecture, algorithms based on neighbor graphs face the dual challenges of high data loading overhead and long search paths. Summary of the invention

[0003] In response to the above-mentioned problems in the prior art, the present invention proposes a novel high-dimensional vector approximate nearest neighbor retrieval method for persistent storage, which is based on a two-layer index structure, combined with multiple entry points, neighbor graph and product quantization technology, and can improve the efficiency and accuracy of searching large-scale data sets.

[0004] The purpose of the present invention can be achieved through the following technical solutions.

[0005] The present invention provides a high-dimensional vector approximate nearest neighbor retrieval method for persistent storage, including an index building phase and a search phase; wherein: 1. Index construction phase The index structure consists of two layers: the first layer is the entry point graph index, which uses the k-means clustering algorithm to partition the vector space and generates the entry point graph through the center replacement strategy; the second layer is the neighbor graph index, which uses the NSG structure to optimize the construction of the neighbor graph to improve the search efficiency; 2. Search Phase Using the entry point graph results, the query vector is quickly located to the closest entry point, entry points are adaptively assigned to the query vector, and a multi-beam search strategy is performed to optimize the data loading process.

[0006] In the present invention, the entry point graph index is stored in the main memory, and the neighbor graph index is stored in the persistent storage SSD.

[0007] In the present invention, in the index construction stage, the method for constructing an entry point graph index includes: firstly, using the k-means clustering algorithm to spatially divide the data set to form multiple clusters; then, using the center replacement strategy for each cluster, selecting a representative vector to construct an entry point graph; wherein, the center replacement strategy includes: a) calculating the distance from the cluster center to the vector in each cluster; b) selecting the vector with the closest distance as the new cluster center. Furthermore, by merging similar nodes or deleting isolated nodes, the construction of the entry point graph is optimized.

[0008] In the present invention, in the index construction stage, the method for constructing a neighbor graph index includes: first, starting from each vector in the data set, using the nearest neighbor search algorithm to determine the K nearest neighbor vectors of each vector, these nearest neighbor vectors and the original vector together constitute the nodes in the graph, and the edges between the nodes represent the nearest neighbor relationship between them; then, the NSG algorithm is used to optimize the initially constructed K nearest neighbor graph. Furthermore, after the NSG algorithm is optimized, a greedy search algorithm is executed, and at each step, the node most similar to the current query vector is selected as the next step of the search target to further refine the neighbor graph.

[0009] Furthermore, in the present invention, in the index construction stage, the optimal product quantization coding strategy is used to compress and store data, and the original vector is quantized and encoded using a codebook; specifically, the following steps are included: 1) Determine the dimension of the compressed vector; 2) Divide the original vector into multiple subspaces; 3) Perform k-means clustering on each subspace to obtain the cluster center and form a product quantization codebook; 4) Use the product quantization codebook to quantize and encode the original vector.

[0010] In the present invention, the steps of the search phase are specifically as follows: S1. In the first layer, the query vector is operated as follows; a) Encode the query vector using the product quantization codebook to obtain a compressed query vector; b) Perform a greedy search on the first-level entry point graph index to find the vertices that are most similar to the encoded query vector. These vertices serve as the entry points for the search. c) assigning one or more entry points according to the characteristics of the query vector to adapt to different search requirements; S2. In the second layer, according to the entry obtained in the first layer, perform the following steps: a) Starting from the entry point, perform a breadth-first search (BFS) on the second-level neighbor graph index, using a dynamic search list to speed up the search process; b) Load relevant full-precision vector data from persistent storage to main memory as needed during the search process; c) Calculate the similarity between the loaded full-precision vector and the query vector to determine their approximate nearest neighbor relationship; d) Merge and sort all the candidate neighbor vectors obtained from the search, and select the top K vectors that are most similar to the query vector; S3. Return the top K most similar vectors in the query results.

[0011] In the present invention, after step S3, step S4 is also included: using feature learning to select the results after the first round of search as additional features, predicting the number of search rounds required for the query vector through the trained model, and realizing an iterative search strategy.

[0012] Compared with the prior art, the present invention has the following beneficial effects: By combining multiple entry points and neighbor graph product quantization technology, the algorithm optimizes the data structure and reduces the search path, thereby improving the search efficiency. In addition, the adaptive entry point allocation strategy adopted by the algorithm can quickly locate the closest entry point according to the characteristics of the query vector, effectively shortening the search time. At the same time, the multi-beam search strategy and iterative search strategy further improve the data loading speed and search accuracy, making the AepANN algorithm significantly better than the existing approximate nearest neighbor search algorithm in performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The following is a schematic diagram of constructing a double-layer index of the present invention on a specific set of data as an example.

[0014] Figure 2 For Figure 1 The adaptive search process of executing a query statement on the built index.

[0015] Figure 3 Performance test results on different data sets in the embodiments; (a) MSTURING10M, (b) DEEP10M, (c) BIGANN10M, (d) SPACEV10M. DETAILED DESCRIPTION

[0016] The present invention is described in detail below in conjunction with the accompanying drawings and the index construction and search method.

[0017] The present invention provides a high-dimensional vector approximate nearest neighbor retrieval method (AepANN) for persistent storage, which is used for efficient vector retrieval in large-scale data sets. The algorithm combines the nearest neighbor graph and product quantization technology; it includes: a) Use the k-means clustering algorithm to spatially partition the data set to form multiple clusters; b) A center replacement strategy is used for each cluster to select representative vectors to construct an entry point graph; c) constructing a second-layer neighbor graph on the original data set; d) Using the best product quantization coding strategy to compress and store data; e) quantize and encode the original vector using the codebook; f) Use neighbor graph index optimization strategy to build index structure; g) In the search phase, adaptively assign entry points to query vectors; h) Implement a multi-beam search strategy to optimize the data loading process.

[0018] The index structure of the AepANN algorithm consists of two layers: the first layer is the entry point graph index, which uses the k-means clustering algorithm to divide the vector space and generates the entry point graph through the center replacement strategy; the second layer is the neighbor graph index, which uses the NSG structure to optimize the construction of the neighbor graph to improve search efficiency.

[0019] In the index building stage, the AepANN algorithm spatially divides the data set through a clustering algorithm and selects the vector point closest to the center of each cluster as the entry point, thereby achieving adaptive entry point allocation for different query vectors.

[0020] The AepANN algorithm uses the best product quantization coding strategy to perform spatial rotation on the original vector through the rotation matrix to reduce the quantization error and improve the accuracy of the search. The process is as follows: a) Determine the dimension of the compressed vector; b) Divide the original vector into multiple subspaces; c) Perform k-means clustering on each subspace to obtain the cluster center and form a codebook; d) Use the codebook to quantize and encode the original vector.

[0021] The search process of the AepANN algorithm includes initialization, pre-computation, adaptive allocation of entry points, neighbor graph search, data loading, full-precision comparison, and result merging to optimize the search performance under the hybrid architecture. The process is as follows: a) receiving a query vector and performing pre-calculation using a codebook to obtain an encoded query vector; b) performing greedy search on the entry point graph to adaptively assign entry points based on the characteristics of the query vector; c) Starting from the entry point, perform a breadth-first search on the neighbor graph and use a dynamic search list to speed up the search process; d) Adopting a multi-beam search strategy, we select the global number of the closest vertex according to the order of the query vector encoding distance and load multiple nearby precise vectors from the SSD; e) Perform full-precision similarity calculation on the loaded precise vector to improve the accuracy of search results; f) Based on the full-precision comparison results, update the dynamic search list and continue the search process until the end condition is met; g) Merge and sort the search results, and extract the most similar first K vectors as the final search results; h) Use feature learning to select the results after the first round of search as additional features, and use the trained model to predict the number of search rounds required for the query vector to implement an iterative search strategy.

[0022] The adaptive allocation entry point process is as follows: a) Perform a greedy search on the entry point graph to find the vertex that is most similar to the query vector; b) Use the found vertex as the starting point of the nearest neighbor graph search.

[0023] The AepANN algorithm introduces a multi-beam search acceleration strategy in the search phase, which accelerates data loading through the page-by-page reading feature and improves the algorithm's throughput. The process is as follows: a) Select the entry point number of the closest vertex according to the order of the query vector encoding distance; b) Calculate the index file offset distance according to the global sequence number and perform data loading operations; c) Load multiple precise vectors that are adjacent in the file index simultaneously.

[0024] The following is a specific implementation method.

[0025] The present application proposes a dual-layer index, including a first-layer entry point graph and a second-layer neighbor graph.

[0026] The acquisition of the first-level entry point graph is a key step in achieving efficient search. The process begins by using the k-means clustering algorithm to spatially partition the entire data set and assign the vectors in the data set to a predetermined number of clusters, each of which contains vectors with similar characteristics. In the initial stage, a portion of the vectors are randomly selected as the center points of the clusters, also known as the centroid. During the iteration process, the positions of these center points are updated according to the rules of the clustering algorithm. For each cluster, the distance between all vectors in the cluster and the current cluster center is calculated, and the vector closest to the cluster center is selected to replace the current cluster center.

[0027] Next, the center point of each cluster, these representative vectors, is used as a node in the graph to construct the entry point graph. The edges between nodes are established based on their mutual relationships in the vector space, usually relying on distance or similarity. In order to improve the search efficiency, the entry point graph may need to be optimized, such as merging similar nodes or deleting isolated nodes. Finally, the constructed entry point graph is stored in the main memory because it is relatively small and needs to be accessed frequently during the search process. The entry point graph constructed in this way will be used to receive query vectors and quickly locate the most relevant entry points, thereby accelerating the subsequent search process. Through these steps, the AepANN algorithm can quickly find one or more entry points most relevant to the query for each query vector, significantly improving the efficiency and accuracy of approximate nearest neighbor search.

[0028] The construction of the second-layer nearest neighbor graph plays a key role in the AepANN algorithm. It further optimizes the search efficiency based on the first-layer entry point graph. First, starting from each vector in the data set, the NN-Descent algorithm or other efficient nearest neighbor search algorithms are used to determine the K nearest neighbor vectors of each vector. These nearest neighbor vectors and the original vector together constitute the nodes in the graph, and the edges between the nodes represent the nearest neighbor relationship between them. Then, the NSG (Navigating Spreading-out Graph) algorithm is used to optimize the initially constructed K nearest neighbor graph, controlling the complexity of the graph by cutting unnecessary edges while maintaining the connectivity and search efficiency of the graph. The NSG algorithm simulates the random walk process to remove edges that contribute less to the search path while retaining key connections.

[0029] After NSG optimization, a greedy search algorithm is executed to further refine the neighbor graph to ensure that the neighbors of each node are the most likely neighbor candidates. The greedy search algorithm selects the node most similar to the current query vector as the next step of the search at each step. Finally, this optimized neighbor graph index is stored in a persistent storage (such as SSD) so that it can be loaded into the main memory on demand when needed. When a query request is received, the constructed neighbor graph will start from the entry point determined by the entry point graph and perform an accurate approximate nearest neighbor search, thereby maintaining high efficiency and accuracy throughout the search process.

[0030] The query method of the above double-layer index specifically includes the following steps: S1. In the first layer, the query vector is operated as follows; a) Encode the query vector using the product quantization codebook to obtain a compressed query vector; b) Perform a greedy search on the first-level entry point graph index to find the vertices that are most similar to the encoded query vector. These vertices serve as the entry points for the search. c) Depending on the characteristics of the query vector, one or more entry points may be assigned to accommodate different search requirements.

[0031] S2. In the second layer, according to the entry obtained in the first layer, perform the following steps: a) Starting from the entry point, perform a breadth-first search (BFS) on the second-level neighbor graph index, using a dynamic search list to speed up the search process; b) Load relevant full-precision vector data from persistent storage (such as SSD) to main memory as needed during the search process; c) Calculate the similarity between the loaded full-precision vector and the query vector to determine their approximate nearest neighbor relationship; d) Merge and sort all the candidate neighbor vectors obtained from the search, and select the top K vectors that are most similar to the query vector.

[0032] S3. Return the top K most similar vectors in the query results.

[0033] The inventors used the above method to test on the MSTURING10M, DEEP10M, BIGANN10M, and SPACEV10M datasets. The results are as follows: Figure 3 As shown in the figure, the 10 closest vectors are selected, and their accuracy is improved by 30%-200% compared with DiskANN.

Claims

1. A high-dimensional vector approximate nearest neighbor retrieval method for persistent storage, characterized in that: It includes the index building phase and the search phase; among them:

1. Index construction phase The index structure consists of two layers: the first layer is the entry point graph index, which uses the k-means clustering algorithm to partition the vector space and generates the entry point graph through the center replacement strategy; the second layer is the neighbor graph index, which uses the NSG structure to optimize the construction of the neighbor graph to improve the search efficiency; 2. Search Phase Using the entry point graph results, the query vector is quickly located to the closest entry point, entry points are adaptively assigned to the query vector, and a multi-beam search strategy is performed to optimize the data loading process.

2. The high-dimensional vector approximate nearest neighbor retrieval method for persistent storage according to claim 1, characterized in that: The entry point graph index is stored in the main memory, and the neighbor graph index is stored in the persistent storage SSD.

3. The high-dimensional vector approximate nearest retrieval method for persistent storage according to claim 1 is characterized in that: In the index construction phase, the method for constructing an entry point graph index includes: first, using the k-means clustering algorithm to spatially divide the data set to form multiple clusters; then, using a center replacement strategy for each cluster, selecting representative vectors to construct an entry point graph; wherein the center replacement strategy includes: a) for each vector in each cluster, calculating the distance to the cluster center; b) selecting the vector with the closest distance as the new cluster center.

4. The high-dimensional vector approximate nearest retrieval method for persistent storage according to claim 3 is characterized in that: During the index building phase, the entry point graph is optimized by merging similar nodes or deleting isolated nodes.

5. The high-dimensional vector approximate nearest retrieval method for persistent storage according to claim 1 is characterized in that: In the index construction stage, the method of constructing a neighbor graph includes: first, starting from each vector in the data set, the nearest neighbor search algorithm is used to determine the K nearest neighbor vectors of each vector. These nearest neighbor vectors and the original vector together constitute the nodes in the graph, and the edges between the nodes represent the nearest neighbor relationship between them; then, the NSG algorithm is used to optimize the initially constructed K nearest neighbor graph.

6. The high-dimensional vector approximate nearest retrieval method for persistent storage according to claim 5 is characterized in that: After the NSG algorithm is optimized, a greedy search algorithm is executed, and at each step, the node most similar to the current query vector is selected as the next step of the search to further refine the neighbor graph.

7. The high-dimensional vector approximate nearest retrieval method for persistent storage according to claim 1 is characterized in that: In the index construction phase, the optimal product quantization coding strategy is used to compress and store data, and the codebook is used to quantize and encode the original vector; specifically, it includes: 1) Determine the dimension of the compressed vector; 2) Divide the original vector into multiple subspaces; 3) Perform k-means clustering on each subspace to obtain the cluster center and form a product quantization codebook; 4) Use the product quantization codebook to quantize and encode the original vector.

8. The high-dimensional vector approximate nearest neighbor retrieval method for persistent storage according to any one of claims 1 to 7, characterized in that: The steps of the search phase are as follows: S1. In the first layer, the query vector is operated as follows; a) Encode the query vector using the product quantization codebook to obtain a compressed query vector; b) Perform a greedy search on the first-level entry point graph index to find the vertices that are most similar to the encoded query vector. These vertices serve as the entry points for the search. c) assigning one or more entry points according to the characteristics of the query vector to adapt to different search requirements; S2. In the second layer, according to the entry obtained in the first layer, perform the following steps: a) Starting from the entry point, perform a breadth-first search (BFS) on the second-level neighbor graph index, using a dynamic search list to speed up the search process; b) Load relevant full-precision vector data from persistent storage to main memory as needed during the search process; c) Calculate the similarity between the loaded full-precision vector and the query vector to determine their approximate nearest neighbor relationship; d) Merge and sort all the candidate neighbor vectors obtained from the search, and select the top K vectors that are most similar to the query vector; S3. Return the top K most similar vectors in the query results.

9. The high-dimensional vector approximate nearest neighbor retrieval method for persistent storage according to claim 8, characterized in that: After step S3, step S4 is also included: using feature learning to select the result after the first round of search as an additional feature, predicting the number of search rounds required for the query vector through the trained model, and realizing an iterative search strategy.

Images