An efficient GPU-based construction method of proximity graph index based on data locality

Abstract

The application belongs to the technical field of databases, and specifically discloses an efficient GPU near neighbor graph index construction method based on data locality, which comprises the following steps: dividing index original data into multiple subsets, and performing data processing on the multiple subsets to obtain rearranged data; constructing subgraphs corresponding to each subset in the rearranged data, and constructing a graph index based on the subgraphs; and refining the structure of the graph index to obtain a final graph index. The method can effectively alleviate the problems of insufficient memory resources and excessive calculation cost in the subgraph merging stage, significantly reduce the construction cost of the graph index, and effectively extend to large-scale data sets to improve scalability.

Description

Technical Field

[0001] This application belongs to the field of database-related technical technology, and more specifically, relates to an efficient GPU nearest neighbor graph index construction method based on data locality. Background Technology

[0002] Graph indexes typically suffer from high construction costs. This is because the process involves searching for nearby candidate outgoing neighbors for each vertex in the vector space and selecting the final outgoing neighbors through pruning rules, resulting in significant computational overhead. Given that the scale of vector data in industrial applications can reach hundreds of millions, efficiently constructing graph indexes becomes a major challenge. Existing methods for graph index construction and searching based on Central Processing Unit (CPU) platforms have several limitations, such as excessively long construction times and high search latency.

[0003] To address the aforementioned issues, existing technologies have proposed methods to accelerate graph construction and search by leveraging the parallel computing capabilities of Graphics Processing Units (GPUs). In graph index construction, GPU-based methods typically fall into two categories. The first category employs a unified processing strategy, directly storing the original vectors in GPU memory for processing. The second category uses a divide-and-conquer strategy, partitioning the original vectors into multiple subsets based on vector IDs, constructing a subgraph for each subset, and finally merging these subgraphs to generate the complete graph index. Despite these advancements, two main challenges remain:

[0004] First, existing methods suffer from scalability limitations. Graph indexes typically consume significant memory resources due to the need to store both the original vectors and the graph structure simultaneously. The limited on-chip memory of GPUs further exacerbates this problem. As data volume increases, unified processing methods face the limitation that not all vector data can be loaded into GPU memory, leading to graph index construction failures. Furthermore, while divide-and-conquer methods alleviate memory pressure to some extent through data partitioning, existing implementations still tend to experience insufficient memory resources during the subgraph merging phase, making it difficult to support the processing needs of large-scale vector data.

[0005] Secondly, existing methods are costly in terms of index construction. Unified processing methods require searching the global vector space when selecting outgoing neighbors for each vector. However, outgoing neighbor selection often exhibits locality, thus introducing significant unnecessary computational overhead. Existing divide-and-conquer methods typically partition the data based on vector ID order, resulting in poor locality within subsets. This means vectors within subsets are scattered across high-dimensional spaces and are far apart. Consequently, each vector's nearest neighbors are distributed across multiple subsets, necessitating extensive Approximate Nearest Neighbor (ANN) searches during the subgraph merging stage to ensure proximity relationships between vertices. This significantly increases computational overhead, making subgraph merging a new bottleneck in the construction process. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application aims to provide an efficient GPU nearest neighbor graph index construction method based on data locality, which addresses the problems of insufficient scalability and high construction cost of existing GPU nearest neighbor graph index construction methods.

[0007] To achieve the above objectives, in a first aspect, this application provides an efficient GPU nearest neighbor graph index construction method based on data locality, comprising:

[0008] The original index data is divided into multiple subsets, and the data of the multiple subsets is processed to obtain rearranged data;

[0009] Construct subgraphs corresponding to each subset in the rearranged data, and build graph indexes based on the subgraphs;

[0010] The structure of the graph index is refined to obtain the final graph index.

[0011] This application reduces the search space for selecting neighbors for each vector by partitioning the data with high data locality, and implements a subgraph merging method that does not require search operations. This effectively alleviates the problems of insufficient memory resources and excessive computational overhead in the subgraph merging stage, and significantly reduces the construction cost of the graph index. By decoupling the graph index construction task into multiple independent out-neighbor selection tasks for vectors, it can be effectively scaled to large-scale datasets, improving scalability. By refining the structure of the graph index, excellent graph search performance is ensured while maintaining low-cost construction.

[0012] According to the efficient GPU nearest neighbor graph index construction method based on data locality provided in this application, the method involves dividing the original index data into multiple subsets and processing the multiple subsets to obtain rearranged data, including:

[0013] The original index data is clustered to obtain multiple subsets;

[0014] Identify the boundary vectors of the plurality of subsets, and obtain the boundary subsets based on the boundary vectors;

[0015] The plurality of subsets and the boundary subset are used as the rearranged data.

[0016] According to the efficient GPU nearest neighbor graph index construction method based on data locality provided in this application, the construction of subgraphs corresponding to each subset in the rearranged data includes:

[0017] Construct navigation graphs corresponding to each subset of the rearranged data;

[0018] Based on the navigation graph, perform a beam search to obtain the candidate outgoing neighbors corresponding to each vector in each subset;

[0019] Based on Vaman pruning rules, suitable outgoing neighbors are selected from the candidate outgoing neighbors to complete the subgraph construction.

[0020] This application employs Vaman pruning rules during the graph index construction process to improve the diversity and connectivity of the graph index.

[0021] According to the efficient GPU nearest neighbor graph index construction method based on data locality provided in this application, the step of constructing a graph index based on the subgraph includes:

[0022] Multiple subgraphs are merged based on a pre-defined neighbor table to obtain an initial graph index;

[0023] Add reverse edges to the initial graph index to complete the construction of the graph index.

[0024] This application employs a reverse edge addition operation during the graph index construction process, thereby improving the diversity and connectivity of the graph index.

[0025] According to the efficient GPU nearest neighbor graph index construction method based on data locality provided in this application, the step of refining the structure of the graph index to obtain the final graph index includes:

[0026] Scan all vertices in the graph index and connect additional edges to vertices whose out-neighbor count does not reach a preset out-degree threshold.

[0027] According to the efficient GPU nearest neighbor graph index construction method based on data locality provided in this application, the method further includes:

[0028] Perform a query-aware search on the final graph index.

[0029] Secondly, this application provides an efficient GPU nearest neighbor graph index construction apparatus based on data locality, comprising:

[0030] The partitioning processing module is used to divide the original index data into multiple subsets and process the data of the multiple subsets to obtain rearranged data;

[0031] A construction module is used to construct subgraphs corresponding to each subset in the rearranged data, and to construct a graph index based on the subgraphs;

[0032] The refinement module is used to refine the structure of the graph index to obtain the final graph index.

[0033] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the efficient GPU nearest neighbor graph index construction method based on data locality as described in the first aspect or any possible implementation thereof.

[0034] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the efficient GPU nearest neighbor graph index construction method based on data locality as described in the first aspect or any possible implementation of the first aspect.

[0035] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to execute the efficient GPU nearest neighbor graph index construction method based on data locality as described in the first aspect or any possible implementation of the first aspect.

[0036] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0037] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art:

[0038] (1) This application reduces the search space when selecting neighbors for each vector by partitioning the data with high data locality, and implements a subgraph merging method without search operation. This effectively alleviates the problems of insufficient memory resources and excessive computational overhead in the subgraph merging stage, and significantly reduces the construction cost of the graph index. By decoupling the graph index construction task into multiple independent out-neighbor selection tasks for vectors, it can be effectively extended to large-scale datasets and improve scalability. By refining the structure of the graph index, excellent graph search performance is ensured while maintaining low-cost construction.

[0039] (2) This application uses Vamana pruning rules in the process of constructing the graph index, thereby improving the diversity and connectivity of the graph index.

[0040] (3) This application uses reverse edge addition operation in the graph index construction process to improve the diversity and connectivity of the graph index. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is one of the flowcharts illustrating the efficient GPU nearest neighbor graph index construction method based on data locality provided in this application embodiment;

[0043] Figure 2 This is a flowchart illustrating the data preprocessing algorithm provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the graph search process provided in an embodiment of this application;

[0045] Figure 4 This is a schematic diagram of the search initialization algorithm for query awareness of small-scale data provided in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of the search initialization algorithm for query awareness of large-scale data provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the construction performance comparison experiment provided in the embodiments of this application;

[0048] Figure 7 This is a schematic diagram of a search performance comparison experiment provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of a comparative experiment on scalable performance provided in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the structure of the efficient GPU nearest neighbor graph index construction device based on data locality provided in the embodiments of this application;

[0051] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0053] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0054] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0056] First, let's introduce the following content:

[0057] Large Language Models (LLMs) are transforming the way natural language processing tasks are implemented by enabling semantic search, knowledge retrieval, and text generation. These functions rely on efficient ANN search in vector space. Specifically, they require real-time retrieval of embeddings represented as high-dimensional vectors. ANN search is also a fundamental problem in many applications, including natural language processing, pattern recognition, data mining, machine learning, and recommender systems. These applications typically involve searching for the vector closest to a query vector in a high-dimensional space based on a given distance metric. Currently, various algorithms have been proposed for ANN search, including tree-based, quantization-based, hash-based, and graph-based methods. Among these, graph-based methods have attracted widespread attention due to their superior performance in both search accuracy and efficiency. Graph-based methods use a nearest-neighbor graph as the core index structure, organizing high-dimensional vectors into vertices of the graph and transforming the search process into traversing paths within the graph, thus achieving efficient search.

[0058] Next, combined Figures 1-9This paper introduces an efficient GPU nearest neighbor graph index construction method based on data locality provided in the embodiments of this application.

[0059] Figure 1 This is a flowchart illustrating the efficient GPU nearest neighbor graph index construction method based on data locality provided in this application embodiment, as shown below. Figure 1 As shown, the method includes the following steps:

[0060] Step 100: Divide the original index data into multiple subsets and process the data of the multiple subsets to obtain rearranged data;

[0061] For a given original index data, the original data is first preprocessed to transform it into rearranged data. The original index data refers to the vector data that the index needs to contain without any processing.

[0062] Data preprocessing involves dividing the original index vector into multiple subsets and generating a boundary subset. By merging the multiple subsets and the boundary subset, the original index vector is transformed into rearranged data.

[0063] The goal of local partitioning is to assign spatially nearest vectors to the same subset to ensure high data locality.

[0064] Step 110: Construct subgraphs corresponding to each subset in the rearranged data, and build graph indexes based on the subgraphs;

[0065] To achieve highly scalable and low-cost graph index construction, an iterative graph building strategy can be adopted to construct subgraphs, and a search-free subgraph merging method can be used to eliminate the dependencies between vectors in graph index construction, thereby decoupling the graph index construction task into a parallelizable vector out-neighbor selection task.

[0066] To ensure excellent search performance, a navigation graph can be constructed first. Then, based on the neighborhood information provided by the navigation graph and special pruning rules, neighbors are iteratively selected for each vector, ultimately constructing a graph index structure that approximates a sparse neighborhood graph (SNG).

[0067] Step 120: Refine the structure of the graph index to obtain the final graph index.

[0068] A graph structure with relatively high accuracy can be obtained by constructing a graph index. However, since the merging process only treats the boundary vectors specially, the graph structure is not optimal. In this case, some vectors that take the boundary vector as their out neighbor may have a close neighbor relationship with vectors in other subsets indirectly through the boundary vector, and thus will also be affected by the partition. Therefore, in order to improve accuracy and make full use of the allocated GPU resources, the structure of the graph index can be refined to ensure excellent graph search performance while ensuring low-cost construction.

[0069] The efficient GPU nearest neighbor graph index construction method based on data locality provided in this application reduces the search space when selecting neighbors for each vector through data partitioning with high data locality, and also implements a subgraph merging method without search operations. This effectively alleviates the problems of insufficient memory resources and excessive computational overhead in the subgraph merging stage, and significantly reduces the construction cost of the graph index. By decoupling the graph index construction task into multiple independent out-neighbor selection tasks for vectors, it can be effectively scaled to large-scale datasets, improving scalability. By refining the structure of the graph index, excellent graph search performance is ensured while maintaining low-cost construction.

[0070] In some embodiments, step 100 specifically includes:

[0071] Step 1001: Cluster the original index data to obtain multiple subsets;

[0072] Step 1002: Identify the boundary vectors of multiple subsets and obtain the boundary subsets based on the boundary vectors;

[0073] Step 1003: Use multiple subsets and boundary subsets as rearranged data.

[0074] Since a subgraph needs to be built on each subset, an unbalanced partition will lead to an unbalanced workload, thereby reducing the efficiency of the construction. The partition should generate subsets of relatively balanced size. Therefore, local partitioning can be achieved through clustering methods.

[0075] In one embodiment of this application, K-means clustering accurately partitions adjacent vectors into the same subset and achieves a relatively balanced partition of the dataset. Furthermore, this application decouples the construction tasks of multiple subgraphs into out-neighbor selection tasks for multiple vectors, transforming the parallel paradigm from the graph level to the vector level. This fine-grained parallelism amortizes the tail delay in the subgraph construction process onto the out-neighbor selection of different vectors, thereby further mitigating the impact of workload imbalance. Therefore, K-means clustering can be used to achieve local partitioning.

[0076] However, clustering algorithms typically require loading all data into memory, resulting in high computational costs and difficulty in scaling to large-scale datasets. To address this issue, a sampling-based clustering strategy can be employed, as follows:

[0077] Figure 2 This is a flowchart illustrating the data preprocessing algorithm provided in an embodiment of this application, as shown below. Figure 2 As shown, first set the subset number parameter. and sampling factor parameters From containing A random sample of size from the original data of vectors A subset is selected, preserving the overall data distribution. Then, K-means clustering is applied to the sampled subset to obtain... Each vector in the original data is assigned a centroid. Finally, each vector is concurrently scanned and assigned to a subset corresponding to its nearest centroid, thus obtaining... A subset with high data locality.

[0078] The boundary vector lies near the partition boundary, and its nearest neighbors are distributed across multiple distinct subsets. Assume the vector... Based on local partitioning, it is assigned to its nearest centroid. corresponding subset If another center of mass exists , making the distance and Almost the same, then It is also very likely to be related to corresponding subset The nearest neighbors of the vectors in the vector. At this time, lie in and Near the dividing boundary between them is a boundary vector.

[0079] Therefore, this application uses boundary factor parameters. To identify boundary vectors. For vectors Given the nearest and second nearest centroids and ,if If true, then the vector It is a boundary vector. Each boundary vector has a copy added to a subset of the boundaries.

[0080] In some embodiments, step 110 specifically includes:

[0081] Step 1101: Construct navigation graphs corresponding to each subset in the rearranged data;

[0082] Step 1102: Perform a beam search based on the navigation graph to obtain the candidate outgoing neighbors corresponding to each vector in each subset;

[0083] Step 1103: Select suitable outgoing neighbors from the candidate outgoing neighbors based on Vamana pruning rules to complete the subgraph construction.

[0084] Optionally, for each subset of the rearranged data, a K-Nearest Neighbor Graph (KNNG) is first quickly constructed using the GPU-based NN-Descent algorithm as a navigation graph, and a vertex is randomly selected as the starting vertex. All these vertices are maintained in a set for subsequent construction and searching. After completing the above steps, subgraph construction is performed concurrently at the vector granularity, selecting neighbors individually for each vector.

[0085] For vectors When selecting neighbors, to ensure that from the starting vertex to... A monotonic path exists, and the candidate neighbor set contains enough... The candidate vectors of the nearest neighbors will be selected by... The beam search results for the query vector are used as a candidate outgoing neighbor set. Since the beam search on the navigation graph starts from the initial vertex, the final route is to... Therefore, the search results retain information from the starting vertex to... The vectors along the path provide sufficient candidate outgoing neighbors for generating monotonic paths. Simultaneously, KNNG provides high-quality local neighborhood information, and the vectors in the search results set are all... . the neighbor.

[0086] When obtaining the candidate outgoing neighbor set, first set the size parameter of the candidate outgoing neighbor set. ,Will As the query vector, a bundle search is performed on the navigation graph starting from the initial vertex, with the bundle list size and result set size set to... , the search results vectors as The candidate neighbors.

[0087] To ensure that the generated graph index has a sufficient number of values ​​from the starting vertex to... A monotonic path, where along the monotonic path from the starting vertex to... As the distance between the starting vertex and the next vertex gradually decreases, Vamana pruning rules can be used to select suitable outgoing neighbors from the candidate outgoing neighbor set. Vamana pruning rules ensure that the outgoing neighbors are selected along a monotonic path from the starting vertex to the next vertex. The distance is reduced by at least one product factor, not just a factor, thus improving search efficiency. Simultaneously, Vaman pruning rules eliminate outgoing neighbors located in similar directions, ensuring that selected outgoing neighbors are evenly distributed across the target area. The variety of graph indexes is enhanced by considering all surrounding directions.

[0088] Specifically, it is necessary to set the out-degree threshold parameter. From the candidate neighbor set Iteration with the most choices One outgoing neighbor. When selecting outgoing neighbors, first, the vectors in the candidate outgoing neighbor set are arranged according to... Sort the neighbors in ascending order of distance. Then select the candidate neighbors from the distance range. The nearest vector As an outgoing neighbor, and removed from the candidate outgoing neighbor set. Then set the product factor parameter. Delete all that meet the criteria vector This process is repeated until the number of selected outgoing neighbors reaches a certain threshold. Or there are no remaining candidate neighbors.

[0089] In some embodiments, step 110 specifically includes:

[0090] Step 1104: Merge multiple subgraphs based on the preset neighbor table to obtain the initial graph index;

[0091] Step 1105: Add reverse edges to the initial graph index to complete the construction of the graph index.

[0092] The core data structure of a nearest neighbor graph is the out-neighbor table, where each row stores the out-neighbor ID of a vertex. Since subgraph construction is performed on rearranged data, the graph vertex IDs initially correspond to the vector IDs in the rearranged data. To build an index on the original data, the graph vertex IDs are mapped back to the vector IDs in the original data, thus merging the subgraphs on the rearranged data into a complete graph on the original data.

[0093] For non-boundary vectors in the original data, their vector IDs correspond to unique vector IDs in the rearranged data, and their outgoing neighbors can be directly mapped to the corresponding rows in the outgoing neighbor table.

[0094] For a boundary vector in the original data, it has two copies in the rearranged data: one in its own subset and the other in the boundary subset. These copies correspond to different vector IDs in the rearranged data and maintain their respective out-neighbors. These two sets of out-neighbors can be mapped to the same row in the out-neighbor table and merged to ensure that out-neighbors from the subset and the boundary subset each account for half, thus connecting the corresponding subgraph and the boundary subgraph.

[0095] After mapping the out-neighbor table, the subgraphs are merged using neighborhood information between boundary vectors. Since the subgraph merging method proposed in this application only involves mapping and update operations on the out-neighbor table, and there are no dependencies between vectors, it can be performed concurrently with high efficiency. Therefore, subgraph merging has high scalability and low computational cost.

[0096] After merging the subgraphs, a reverse graph can be constructed where all edges are oriented in the opposite direction to those in the merged graph. Since the in-degree of each vertex in the merged graph is unconstrained, the out-degree of each vertex in the reverse graph is also variable and may exceed [a certain value]. To ensure efficient GPU processing, the same out-degree threshold parameter can be set on the inverse graph. Reserve the most for each vertex Find the nearest outgoing neighbor. Then, add the edges retained in the reverse graph to the merged graph, thus ultimately generating a graph index with stronger connectivity.

[0097] In some embodiments, step 120 specifically includes:

[0098] Scan all vertices in the graph index and connect additional edges to vertices whose out-neighbor count does not reach a preset out-degree threshold.

[0099] First, concurrently scan all vertices in the graph. For each vertex, check if it has free space to store additional outgoing neighbors, i.e., whether its number of outgoing neighbors has reached a preset out-degree threshold. .

[0100] If the number of out-neighbors of a vertex does not reach the target The vector corresponding to the vertex is used as the query vector to perform an ANN search on the current graph structure, and the search results are used to fill the empty space, thereby enriching the vertex's out-neighbor information with more comprehensive neighborhood information. If the number of out-neighbors of a vertex has reached... If the vertex is not pruned, no additional operation is required. Since the refinement step only performs edge addition operations on a limited number of vertices without pruning, and the vertices are independent of each other and can be fully parallelized, the refinement step can effectively improve accuracy with low time overhead.

[0101] In some embodiments, the method further includes:

[0102] Perform a query-aware search on the final graph index.

[0103] Figure 3 This is a schematic diagram of the graph search process provided in the embodiments of this application, such as... Figure 3As shown, two query-aware beam search algorithms can be used to provide efficient ANN search for different query vectors, depending on the data size. Beam search maintains multiple candidate paths and applies backtracking to make the search results approach the global optimum.

[0104] A variable of size can be used (in , The list of bundles (the number of results to be returned) and its size are... A sequential array buffer, consisting of candidate lists, is used to store intermediate results during the search process. Each vertex in the bundle list is marked as either explored or unexplored. To avoid redundant distance calculations, a hash table can be used to track whether a node has been visited; for ease of subsequent description, a hash table will be used. This represents the query vector.

[0105] Figure 4 This is a schematic diagram of the query-aware search initialization algorithm for small-scale data provided in an embodiment of this application, as shown below. Figure 4 As shown, for small-scale data, subgraph construction attempts to guarantee that a monotonic path exists from the starting vertex to all other vertices. However, after subgraph merging, global monotonicity is difficult to guarantee. For example, although a monotonic path exists from the starting vertex to other vertices in a subgraph... arrive A monotonic path exists in another subgraph from... arrive The monotonic path, but in the merged graph arrive A monotonic path may not exist. This means that for a vector, the path from the starting node chosen from its subset to the vector remains monotonic, but the path from the starting node chosen from other subsets to the vector may no longer be monotonic. Therefore, choosing an appropriate starting node based on the query vector is crucial for ensuring global monotonicity and accelerating the search.

[0106] Since each vector is assigned to a subset corresponding to its nearest centroid, the search should also begin from the starting vertex corresponding to that centroid. Specifically, first determine the centroid closest to the query vector, and then perform the search starting from its corresponding starting node.

[0107] Figure 5 This is a schematic diagram of the query-aware search initialization algorithm for large-scale data provided in an embodiment of this application, as shown below. Figure 5As shown, for large-scale data, since the original vectors and graph indexes cannot be stored in GPU memory simultaneously, a large amount of CPU-GPU data transfer is required during the search process, severely limiting search efficiency. This problem can be solved using an optimized graph index structure and search algorithm. In terms of graph index construction, subgraph merging is no longer performed; instead, multiple subgraphs are used together to form the index.

[0108] Because each subgraph exhibits high data locality, searching only a subset of subgraphs is often sufficient to achieve high search accuracy. The number of subgraphs required to search varies depending on the query vector. Query vectors located near the partition boundary typically require searching multiple nearby subgraphs to ensure accuracy, while query vectors far from the partition boundary require fewer subgraphs. Therefore, the subgraphs to be searched can be dynamically selected based on the query vector. Specifically, given the set of centroids corresponding to the subgraphs, the distances are first determined. Recent center of mass Then check each centroid. ,if Then The corresponding subgraphs are included in the search scope. A minimum number of subgraphs can be set to avoid losing results due to insufficient subgraphs being searched.

[0109] like Figure 3 As shown, the next step is to perform a beam search on the graph. The specific steps are as follows:

[0110] 1. First calculate The distances between the starting vertex and all outgoing neighbors are calculated, and the outgoing neighbor IDs and distance pairs are stored in a candidate list. The bundle list is then initialized with invalid entries, where each entry is assigned a sufficiently large distance to ensure they are replaced in subsequent updates.

[0111] 2. Select the distance from the bundle list and the candidate list. Recent We have vertices and store them in a bundle list, sorted in ascending order of distance. Since the buffer is implemented as an array, a sorting algorithm can be applied for efficient updates. First, we sort the candidate list using bitonic sort. Because the bundle list is always sorted in ascending order of distance, the merging process using bitonic sort can merge the sorted candidate list with the bundle list, ensuring that the bundle list stores the distances from the buffer. Recent One vertex;

[0112] 3. Select the distance from the bundle list. The most recently unexplored vertex is designated as the currently explored vertex and marked as explored. To avoid additional overhead when tracking vertex exploration status, the most significant bit (MSB) of the vertex ID can be used to represent its exploration status; an MSB of 1 indicates exploration, and 0 indicates unexplored. Since the bundle list is sorted in ascending order of distance, the first unexplored vertex is the one with the highest distance. The most recently explored vertices can be selected using a single warp: each thread in the warp is responsible for checking the MSB of a vertex and then using warp-level instructions. and Identify the first unexplored vertex in the beam list;

[0113] 4. For each outgoing neighbor of an explored vertex, first add it to the candidate list and check if it has already been visited. Each vertex is only computed with its neighbor on its first visit. The distance between the nodes is calculated. If a vertex has appeared in the candidate list before, its distance has already been calculated in a previous iteration. In this case, there are only two possibilities: if the distance is small enough, the vertex will remain in the bundle list; if the distance is too large, the vertex should not be considered again. Therefore, there is no need to recalculate the distance. The distance calculation is then performed in batches at the warp level: each warp is responsible for calculating the distance of one vertex, each thread in the warp calculates a partial distance on a specific dimension, and then uses warp reduction to summarize these partial distances to obtain the final distance;

[0114] 5. Iterate through steps 2 to 4 until all vertices in the bundle list have been explored. Finally, the search will calculate the distances in the bundle list. Recent The vertices are used as the result of the ANN search.

[0115] Figure 6 This is a schematic diagram of the construction performance comparison experiment provided in the embodiments of this application, such as... Figure 6 As shown, this application presents an efficient GPU nearest neighbor graph index construction method based on data locality, as well as comparative experiments on the construction performance of two GPU-based methods (CAGRA and GGNN) and a CPU-based method (NSG). CAGRA is currently the most advanced GPU-based method; GGNN adopts a divide-and-conquer strategy, dividing the data according to the vector ID order and using ANN search to merge subgraphs; NSG is a traditional approximate SNG graph, and its construction and search process is similar to that of this application.

[0116] Experiments were conducted on the SIFT1M dataset. To ensure fairness, all methods used the same precision target to construct graph indices. It can be seen that when constructing graph indices with the same precision, the construction efficiency of this application is significantly higher than other methods. Specifically, the construction speed of this application is 2.2 times that of CAGRA, 3.8 times that of GGNN, and 45.7 times that of NSG.

[0117] Figure 7 This is a schematic diagram of a search performance comparison experiment provided in an embodiment of this application, such as... Figure 7 As shown, the search performance of this application and CAGRA is almost identical, and both outperform GGNN and NSG. Overall, this application demonstrates competitive search performance compared to existing GPU-based graph indexing methods.

[0118] Figure 8 This is a schematic diagram of a scalable performance comparison experiment provided in an embodiment of this application, such as... Figure 8 As shown, to simulate the scenario where the total memory usage of the original vectors and graph structures exceeds GPU memory, the SIFT100M dataset was used in the experiment. In this case, CAGRA fails to construct its graph due to its uniform processing strategy; while GGNN provides a GPU sharding implementation to handle large-scale datasets. Therefore, this application only compares with GGNN. It can be seen that, under the same accuracy target, the graph index construction speed of this application is 1.9 times that of GGNN; in terms of search performance, this application is 2.0 times faster than GGNN in the recall range of 80% to 95%.

[0119] The efficient GPU nearest neighbor graph indexing method based on data locality provided in this application has flexibly adjustable parameters, allowing for trade-offs between indexing accuracy and construction efficiency. Different parameter settings can be used for different use cases and requirements, thus adapting to more complex real-world scenarios.

[0120] Figure 9 This is a schematic diagram of the structure of an efficient GPU nearest neighbor graph index construction device based on data locality provided in an embodiment of this application, as shown below. Figure 9 As shown, the system includes a partitioning module 910, a construction module 920, and a refinement module 930, wherein:

[0121] The partitioning and processing module 910 is used to divide the original index data into multiple subsets and process the data of the multiple subsets to obtain rearranged data.

[0122] Module 920 is used to construct subgraphs corresponding to each subset in the rearranged data, and to construct graph indexes based on the subgraphs;

[0123] The refinement module 930 is used to refine the structure of the graph index to obtain the final graph index.

[0124] Based on the methods in the above embodiments, Figure 10 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 10 As shown in the illustration, this application provides an electronic device that may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. The processor 1010, communications interface 1020, and memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions stored in the memory 1030 to execute the efficient GPU nearest neighbor graph index construction method based on data locality described in the above embodiment.

[0125] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the efficient GPU nearest neighbor graph index construction method based on data locality described in the various embodiments of this application.

[0126] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program. When the computer program runs on a processor, it causes the processor to execute the efficient GPU nearest neighbor graph index construction method based on data locality in the above embodiments.

[0127] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the efficient GPU nearest neighbor graph index construction method based on data locality in the above embodiments.

[0128] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0129] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0130] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0131] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0132] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for constructing an efficient GPU nearest neighbor graph index based on data locality, characterized in that, include: The original index data is divided into multiple subsets, and the data of the multiple subsets is processed to obtain rearranged data; Construct subgraphs corresponding to each subset in the rearranged data, and build graph indexes based on the subgraphs; The structure of the graph index is refined to obtain the final graph index; The process of dividing the original index data into multiple subsets and processing the subsets to obtain rearranged data includes: The original index data is clustered to obtain multiple subsets; Identify the boundary vectors of the plurality of subsets, and obtain the boundary subsets based on the boundary vectors; The plurality of subsets and the boundary subsets are used as the rearranged data; The construction of subgraphs corresponding to each subset in the rearranged data includes: Construct navigation graphs corresponding to each subset in the rearranged data; Based on the navigation graph, perform a beam search to obtain the candidate outgoing neighbors corresponding to each vector in each subset; Based on Vamana pruning rules, suitable outgoing neighbors are selected from the candidate outgoing neighbors to complete the subgraph construction; The construction of a graph index based on the subgraph includes: Multiple subgraphs are merged based on a pre-defined neighbor table to obtain an initial graph index; Add reverse edges to the initial graph index to complete the construction of the graph index; The process of merging multiple subgraphs based on a pre-defined neighbor table to obtain an initial graph index includes: For non-boundary vectors in the original data, their out-neighbors are directly mapped to the corresponding rows in the out-neighbor table; For a boundary vector in the original data, it has two copies in the rearranged data: one in its own subset and the other in the boundary subset. These two sets of out neighbors are mapped to the same row in the out neighbor table and merged to ensure that out neighbors from the subset and the boundary subset each account for half, thereby connecting the corresponding subgraph and the boundary subgraph.

2. The efficient GPU nearest neighbor graph index construction method based on data locality according to claim 1, characterized in that, The refinement of the graph index structure to obtain the final graph index includes: Scan all vertices in the graph index and connect additional edges to vertices whose out-neighbor count does not reach a preset out-degree threshold.

3. The efficient GPU nearest neighbor graph index construction method based on data locality according to claim 1, characterized in that, The method further includes: Perform a query-aware search on the final graph index.

4. A highly efficient GPU nearest neighbor graph index construction device based on data locality, characterized in that, include: The partitioning processing module is used to divide the original index data into multiple subsets and perform data processing on the multiple subsets to obtain rearranged data; A construction module is used to construct subgraphs corresponding to each subset in the rearranged data, and to construct a graph index based on the subgraphs; The refinement module is used to refine the structure of the graph index to obtain the final graph index; The partitioning processing module is specifically used for: The original index data is clustered to obtain multiple subsets; Identify the boundary vectors of the plurality of subsets, and obtain the boundary subsets based on the boundary vectors; The plurality of subsets and the boundary subsets are used as the rearranged data; The building module is specifically used for: Construct navigation graphs corresponding to each subset of the rearranged data; Based on the navigation graph, perform a beam search to obtain the candidate outgoing neighbors corresponding to each vector in each subset; Based on Vamana pruning rules, suitable outgoing neighbors are selected from the candidate outgoing neighbors to complete the subgraph construction; Multiple subgraphs are merged based on a pre-defined neighbor table to obtain an initial graph index; Add reverse edges to the initial graph index to complete the construction of the graph index; The process of merging multiple subgraphs based on a pre-defined neighbor table to obtain an initial graph index includes: For non-boundary vectors in the original data, their out-neighbors are directly mapped to the corresponding rows in the out-neighbor table; For a boundary vector in the original data, it has two copies in the rearranged data: one in its own subset and the other in the boundary subset. These two sets of out neighbors are mapped to the same row in the out neighbor table and merged to ensure that out neighbors from the subset and the boundary subset each account for half, thereby connecting the corresponding subgraph and the boundary subgraph.

5. An electronic device, characterized in that, include: At least one memory for storing computer programs; At least one processor is configured to execute a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the efficient GPU nearest neighbor graph index construction method based on data locality as described in any one of claims 1-3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is run on a processor, the processor performs the efficient GPU nearest neighbor graph index construction method based on data locality as described in any one of claims 1-3.

7. A computer program product, characterized in that, When the computer program product is run on a processor, the processor performs the efficient GPU nearest neighbor graph index construction method based on data locality as described in any one of claims 1-3.

Images