Distributed approximate neighbor search graph index construction method and device

Abstract

The invention provides a method and a device for constructing a distributed approximate neighbor search graph index. The method comprises the following steps of: initializing each process, dividing batches for a read-only vector data set, and distributing a message buffer area; if the value of the counter is an even number, unprocessed data set batches are taken out, an adjacency list of each point is calculated in parallel, and a tuple is generated; if the counter value is an odd number, determining the affected points in the graph index updating process, and performing parallel pruning again and updating; the adjacency list length information is written into an adjacency list length array in parallel in the process; copying the information to a message sending buffer area in parallel by utilizing a parallel prefix and a calculation offset; broadcasting a sending message buffer area of each process; deserializing a tuple from the received message buffer area by each process, and updating a graph index in the process; generating tuples for the adjacency table of the updated points; printing related results by one process and outputting a graph index. According to the method, the index construction time is greatly shortened; and higher query accuracy can be provided in a query stage.

Description

Technical Field

[0001] This invention belongs to the field of graph data processing and analysis technology, specifically relating to a method and apparatus for constructing a distributed approximate nearest neighbor search graph index. Background Technology

[0002] Graph-based Approximate Nearest Neighbor Search (ANNS) is a method for quickly finding the approximate nearest neighbors of data points in a high-dimensional space. It optimizes search efficiency by constructing a graph index structure. Although graph-based ANNS search is computationally efficient and requires fewer distance calculations, the graph index construction process is inefficient and time-consuming when dealing with large-scale datasets.

[0003] ParlayDiskANN is one of the most advanced ANNS graph index building algorithms. It uses multi-threaded parallel computing to reduce index building time and significantly improves the scalability and efficiency of graph index building through parallelization technology. However, its performance is still limited by single-machine hardware resources (such as memory capacity and computing power), making it difficult to further scale to larger datasets or higher concurrency requirements.

[0004] Pyramid is an implementation of a distributed method for constructing ANN graph structures. It builds the ANN graph by having different computing nodes construct the meta-graph and multiple sub-graphs. However, the Pyramid algorithm performs a two-level search from the meta-graph to the sub-graphs during the search process, which also involves communication between different nodes. This results in low overall search efficiency, with query throughput (QPS) and recall rate lower than the graph index structure of ParlayDiskANN.

[0005] The core differences between graph-based approximate nearest neighbor search methods lie primarily in their proximity graph construction strategies: DiskANN employs a greedy search mechanism to iteratively construct the relative neighborhood graph; HNSW introduces a hierarchical structure to construct a graph model with small-world properties to support efficient starting point localization; and NSG reduces graph out-degree and improves query efficiency by systematically pruning the long sides of triangles in the proximity graph. Since these methods typically require pre-calculating the proximity relationships of all samples in the dataset, their index construction process often involves high computational overhead, resulting in significantly longer index construction times compared to tree-based, hash-based, or quantization-based methods. This becomes a major bottleneck restricting their application in large-scale real-world scenarios.

[0006] Therefore, there is an urgent need for a distributed ANNS graph index structure construction method that can solve the above technical problems and support large-scale datasets. Summary of the Invention

[0007] Purpose of the invention: To address these issues, this invention proposes a distributed ANNS graph index structure construction method and apparatus. By extending the index construction process from single-machine multi-threaded parallelism to distributed multi-machine multi-threaded parallelism, it solves the problem of excessively long index construction time on large-scale datasets, while simultaneously ensuring that the quality of the graph index structure is the same as that of existing single-machine multi-threaded parallel methods.

[0008] Technical solution: The present invention provides a method for constructing a distributed approximate nearest neighbor search graph index, comprising the following steps:

[0009] (1) Initialize each process, divide the read-only vector dataset into batches, allocate message buffers, and set the iteration counter Superstep=0;

[0010] (2) If the value of the counter Superstep is an even number, then proceed to step (3); if the value of the counter Superstep is an odd number, then proceed to step (8).

[0011] (3) If there are unprocessed batches, retrieve the unprocessed dataset batches, compute the adjacency list of each point in parallel, and generate tuples; if there are no unprocessed batches, execute step (10).

[0012] (4) The adjacency list length information is written to the adjacency list length array in parallel within the process;

[0013] (5) Calculate the offset using parallel prefix sums and copy the information to the message sending buffer in parallel within the process;

[0014] (6) Broadcast the message buffer of each process and increment the iteration counter Superstep by 1;

[0015] (7) Each process deserializes tuples from the received message buffer and updates the in-process graph index;

[0016] (8) Identify the points affected during the graph index update process, and re-prune and update in parallel;

[0017] (9) Generate tuples for the adjacency list of the updated points: For each process, the vector updated in step (8) is... Its updated set of neighbors Generating tuples For all tuples generated in this process, execute steps (4) to (7). After execution, jump to step (2) for execution.

[0018] (10) One of the processes prints the relevant results and outputs the graph index of the process. .

[0019] Furthermore, the implementation process of step (1) is as follows:

[0020] The set of all processes involved in the computation Create and initialize an identical shared object in their respective memory locations in parallel. The shared object Includes all global information required for execution, including an empty nearest neighbor graph index. At this time, the vertex set and edge set All empty; a complete vector dataset that remains read-only throughout the subsequent process. A counter, Superstep, used to record the number of communication iterations in the program, initialized to 0; and a user-defined upper limit parameter for the number of neighbors for each vector in the nearest neighbor graph. ;right The elements in the vector dataset are converted using prefix doubling. The message queue is divided into multiple batches; at the same time, memory is pre-allocated for the message queue as a buffer for inter-process message exchange, reducing the time consumption caused by frequent memory allocation and release during subsequent message passing.

[0021] Furthermore, the implementation process of step (3) is as follows:

[0022] If there are unprocessed batches, select one of them. Insert as a vertex into the graph index If all batches have been processed, proceed to (10); unprocessed batches... The insertion task is evenly distributed among all processes. Each process Will be allocated to this batch A portion of the vectors Processing is performed; subsequently, each process Multiple threads will be started internally to... The set is further divided into multiple subsets, with each thread processing one subset; for each vector in the subset... The method of constructing vectors using an approximate nearest neighbor graph is used. Calculate its index in the nearest neighbor graph. Neighbor set .

[0023] Furthermore, the neighbor set The construction process employs a greedy search strategy, within the current graph. China Find the largest scale candidate neighbor set Then the candidate set Using existing methods for pruning, retain the most Find suitable neighbors to generate final neighbor set and generate tuples .

[0024] Further, the implementation process of step (4) is as follows: each process maintains an adjacency list length array AdjLen and an offset array Offset, the lengths of which are determined by the number of vectors processed by the process; each process uses multiple threads to process each tuple in parallel. This creates a local message buffer for the tuple and writes the length of the local message buffer to the corresponding position in the adjacency list length array.

[0025] Furthermore, the implementation process of step (5) is as follows:

[0026] Each process performs multi-threaded parallel prefix sum calculation on the adjacency list length array AdjLen and writes the prefix sum calculation result to the offset array Offset; each process uses multiple threads to process each local message buffer in parallel, sets the corresponding adjacency list length and vertex number fields in the process's send buffer according to the local message buffer, and copies the data to the process's send buffer;

[0027] The starting position of the sending buffer is calculated based on the offset values ​​stored in the offset array. The length of the copied data is the length of the neighbor set stored in the local message buffer. Since the offsets do not overlap, the copying process of different local message buffers does not require additional locking to protect the sending message buffer of the process. The sending buffer obtained after all threads of each process have completed the copying is the message content that the process needs to broadcast in this round of iteration.

[0028] Furthermore, the implementation process of step (6) is as follows:

[0029] All processes will send data to their own send buffer. Treat it as a message, and share your message through the network. The message is broadcast to all other processes, including itself; each process increments its counter Superstep by 1. At the start of the next superstep, each process has received messages from all other processes, and the collection of all messages constitutes the global message. .

[0030] Furthermore, the implementation process of step (7) is as follows:

[0031] Each process receives global messages The code utilizes multiple threads to deserialize all tuples in parallel. Each tuple in Update the graph index structure of the current process. ,Will Mid-vector The corresponding neighbor set is updated to ,Right now .

[0032] Furthermore, the implementation process of step (8) is as follows:

[0033] Updating the graph index During the process of neighbor set, the graph index will be used. The vector whose neighbor information changes is defined as the affected vector. The set of all affected vectors is calculated. ;Will Decompose into multiple disjoint subsets, and assign different subsets to different processes for processing; each process Utilize multiple threads to process the affected vector subsets they are responsible for in parallel; for each affected vector... If the graph index Mid-vector Neighbor set Size exceeds parameters Then, by utilizing the nearest neighbor graph index construction method of existing nearest neighbor search algorithms, the neighbor set is... Prune the branches, retaining no more than Find one neighbor; update the graph index in this process based on the pruned neighbor set. Mid-vector The set of neighbors.

[0034] An electronic device according to the present invention includes a memory and a processor, wherein:

[0035] Memory is used to store computer programs that can run on a processor;

[0036] A processor, configured to, while running the computer program, perform the steps of the method for constructing a distributed approximate nearest neighbor search graph index as described above.

[0037] Beneficial Effects: Compared with existing technologies, the beneficial effects of this invention are as follows: This invention extends the single-machine multi-threaded graph structure index construction process to a distributed environment, adopting a task parallel approach. Each node process maintains a complete copy of the graph index, and the adjacency list is broadcast using a broadcast-based BSP framework, enabling inter-process communication and parallel construction. This invention employs a variant of the BSP framework structure that supports shared objects and local message broadcasting. This variant simplifies message transmission between processes during distributed index construction and improves network communication efficiency. This invention solves the locking overhead problem of the write buffer caused by the variable length of the adjacency list when writing to the adjacency list in parallel with multiple threads. By calculating the adjacency list offset using prefix sums, threads can write and read from the adjacency list in parallel without conflicts during parallel writes. Therefore, compared with existing single-machine multi-threaded parallel construction methods, the multi-machine parallel construction proposed in this invention can significantly shorten the index construction time. Compared with existing distributed ANNS graph structure index construction methods, this invention can provide higher query accuracy during the query phase. Attached Figure Description

[0038] Figure 1 This is a flowchart of the method for constructing a distributed approximate nearest neighbor search graph index proposed in this invention;

[0039] Figure 2 This is the process of organizing the parallel send buffer for writing without conflict, as used in this invention.

[0040] Figure 3 This is a schematic diagram of the broadcast-based message communication process used in this invention;

[0041] Figure 4 This is a schematic diagram of an example dataset used in this invention;

[0042] Figure 5 This is an example of the approximate nearest neighbor graph index structure constructed in this invention;

[0043] Figure 6 The figure shows the experimental results comparing the time taken to build graph indexes on different datasets with the existing single-machine multi-threaded graph index building algorithm ParlayDiskANN.

[0044] Figure 7 This is a graph showing the variation in graph index construction time under different numbers of computing nodes according to the present invention;

[0045] Figure 8 The figure shows the experimental results comparing the query throughput performance of the graph index generated by this invention with that of the existing distributed graph index construction algorithm Pyramid. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to the accompanying drawings.

[0047] like Figure 1 As shown, this invention proposes a method for constructing a distributed approximate nearest neighbor search graph index. Based on the single-machine multi-threaded construction method ParlayDiskANN, it is improved and extended to multiple computing nodes, employing distributed parallelization to accelerate the construction process of the approximate nearest neighbor search graph. Specifically, it includes the following steps:

[0048] Step 1: Initialize each process, divide the read-only vector dataset into batches, allocate message buffers, and set the iteration counter Superstep=0.

[0049] At the start of the algorithm, the set of all processes participating in the computation. They will create and initialize an identical shared object in their respective memory locations in parallel. The shared object It contains all the global information needed for algorithm execution, mainly including an empty nearest neighbor graph index. (At this time, the vertex set) and edge set All empty), a complete vector dataset that remains read-only throughout the subsequent process. Superstep, a counter used to record the number of communication iterations in the program, is initialized to 0, along with the upper limit parameter for the number of neighbors of each vector in the nearest neighbor graph provided by the user. In addition, regarding The elements in the vector dataset are converted using prefix doubling. The message queue is divided into multiple batches. Simultaneously, memory is pre-allocated to the message queue as a buffer for inter-process message exchange, reducing the time consumption caused by frequent memory allocation and deallocation during subsequent message passing.

[0050] process collection There are three processes, and the dataset There are 7 three-dimensional vectors. A parameter limits the maximum number of neighbors for each vector in the nearest neighbor graph. The seven 3D vectors in dataset D are feature vectors obtained after feature extraction from an image dataset containing seven images, such as... Figure 4 As shown.

[0051] Based on the prefix doubling batch processing strategy, the dataset It was divided into 3 batches, and each batch contained vectors. , , Each process sends its own message buffer in advance. Allocate sufficient memory space.

[0052] Step 2: If the value of the counter Superstep is an even number, proceed to step 3; if the value of the counter Superstep is an odd number, proceed to step 8.

[0053] Step 3: If there are unprocessed batches, retrieve the unprocessed dataset batches, compute the adjacency list for each point in parallel, and generate tuples. If there are no unprocessed batches, proceed to step 10.

[0054] If there are unprocessed batches, select one of them. Insert as a vertex into the graph index If all batches have been processed, proceed to step 10. This batch... The insertion task is evenly distributed among all processes. Each process Will be allocated to this batch A portion of the vectors Processing is then performed. Subsequently, each process... Multiple threads will be started internally to... The set is further divided into multiple subsets, with each thread processing one subset. For each vector in the subset... By using an approximate nearest neighbor graph construction method (such as the existing DiskANN algorithm), vectors are... Calculate its index in the nearest neighbor graph. Neighbor set The construction process employs a greedy search strategy, within the current graph. China Find the largest scale candidate neighbor set Then the candidate set Using existing methods for pruning, retain the most Find suitable neighbors to generate final neighbor set and generate tuples .

[0055] If Superstep = 0, insert the first batch. ,process Responsible subset ,process and The responsible subset is an empty set. (Graph index at this point) The set of neighbors obtained by each thread through searching and pruning is empty. ,process The generated tuple is .

[0056] If Superstep = 2, insert the second batch. process Responsible subset ,process Responsible subset ,process The responsible subset is an empty set. (Graph index at this point) Only in China The neighbor set obtained by each thread through searching and pruning is: .process The generated tuple is ,process The generated tuple is .

[0057] If Superstep = 4, insert the third batch. process Responsible subset ,process Responsible subset ,process Responsible subset At this time, the image contains... The three vertices, and the neighbor set obtained by each thread through search and pruning are as follows: .process The generated tuple is ,process The generated tuple is ,process The generated tuple is .

[0058] Step 4: The adjacency list length information is written to the adjacency list length array in parallel within the process.

[0059] Each process maintains an adjacency list length array `AdjLen` and an offset array `Offset`. The lengths of these two arrays are determined by the number of vectors the process is responsible for processing. Each process uses multiple threads to process each tuple in parallel. This creates a local message buffer for the tuple and writes the length of the local message buffer to the corresponding position in the adjacency list length array.

[0060] Taking the case where Superstep=4 as an example, the process Responsible for processing vectors ,process Responsible for processing vectors ,process Responsible for processing vectors .process The adjacency list length array AdjLen and the offset array Offset both have a length of 2. (Process) and process The adjacency list length array AdjLen and the offset array Offset both have a length of 1. (Process) The local message buffer generated by computation thread 1 is Write the buffer length of 2 into the first position of the array AdjLen; the local message buffer generated by process calculation thread 2 is... The buffer length of 2 is written to the second position of the array AdjLen; the final value of the AdjLen array is [2, 2]. Process The generated local message buffer includes .process The generated local message buffer includes .

[0061] Step 5: Calculate the offset using parallel prefix sums, and copy the information to the send message buffer in parallel within the process.

[0062] Each process performs multi-threaded parallel prefix sum calculation on the adjacency list length array `AdjLen` and writes the result to the offset array `Offset`. Each process uses multiple threads to process each local message buffer in parallel. Based on the local message buffer, it sets the corresponding adjacency list length and vertex number fields in its send buffer and copies the data to that process's send buffer. During the copying process, the starting position of the send buffer is calculated based on the offset values ​​stored in the offset array. The length of the copied data is the length of the neighbor set stored in the local message buffer. Since the offsets do not overlap, the copying process between different local message buffers does not require additional locking to protect the process's send buffer. The send buffer obtained after all threads in each process have completed the copying process is the message content that the process needs to broadcast in this iteration.

[0063] like Figure 2 As shown, the process with Superstep=4 For example, the adjacency list length array AdjLen of this process is [2,2]. After multi-threaded parallel calculation of the prefix sum, the calculation result is written to the offset array Offset, which contains Offset[0,2,4]. Two local message buffers are processed in parallel using two threads. and Each thread sets the adjacency list length and vertex number field in the send buffer of the process according to the local message buffer. The adjacency list is stored in the send buffer starting from the 12th element. The two threads copy the neighbor sets stored in the two local buffers to the regions in the send buffer starting from the 12th and 14th elements respectively, according to the offsets 0 and 2 stored in the offset array Offset.

[0064] Step 6: As Figure 3 As shown, each process broadcasts its message buffer, and the iteration counter Superstep+1.

[0065] All processes will send data to their own send buffer. Treat it as a message, and share your message through the network. It broadcasts to all other processes, including itself. Figure 3 Taking the calculation process where the counter Superstep is 0 (i.e., in superstep 0) as an example, the message sending buffer of process 0 at this time is... Process 1 is Process 2 is Each process broadcasts its message buffer to all other processes. The broadcast operation is blocking; subsequent computations will only proceed after all processes have completed their broadcasts. Each process increments its counter, Superstep, by 1. At the start of the next superstep (superstep 1 in this example), each process has received messages from all other processes. Figure 3 Process 0, process 1, and process 2 all received it. , , Three messages. The collection of all messages constitutes the global message. .

[0066] Taking the case where Superstep=4 as an example, the process Send the packaged message , Send the packaged message , Send the packaged message Broadcast communication is implemented using the MPI Alltoallv function. After broadcast communication, , , Each received three messages, which were then merged to form the global message. The value of Superstep becomes 5 after incrementing.

[0067] Step 7: Each process deserializes tuples from the received message buffer and updates the in-process graph index.

[0068] Each process receives global messages The code utilizes multiple threads to deserialize all tuples in parallel. Each tuple in Update the graph index structure of the current process. ,Will Mid-vector The corresponding neighbor set is updated to ,Right now Because all processes received global messages. The graph indexes maintained by all processes are completely identical, and the update logic executed is deterministic. Therefore, after this step, the graph indexes maintained by all processes are... Maintain a completely consistent state.

[0069] Each process sequentially parses messages from other processes in the global message list. Where p is the process ID. Based on the message... Parallel prefix sum calculation is performed on the adjacency list length field, such as... Figure 2 As shown, assuming the adjacency list length field is [2,2], the prefix sum is calculated as [0,2,4]. Based on the prefix sum, the offset information of the neighbor set in each tuple in the message is obtained in the message buffer, for example, the offset is [0,2]. Based on the offset, the corresponding area in the message buffer is read to obtain the value of each tuple. The corresponding neighbor set, and obtain the tuple based on the vertex ID field in the message. For the case where Superstep=4, the tuples parsed from the global message include... ,renew

[0070] Post-process graph index for:

[0071] After the graph index is updated, the Superstep counter is incremented by one, becoming... Continue to step 7. If the result of adding one to Superstep is even, return to step 2 to insert the next batch of points.

[0072] Step 8: Identify the points affected during the graph index update process, and re-prune and update in parallel.

[0073] If the value of the counter Superstep is odd, execute this step; otherwise, skip to step 2. Update the graph index. During the process of neighbor set, the graph index will be used. The vector whose neighbor information changes is defined as the affected vector. The set of all affected vectors is calculated. .Will It is decomposed into multiple disjoint subsets, and the different subsets are assigned to different processes for processing. Each process Multiple threads are used to process the affected subset of vectors they are responsible for in parallel. For each affected vector... If the graph index Mid-vector Neighbor set Size exceeds parameters Then, by utilizing the nearest neighbor graph index construction method of existing nearest neighbor search algorithms, the neighbor set is... Prune the branches, retaining no more than Each neighbor. Update the graph index in this process based on the pruned neighbor set. Mid-vector The set of neighbors.

[0074] Taking the case of Superstep = 5 as an example, the set of affected vectors ; set Divide into two disjoint sets and These tasks are handled by process 0 and process 1, respectively. For process... Two threads are used to process the two affected vectors respectively. The parameters of the affected vectors are... For vectors The pruned neighbor set can be constructed using the nearest neighbor graph construction method of the existing ParlayANN algorithm. For vectors Based on the existing ParlayANN algorithm's nearest neighbor graph construction method, the pruned neighbor set... Process Update the graph index based on the pruned neighbor set. and The set of neighbors. Process The processing procedure is similar.

[0075] Step 9: Generate tuples for the adjacency list of the updated nodes.

[0076] For each process, the vector updated in step 8 is... Its updated set of neighbors Generating tuples For all tuples generated in this process, execute steps 4 to 7. After execution is complete, jump to step 2 for execution.

[0077] Superstep=5 process For example, in the case of process The generated tuples include After executing steps 3 through 6, the Superstep counter becomes 6, and execution jumps to step 2 to continue.

[0078] Step 10: One of the processes prints the relevant results and outputs the graph index of the process. .

[0079] By process Print the relevant results and index the graph in the process. Output to a file.

[0080] The present invention also provides an electronic device, including a memory and a processor, wherein: the memory is used to store a computer program that can run on the processor; and the processor is used to execute, when running the computer program, the steps of the method for constructing a distributed approximate nearest neighbor search graph index as described above.

[0081] Prepare a vector dataset. The vectors in the vector dataset are usually feature vectors extracted from real-world objects such as images and text using a machine learning model. Feature vectors characterize the similarity between real-world objects. If two images are approximately similar, the higher the similarity of their extracted feature vectors, the better. Figure 4 The presentation shows feature vectors extracted from different images in a dataset containing 7 images. Each feature vector consists of 3 dimensions.

[0082] The goal of approximate nearest neighbor search is to find the K objects in a dataset that are most similar to a given query object. Figure 4 Taking the image dataset shown as an example, for a user-given query image Q, the same feature vector extraction method is first used to extract a feature vector Vq for the query image. Then, the K image objects whose feature vectors are closest to Vq are found from the dataset as the results of an approximate nearest neighbor search. For large datasets, if it is necessary to traverse the feature vectors of all objects in the dataset and calculate the distance during the query process, the computational cost will be too high, making the retrieval time unacceptable.

[0083] To improve the efficiency of near-nearest neighbor search and reduce the computational load during retrieval, graph-based near-nearest neighbor search indexes have been proposed. By pre-constructing a graph-based index for the dataset, the number of dataset feature vectors requiring distance calculations is significantly reduced during retrieval, thus shortening the retrieval time. However, for large datasets containing numerous objects, constructing a graph-based index can be time-consuming. This invention proposes a distributed parallel computing method for graph-based indexes, which can construct high-quality graph-based indexes for datasets in a shorter time.

[0084] Through multiple rounds of iterative calculations, this step will be executed when the Superstep counter reaches 6. At this point, the process... The graph index structure G stored in it is as follows Figure 5 As shown, each vertex in the graph represents a vector in the dataset, corresponding to the feature vector of an image in the image dataset. If there is an edge between two vertices in the graph, it means that the feature vectors of the two vertices are relatively close, and they can be used as potential expansion objects in the process of approximate nearest neighbor search.

[0085] This invention achieves multi-node, multi-threaded parallel index construction of an approximate nearest-neighbor graph index structure through distributed task parallelism. A series of performance tests were conducted on five real-world datasets to compare the computational performance of the proposed distributed parallel index construction method with existing distributed graph index construction and single-machine multi-threaded parallel index construction methods. Figure 6 The paper compares the construction speed of the method of the present invention with that of the existing single-machine multi-threaded parallel graph index construction method ParlayDiskANN in an eight-node experimental environment. The method of the present invention achieves a speed improvement of 6.00 to 7.08 times, and has a good acceleration effect on datasets of different sizes. Figure 7 The invention demonstrates the speedup effect of the method using different numbers of computing nodes when dealing with different datasets. The ratio of the multi-node index construction time to the single-node index construction time (speedup ratio) generally increases linearly with the increase of the number of nodes under different datasets. Figure 8 The performance of the proposed method in this invention was compared with that of the existing distributed graph index construction method Pyramid in the approximate query phase. At the same recall level, the index constructed by the proposed method achieves a significantly higher query throughput per second compared to the existing Pyramid method. The results show that the proposed method effectively shortens the graph structure index construction time through a multi-node, multi-threaded joint parallel computation approach.

[0086] The graph index structure generated by the method of the present invention has the same quality as the existing single-machine multi-threaded parallel construction method. Furthermore, the method of the present invention requires only one layer of search in the search process, each computing node has a complete copy of the graph index, and the query process does not require inter-node communication. Figure 8 The proposed method was compared with the graph index structure constructed by the existing distributed graph index construction method Pyramid on various datasets, showing the change in query throughput (i.e., the number of queries completed per second) with query recall during the query phase. Experimental results consistently demonstrate that, with consistent query throughput, the proposed method provides a higher query recall compared to the graph index structure constructed by the Pyramid method.

[0087] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a distributed approximate nearest neighbor graph index, characterized in that, The method comprises the following steps: (1) each process initializes, divides batches for the read-only vector dataset, allocates a message buffer, and sets an iteration counter Superstep=0; (2) if the value of the counter Superstep is an even number, step (3) is performed; if the value of the counter Superstep is an odd number, step (8) is performed; (3) there is an unprocessed batch, the unprocessed dataset batch is taken out, and the adjacency list of each point is calculated in parallel to generate a tuple; if there is no unprocessed batch, step (10) is performed; (4) the length information of the adjacency list is written into an adjacency list length array in parallel within the process; (5) the offset is calculated by using a parallel prefix sum, and the information is copied to the sending message buffer in parallel within the process; (6) the sending message buffer of each process is broadcasted, and the iteration counter Superstep+1 is updated; (7) each process deserializes the tuple from the received message buffer, and updates the graph index within the process; (8) the affected points in the process of updating the graph index are determined, and the parallel pruning and updating are performed again; (9) Generating tuples for the adjacency list of the updated node: for each process, generating tuples for the updated vector of step (8) and its updated neighbor set Generating tuples ; for all tuples generated for the current process, performing steps (4) to (7), and after performing steps (4) to (7), jumping to step (2) for execution; (10) Print the relevant results by one of the processes and put the output in the figure index of the process .

2. The method of claim 1, wherein, The step (1) is implemented as follows: a set of processes participating in the computation a completely identical shared object is created and initialized in each memory in parallel ; the shared object including all global information needed for execution, including an empty index of the nearest neighbor graph at this point the vertex set and the edge set are empty; A complete vector dataset that remains read-only throughout the subsequent process. A counter, Superstep, used to record the number of communication iterations in the program, initialized to 0; and a user-defined upper limit parameter for the number of neighbors for each vector in the nearest neighbor graph. ;right The elements in the vector dataset are converted using prefix doubling. The message queue is divided into multiple batches; at the same time, memory is pre-allocated for the message queue as a buffer for inter-process message exchange, reducing the time consumption caused by frequent memory allocation and release during subsequent message passing.

3. The method of claim 1, wherein, The step (3) is implemented as follows: If there are unprocessed batches, select one of them. Insert as a vertex into the graph index If all batches have been processed, proceed to (10); unprocessed batches... The insertion task is evenly distributed among all processes. Each process Will be allocated to this batch A portion of the vectors Processing is performed; subsequently, each process Multiple threads will be started internally to... The set is further divided into multiple subsets, with each thread processing one subset; for each vector in the subset... The method of constructing vectors using an approximate nearest neighbor graph is used. Calculate its index in the nearest neighbor graph. Neighbor set .

4. The method of claim 3, wherein, The neighbor set The construction process employs a greedy search strategy, within the current graph. China Find the largest scale candidate neighbor set Then the candidate set Using existing methods for pruning, retain the most Find suitable neighbors to generate final neighbor set and generate tuples .

5. The method of claim 1, wherein, The step (4) is implemented as follows: each process maintains an adjacency list length array AdjLen and an offset array Offset, the length of the two arrays is determined by the number of vectors handled by the process; each process uses multiple threads to handle each tuple in parallel , creates a local message buffer for the tuple, and writes the length of the local message buffer to the corresponding position of the adjacency list length array.

6. The method of claim 1, wherein, The step (5) is implemented as follows: Each process performs a multi-thread parallel prefix sum calculation on the adjacency list length array AdjLen, and writes the prefix sum calculation result into the offset array Offset; each process uses multiple threads to process each local message buffer in parallel, sets the adjacency list length and vertex number field in the sending buffer of the process according to the local message buffer, and copies the data into the sending buffer of the process; The starting position of the sending buffer is calculated according to the offset value stored in the offset array, and the data length is the length of the neighbor set stored in the local message buffer; the copy processes of different local message buffers do not need to be additionally locked to protect the sending message buffer of the process because the offsets are not overlapped; after all the threads of each process complete the copying, the sending buffer obtained is the message content that needs to be broadcasted by the process in the current iteration.

7. The method of claim 1, wherein, The step (6) is implemented as follows: All processes will send data to their own send buffer. Treat it as a message, and share your message through the network. The message is broadcast to all other processes, including itself; each process increments its counter Superstep by 1. At the start of the next superstep, each process has received messages from all other processes, and the collection of all messages constitutes the global message. .

8. The method of claim 1, wherein, The step (7) is implemented as follows: Each process receives global messages The code utilizes multiple threads to deserialize all tuples in parallel. Each tuple in Update the graph index structure of the current process. ,Will Mid-vector The corresponding neighbor set is updated to ,Right now .

9. The method of claim 1, wherein, The step (8) is implemented as follows: Updating the graph index During the process of neighbor set, the graph index will be used. The vector whose neighbor information changes is defined as the affected vector. The set of all affected vectors is calculated. ;Will Decompose into multiple disjoint subsets, and assign different subsets to different processes for processing; each process Utilize multiple threads to process the affected vector subsets they are responsible for in parallel; for each affected vector... If the graph index Mid-vector Neighbor set Size exceeds parameters Then, by utilizing the nearest neighbor graph index construction method of existing nearest neighbor search algorithms, the neighbor set is... Prune the branches, retaining no more than Find one neighbor; update the graph index in this process based on the pruned neighbor set. Mid-vector The set of neighbors.

10. An electronic device, comprising: The method comprises a memory and a processor, wherein: The memory is used to store a computer program capable of running on the processor; The processor is used to execute the steps of the construction method of the distributed approximate near neighbor search graph index according to any one of claims 1 to 9 when the computer program is run.

Images