A Distributed Graph Indexed Nearest Neighbor Search Method for High-Dimensional Spaces

Abstract

This invention discloses a distributed graph index nearest neighbor search method for high-dimensional spaces. First, it designs a boundary-aware balanced partitioning strategy, quantifying the boundary characteristics of nodes through connection ratio metrics. Then, it constructs a hierarchical hybrid index architecture, vertically integrating a sparse global navigation layer and a fine-grained local precision layer. Simultaneously, it employs an M-Tree structure to manage highly important boundary nodes, leveraging the relative insensitivity of the tree structure to the dimensional curse in boundary regions to significantly improve cross-partition retrieval efficiency. Finally, it proposes a distributed multi-starting-point parallel search framework, dynamically generating complementary entry points based on query features and graph topology to achieve parallel exploration of multiple regions in the vector space. Irrelevant partitions are first filtered through the navigation layer, and then precise similarity calculations are performed on the selected partitions. This invention can significantly improve the throughput of distributed high-dimensional vector search while maintaining high recall, providing an efficient and reliable solution for large-scale high-dimensional vector retrieval.

Description

Technical Field

[0001] This invention belongs to the field of high-dimensional data retrieval and distributed computing technology, specifically relating to a distributed graph index nearest neighbor search method for high-dimensional spaces. Background Technology

[0002] With the rapid development of large-scale database systems and large language models, high-dimensional vector data is experiencing explosive growth in fields such as computer vision, bioinformatics, and recommender systems, creating an urgent need for efficient and scalable nearest neighbor (NNS) search solutions.

[0003] Nearest neighbor search (NNS) aims to identify the most similar data vector to a given query vector from a dataset and is a core operation in many applications. To support efficient NNS in high-dimensional spaces, researchers have developed Approximate Nearest Neighbor Search (ANNS) methods, which are mainly divided into four categories: tree-based methods, hash-based methods, quantization-based methods, and graph-based methods. Tree-based methods, such as KD-Tree (K-dimensional tree) and R*-Tree (R*-tree), organize vectors through hierarchical spatial partitioning, achieving efficient search in low-dimensional spaces. KD-Tree recursively partitions the space along the coordinate axes, while R*-Tree uses minimum bounding rectangles to combine neighboring objects. However, these structures are severely affected by the "curse of dimensionality," and search efficiency decreases exponentially with increasing dimensionality. In distributed environments, tree structures also face additional challenges from partition boundaries and global traversal. Hash-based methods, such as Locality Sensitive Hashing (LSH), project high-dimensional vectors into a low-dimensional space, giving similar items a high collision probability. Although these methods provide theoretical guarantees and sublinear search time, they perform poorly in terms of precision-recall tradeoffs and require multiple hash tables to achieve competitive accuracy. Quantization-based methods such as product quantization (PQ) reduce memory requirements and significantly improve distance calculation by quantizing the decomposed vectors. However, the quantization errors introduced by these methods accumulate in high dimensions, leading to a decline in search quality. Distributed implementations also face the problem of communication overhead caused by the cross-partitioning of quantization units.

[0004] Graph-based methods, which construct proximity graphs utilizing the nearest neighbor relationships between objects, are the optimal approach for high-dimensional NNS (Nearest Neighbor Search) and have become the mainstream solution due to their superior performance in query efficiency and accuracy. Hierarchical Navigable Small World (HNSW) and Navigation Spread Graph (NSG) methods have demonstrated excellent recall and query efficiency. These techniques create navigable search networks based on infrastructures such as k-nearest neighbor graphs, Delaunay graphs, and relative neighborhood graphs. Despite the superior performance of graph-based methods, two key challenges remain: first, the high construction cost and massive memory consumption make them unsuitable for single-machine environments as data scales; second, in distributed environments, graph partitioning can disrupt the proximity relationships between nodes, leading to extensive cross-node communication and severely impacting search efficiency and accuracy. For example, the HNSW method can consume up to 100GB of main memory storage for datasets of billions. While secondary storage-based methods such as DiskANN (Disk Approximate Nearest Neighbor Search) partially address the high storage cost problem by storing data and indexes on disk instead of main memory, they still face challenges such as long construction times and single-machine storage limitations.

[0005] Existing distributed graph indexing methods attempt to address these issues. Pyramid employs a meta-HNSW structure for dataset partitioning, constructing subgraphs on similar data clusters and reducing query computation by routing to relevant partitions. However, this method lacks sensitivity to boundary nodes, and its simple single-layer navigation structure performs poorly under complex data distributions. D-HNSW (Distributed Hierarchical Navigation Small World Graph) extends the HNSW algorithm to distributed environments, using sampled graphs for partitioning while building local indexes on worker nodes. While maintaining search quality, its partitioning strategy ignores computational balance, leading to workload skew and low resource utilization. Furthermore, its reliance on a single-entry search model limits scalability under high concurrency. Parallel and distributed graph frameworks built on Spark and MapReduce, such as Sheep and CAGRA (a GPU-based high-performance parallel graph approximate nearest neighbor search algorithm), parallelize or distribute graph construction through partitioning and edge generation. However, these methods face fundamental challenges in minimizing communication overhead while maintaining global connectivity, primarily due to the conflict between the inherent connectivity of proximity graphs and distributed settings. Hybrid indexes combine the complementary advantages of different methods to overcome individual limitations. Graph-tree hybrid methods such as ELPIS (a domain model-based system building framework) and GTI (a tree induction method based on information gain) use innovative methods to partition and organize the graph before construction, significantly reducing memory overhead and construction costs. However, the application of these methods in distributed environments faces challenges such as unbalanced load, excessive cross-partition communication, and difficulty in maintaining a consistent global navigation structure.

[0006] The limitations of the existing methods highlighted the urgent need to develop techniques specifically designed for the distributed nature of large-scale vector retrieval. Therefore, there is a pressing need for a distributed graph indexing method that can simultaneously achieve boundary-aware balanced partitioning, efficient hierarchical indexing, and high-throughput query processing. Summary of the Invention

[0007] To address the technical problems of existing distributed graph indexing methods in large-scale high-dimensional vector retrieval, such as high memory consumption, uneven partitioning, high communication overhead, and low search efficiency, this invention provides a distributed graph indexing nearest neighbor search method for high-dimensional spaces. Through the collaborative design of boundary-aware partitioning, hierarchical indexing, and distributed parallel search, it achieves efficient and reliable large-scale vector retrieval.

[0008] A distributed graph index nearest neighbor search method for high-dimensional spaces includes the following steps:

[0009] (1) Obtain a high-dimensional dataset and construct a distributed computing environment consisting of a master node and multiple worker nodes;

[0010] (2) Perform boundary-aware balanced partitioning on the high-dimensional dataset;

[0011] (3) Construct a hierarchical index structure based on the partitioning results;

[0012] (4) Based on the hierarchical index structure, perform distributed multi-starting point parallel search.

[0013] Furthermore, the high-dimensional dataset in step (1) contains n There are n vectors, each with dimensions of 1. d , d The value range is from 128 to 960. n The scale ranges from millions to billions; in the distributed computing environment, the master node is responsible for global coordination, partition management, and query routing, while the worker nodes are responsible for local index building and query execution.

[0014] Furthermore, the specific implementation of step (2) is as follows:

[0015] S21: Input a high-dimensional dataset, partition it using the KaFFPa (Karlsruhe Fast Flow Partitioning) algorithm, and obtain... m An initial partition m The number of worker nodes;

[0016] S22: Construct a k-nearest neighbor graph based on a high-dimensional dataset to determine the connection relationships between vectors;

[0017] S23: Calculate the connection ratio of each vector in the high-dimensional dataset = number of cross-partition connections / total number of connections;

[0018] S24: Include vectors with a connection ratio higher than a set threshold into the HBN (High Importance Boundary Point Set);

[0019] S25: Remove vectors belonging to HBN from the initial partition and output the final partition result.

[0020] Furthermore, the specific implementation of step (3) is as follows:

[0021] S31: Based on a high-dimensional dataset, a global navigation layer is built on the master node. This layer serves as a global routing skeleton and is encapsulated as a Spark broadcast variable and distributed to all worker nodes.

[0022] S32: Construct an M-Tree index structure based on HBN, whose internal nodes are used to maintain routing objects and coverage radii; store the M-Tree index structure on the master node and broadcast it to each worker node for distance pruning in cross-partition query operations;

[0023] S33: Construct a local HNSW graph structure for the corresponding partition at each working node, assign random levels to the vectors in the partition and insert them layer by layer into the local HNSW graph structure to form a local precise layer.

[0024] Furthermore, the specific implementation of constructing the global navigation layer in step S31 is as follows: firstly, a certain proportion is randomly sampled from each partition. or The vectors form a landmark set, and an initialization is performed. L max The HNSW graph structure of the layer; for any landmark vector in the landmark set. v Assign it a hierarchy l v :

[0025]

[0026] Where: random is a uniformly random number between 0 and 1. , For the set number of floors, ;

[0027] Then, according to the assigned level, the landmark vectors are inserted into the HNSW graph structure layer by layer, and multiple connections are established in each layer to form a global navigation layer of the hierarchical navigation structure.

[0028] Furthermore, the specific implementation of step (4) is as follows:

[0029] S41: Select the appropriate entry point from the global navigation layer based on the diversity scoring function;

[0030] S42: Search for the corresponding boundary points from the M-Tree index structure based on the dynamic radius;

[0031] S43: Take the intersection of the partition to which the entry point belongs and the partition to which the boundary point belongs as the relevant partition;

[0032] S44: For the query vector, perform a local HNSW search in parallel on the relevant partitions, aggregate the search results, and then perform a k-nearest neighbor search to output the final query result.

[0033] Further, in step S41, the two elements closest to the query vector are first found in the global navigation layer. k e A set Z is formed from several landmark vectors. The landmark vector in set Z that is closest to the query vector is taken as the first entry point and added to set S. For any landmark vector in set Z... v The diversity score is calculated according to the following formula:

[0034]

[0035] in: Landmark Vector v Diversity score, q Represents the query vector. Represents the query vector q Landmark Vector v distance, u Denotes any entry point in set S. Indicates the entry point u Landmark Vector v distance, l As a diversity factor, k e The set number of entry points;

[0036] The landmark vectors in set Z are sorted according to their diversity scores. The landmark vector with the highest diversity score is added to set S. After each addition, the diversity score is recalculated. This process is repeated to select landmark vectors from set Z and add them to set S until the size of set S reaches a certain threshold. k e indivual.

[0037] Furthermore, in step S42, the relationship between each entry point and the query vector in set S is first calculated. q The distance, taking the average value as m The standard deviation is s Then, the dynamic radius is calculated and determined. ,in c The filtering threshold is used; then, based on the dynamic radius... rSearch the M-Tree index structure and retrieve the values ​​in HBN that match the query vector. q Distance less than r The vector is used as the boundary point.

[0038] A computer device includes a memory and a processor, wherein the memory stores a computer program and the processor executes the computer program to implement the above-described distributed graph index nearest neighbor search method for high-dimensional spaces.

[0039] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described distributed graph index nearest neighbor search method for high-dimensional spaces.

[0040] This invention innovatively combines the retrieval efficiency of graph structures with the boundary management capabilities of tree structures, specifically including the following three key technological innovations:

[0041] 1. Boundary-aware balanced partitioning strategy; This invention quantifies the "boundary characteristics" of each node by measuring the connection ratio, and dynamically identifies high-importance boundary points by combining a multi-dimensional importance evaluation function. This strategy solves the inherent trade-off between balancing computing resources and minimizing cross-node communication in a distributed environment.

[0042] 2. Hierarchical distributed graph-tree hybrid index architecture; This invention vertically integrates a sparse global navigation layer and a fine-grained local precision layer. The global navigation layer contains only 1-5% of the nodes as a sparse skeleton for coarse-grained routing, while the local precision layer constructs specific graphs in each partition to achieve fine-grained retrieval. At the same time, it utilizes the relative insensitivity of tree index to dimensional curse in specific boundary areas to significantly improve cross-partition retrieval efficiency.

[0043] 3. Distributed multi-starting point parallel search framework; This invention dynamically generates complementary entry points based on query features and graph topology to achieve parallel exploration of multiple regions in the vector space, and balances retrieval accuracy and throughput through a two-stage progressive search strategy.

[0044] Therefore, the present invention has the following beneficial technical effects:

[0045] 1. This invention achieves dual optimization of computational load balancing and communication overhead minimization by accurately identifying and managing highly important boundary points through a boundary-aware balanced partitioning strategy.

[0046] 2. This invention adopts a hierarchical hybrid index architecture, which fully utilizes the efficient retrieval capabilities of graph structures and the boundary management advantages of tree structures, significantly reducing memory usage.

[0047] 3. This invention designs a distributed, multi-starting-point parallel search framework, which significantly improves query efficiency while ensuring high recall rate through dynamic entry point selection and a two-stage progressive strategy.

[0048] 4. This invention provides a complete cost model and optimization strategy, which facilitates the tuning of system parameters according to actual application scenarios and has good scalability and practicality. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the overall framework of the distributed graph index in an embodiment of the present invention.

[0050] Figure 2 This is a schematic diagram of the three-layer distributed graph index structure in an embodiment of the present invention.

[0051] Figure 3 This is a schematic diagram of the boundary-aware equalization partitioning process in an embodiment of the present invention.

[0052] Figure 4 This is a schematic diagram illustrating the construction process of the global navigation layer and the local precision layer in an embodiment of the present invention.

[0053] Figure 5 This is a schematic diagram of the distributed query processing flow in an embodiment of the present invention. Detailed Implementation

[0054] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] This embodiment uses an image retrieval system as a specific application scenario, applying the distributed graph index nearest neighbor search method of the present invention to a large-scale visual feature retrieval task.

[0056] In image retrieval, each image is extracted into a high-dimensional feature vector through a deep neural network. This vector represents the semantic and visual features of the image. If the system contains hundreds of millions to billions of images, it can form a high-dimensional feature database on the scale of millions to billions. At this time, each image feature vector corresponds to the "high-dimensional vector" in the method of this invention, and the entire database constitutes a "high-dimensional dataset". During a query, a user-uploaded image is used to extract a "query vector". The distributed operating environment of the system consists of a master node and multiple worker nodes, corresponding to the distributed storage and computing cluster of the image database.

[0057] In this practical task, the boundary-aware partitioning strategy of this invention can divide massive image feature data across multiple working nodes, while automatically identifying "boundary vectors" with frequent cross-partition connections to minimize cross-node communication volume; the hierarchical hybrid index architecture can build a sparse global navigation layer on the master node to achieve fast routing and coarse-grained search, and then build a local precision layer on each working node to achieve high-precision similarity retrieval; the distributed multi-starting point parallel search framework supports parallel search from multiple candidate entry points, significantly improving query throughput and response speed.

[0058] Through the above design, this invention can accurately find the candidate result most similar to the query image from a feature library of billions in image retrieval tasks, balancing high accuracy and high concurrency performance. Furthermore, the technical solution of this invention can also be extended to other fields requiring high-dimensional vector retrieval, such as social network recommendations (using user interest vectors as the retrieval object), personalized e-commerce recommendations (using product feature vectors as the retrieval object), medical image similarity case search, and biometric recognition, possessing broad industrial applicability and practical application value.

[0059] The overall framework of the distributed graph index in this embodiment is as follows: Figure 1 As shown, this system is applied to large-scale, high-dimensional vector image retrieval scenarios. It includes a data partitioning module, an index building module, a query processing module, and a cost optimization model. The data partitioning module employs a boundary-aware balanced partitioning strategy, the index building module implements a hierarchical index architecture, the query processing module provides a multi-starting-point parallel search framework, and the cost optimization model is responsible for adaptive tuning of system parameters. The specific implementation steps of the distributed graph index nearest neighbor search method in this embodiment are as follows:

[0060] Step S1: System initialization and data acquisition.

[0061] In the specific implementation process, the first step is to obtain a high-dimensional vector dataset. , where each vector v i ∈R d , d For vector dimensions, n For dataset size. In practical applications, d The value range is from 128 to 960. n The scale ranges from millions to billions. For example, in image retrieval systems... d =256 indicates the dimension of image features extracted by deep learning. n =10 9 This represents the feature vectors of one billion images.

[0062] Build a distributed computing environment, including one master node and mEach node consists of several worker nodes. The master node is responsible for global coordination, partition management, and query routing; worker nodes are responsible for local index building and query execution; a distance function is defined. It is used to measure the similarity between vectors and supports multiple measures such as Euclidean distance and cosine similarity.

[0063] Step S2: Boundary-aware equalization partitioning.

[0064] The specific implementation process of this step is as follows: Figure 3 As shown:

[0065] S21: Input the original dataset D The KaFFPa algorithm is used for initial partitioning to obtain... m Initial partitions KaFFPa employs a multi-level graph shrinkage strategy to minimize edge cutting while ensuring that the size difference between each partition does not exceed 3%.

[0066] S22: Construct the k-nearest neighbor graph G( D , E ), where the set of edges Represent each vector and its... k The nearest neighbor connection relationship; in this embodiment k Set it to 20 to use an approximation algorithm to speed up the construction process.

[0067] S23: Calculate each vector v Connection ratio B ( v = (Number of cross-partition connections / Total number of connections) = (Number of cross-partition connections) / (Total number of connections). Iterate through all vectors and count their number of cross-partition connections. and total connection The connection ratio quantifies the frequency with which a vector participates in cross-partition operations, B( v The higher the value, the more likely the vector is to participate in cross-partition queries, resulting in heavier communication overhead.

[0068] S24: Identify high-importance boundary points. Set a threshold. t =0.7, which will satisfy B( v )> t Add the vector to the set Experiments show that HBN typically accounts for 1-5% of the dataset, and the vectors in this dataset incur higher communication costs due to their special role in cross-partition operations.

[0069] S25: Construct an M-Tree index structure to manage boundary points. Each internal node maintains a routing object and coverage radius, utilizing the tree structure's efficient pruning capabilities in the boundary region.

[0070] S26: Finally, remove the boundary points from the original partitions and output the final partitions. ,in .

[0071] Step S3: Construct as follows Figure 2 The hierarchical index structure shown.

[0072] S31: Global Navigation Layer Construction: On the master node, 1-5% of the vectors from the dataset are sampled as landmark vectors to construct a sparse HNSW graph; this layer serves as the global routing skeleton and is distributed to all worker nodes through Spark broadcast variables, transforming the traditional point-to-point communication mode into an efficient broadcast mode.

[0073] S32: Boundary Management Tree Construction: An M-Tree index is constructed on the identified set of highly important boundary points (HBN), stored in the master node, and broadcast to worker nodes for efficient pruning across partition queries. The M-Tree is a metric space index structure that defines spherical regions through hierarchical routing objects and coverage radii, supporting efficient distance-based pruning. Each internal node maintains a routing object and coverage radius, facilitating distance pruning in cross-partition search operations.

[0074] S33: Locally Precise Layer Construction: At each working node N i Above is the local partition D. i Constructing a local HNSW graph G i And identify the entry point set EP i Establish a connection with the global navigation layer.

[0075] The construction process of the global navigation layer is as follows: Figure 4 As shown: From each partition D i proportionally or =0.03 Layered sampling of landmark vectors to form a landmark set L To ensure coverage of the entire vector space while minimizing redundancy, the number of samples... and Initialize a L max =16-layer empty HNSW structure, for each landmark vector v ∈ L Allocation hierarchy ,in From the hierarchy l v Insert layer by layer up to 0, and establish each layer M =16 connections; after optimizing the graph connections, the global navigation graph is broadcast to all worker nodes.

[0076] The construction process of the local precision layer is as follows Figure 4 As shown: On each working node, partition D i Constructing a local HNSW graph G iAssign random levels to each vector and insert them layer by layer; identify partition D i The distance between the center and the landmark is less than i The vector is used as the entry point EP i ,in i A mapping relationship between entry points and corresponding landmarks is established, which is twice the average nearest neighbor distance. These entry points establish key connections between the global navigation layer and the local precision layer. A multi-layer graph structure is constructed for the vectors within the partition according to the HNSW algorithm to ensure local high-precision retrieval capability.

[0077] Step S4: Distributed query processing.

[0078] This implementation method addresses the query vector. q Performing three-phase parallel query processing as follows Figure 5 As shown:

[0079] S41: Entry point selection phase. Locate 2× in the global navigation graph. k e The nearest landmark is selected as the first entry point, and a diversity scoring function is used. Select subsequent entry points, where S is the set of selected entry points. l =0.5; Each worker node filters the entry point based on local relevance to achieve distributed parallel execution.

[0080] S42: Partition filtering stage. Calculate the dynamic search radius. ,in m The average distance, s Standard deviation, c =2.0, perform a range search for relevant boundary points in the boundary tree, and take the intersection of the partition to which the entry point belongs and the partition to which the boundary point belongs as the relevant partition P. r .

[0081] S43: Precise Search Phase. Local HNSW searches are performed in parallel on relevant partitions, collecting candidate results from each partition into a global candidate list; based on candidate quality, it is determined whether boundary verification is needed, the precise distance is calculated, and the final candidate is selected through a priority queue. k The nearest neighbor.

[0082] S44: The result aggregation returns the final result. k Nearest neighbor result set.

[0083] Step S5: Cost optimization.

[0084] The system optimizes query performance through a cost model, and the cost calculation includes global pruning. and local search , communication cost ,in OIndicate the time complexity. P prune This represents the probability that a partition will be pruned. N w Number of worker nodes; Total query time The sum of computational and communication costs is determined by adjusting the number of partitions. m Optimize overall query performance; experiments show m Optimal performance is achieved at 8.

[0085] The above description of the embodiments is provided to enable those skilled in the art to understand and apply the present invention. Those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and any improvements and modifications made to the present invention by those skilled in the art based on the disclosure thereof should be within the scope of protection of the present invention.

Claims

1. A distributed graph index nearest neighbor search method for high-dimensional spaces, characterized in that, Includes the following steps: (1) Obtain a high-dimensional dataset and build a distributed computing environment consisting of a master node and multiple worker nodes; (2) Perform boundary-aware balanced partitioning on the high-dimensional dataset; (3) Construct a hierarchical index structure based on the partitioning results; (4) Based on the hierarchical index structure, perform a distributed multi-starting-point parallel search, the specific implementation of which is as follows: S41: Select the appropriate entry point from the global navigation layer based on the diversity scoring function, specifically: First, find the 2k closest to the query vector in the global navigation layer. e A set Z is formed from several landmark vectors. The landmark vector in set Z that is closest to the query vector is taken as the first entry point and added to set S. For any landmark vector v in set Z, its diversity score is calculated according to the following formula: scroe(v)=δ(q,v) - 1×λ×min u∈S δ(v,u) Where: score(v) is the diversity score of landmark vector v, q represents the query vector, δ(q,v) represents the distance between query vector q and landmark vector v, u represents any entry point in set S, δ(v,u) represents the distance between entry point u and landmark vector v, λ is the diversity factor, and k e The set number of entry points; The landmark vectors in set Z are sorted according to their diversity scores. The landmark vector with the highest diversity score is added to set S. After each addition, the diversity score is recalculated. This process is repeated to select landmark vectors from set Z and add them to set S until the size of set S reaches k. e indivual; S42: Search for corresponding boundary points in the M-Tree index structure based on dynamic radius. Specifically: First, calculate the distance between each entry point in set S and the query vector q, take the average value as μ and the standard deviation as σ, and then calculate and determine the dynamic radius r = μ + γ·σ, where γ is the filtering threshold; then search the M-Tree index structure according to the dynamic radius r, and take the vectors in HBN whose distance from the query vector q is less than r as boundary points. HBN is the set of high importance boundary points. S43: Take the intersection of the partition to which the entry point belongs and the partition to which the boundary point belongs as the relevant partition; S44: For the query vector, perform a local HNSW search in parallel on the relevant partitions, aggregate the search results, and then perform a k-nearest neighbor search to output the final query result.

2. The distributed graph index nearest neighbor search method for high-dimensional space according to claim 1, characterized in that: The high-dimensional dataset in step (1) contains n vectors, each vector having a dimension of d, with d ranging from 128 to 960 dimensions, and n ranging from millions to billions. In the distributed computing environment, the master node is responsible for global coordination, partition management, and query routing, while the worker nodes are responsible for local index building and query execution.

3. The distributed graph index nearest neighbor search method for high-dimensional space according to claim 1, characterized in that, The specific implementation method of step (2) is as follows: S21: Input a high-dimensional dataset, partition it using the KaFFPa algorithm to obtain m initial partitions, where m is the number of worker nodes; S22: Construct a k-nearest neighbor graph based on a high-dimensional dataset to determine the connection relationships between vectors; S23: Calculate the connection ratio of each vector in the high-dimensional dataset = number of cross-partition connections / total number of connections; S24: Include vectors with a connection ratio higher than a set threshold into HBN; S25: Remove vectors belonging to HBN from the initial partition and output the final partition result.

4. The distributed graph index nearest neighbor search method for high-dimensional space according to claim 1, characterized in that, The specific implementation method of step (3) is as follows: S31: Based on a high-dimensional dataset, a global navigation layer is built on the master node. This layer serves as a global routing skeleton and is encapsulated as a Spark broadcast variable and distributed to all worker nodes. S32: Construct an M-Tree index structure based on HBN, whose internal nodes are used to maintain routing objects and coverage radii; store the M-Tree index structure on the master node and broadcast it to each worker node for distance pruning in cross-partition query operations; S33: Construct a local HNSW graph structure for the corresponding partition at each working node, assign random levels to the vectors in the partition and insert them layer by layer into the local HNSW graph structure to form a local precise layer.

5. The distributed graph index nearest neighbor search method for high-dimensional spaces according to claim 4, characterized in that, The specific implementation of constructing the global navigation layer in step S31 is as follows: First, a certain proportion η of vectors are randomly sampled from each partition to form a landmark set, and an L is initialized. max The HNSW graph structure of the layers; for any landmark vector v in the landmark set, assign it a layer l. v : L v min(-log(random)×m L ,L max ) Where: random is a uniformly random number between 0 and 1, m L =1 / ln(2), L max The set number of layers, η∈[0.01,0.05]; Then, according to the assigned level, the landmark vectors are inserted into the HNSW graph structure layer by layer, and multiple connections are established in each layer to form a global navigation layer of the hierarchical navigation structure.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: The processor is used to execute the computer program to implement the distributed graph index nearest neighbor search method for high-dimensional space as described in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, it implements the distributed graph index nearest neighbor search method for high-dimensional space as described in any one of claims 1 to 5.

Images