An approximate nearest neighbor graph search method

Abstract

This invention discloses an approximate nearest neighbor graph search method, applied to the fields of information retrieval and artificial intelligence. It addresses the problem that existing technologies do not systematically optimize redundancy at the graph search path level. First, the vectors in the database to be queried are grouped by clustering, with each vector as a node and the navigation paths between nodes as edges. A directed local nearest neighbor graph is constructed for each group based on the distance relationships between vectors. Then, the historical query utilization value of each edge is learned offline. During online search, high-value edges are prioritized, while low-value edges are probabilistically skipped. Finally, the search results are obtained. This method significantly reduces the amount of invalid distance calculation while maintaining recall, thereby improving system throughput.

Description

Technical Field

[0001] This invention belongs to the field of information retrieval and artificial intelligence technology, and specifically relates to an approximate nearest neighbor search technology. Background Technology

[0002] Nearest neighbor graphs are among the most efficient near-nearest neighbor search index structures. They often employ hierarchical and navigational graph index structures, such as navigation-based small-world graphs (e.g., Navigable Small World, NSW) or scatter graphs (e.g., HNSW), to improve search quality through the navigational diffusion of multi-level nearest neighbor graphs or control graphs. The search process of these methods is based on greedy iteration: starting from the entry node, a candidate priority queue is maintained. Each time, the node closest to the query vector is selected from the queue, all its neighbor nodes are traversed and their distances are calculated, and better neighbors are added to the queue until no better node is reachable, at which point the current Top-k result is returned. The performance bottleneck of this type of method lies in the fact that each step requires calculating the distance to all neighbors of the current node, while the actual number of neighbors that effectively advance the search process is only a very small percentage.

[0003] Experimental observations show that on large-scale datasets, the vast majority (e.g., over 80%) of edges in the nearest neighbor graph are never effectively utilized in actual queries (i.e., neighbor nodes visited along the edge never update the result set); while a very small number of edges bear the majority of the effective search traffic, exhibiting a significant "hot path" distribution characteristic. However, existing standard greedy graph search algorithms do not distinguish the utilization value of edges when selecting the neighbor traversal order, calculating the distance to all outgoing edges of each node equally. This results in a large amount of distance calculation resources being consumed on "cold edges" that have never contributed effectively in the past, causing a loss in the overall system throughput (QPS). Existing graph pruning schemes can permanently remove some redundant edges, but they will lose the connectivity and navigation capabilities of the graph; quantization approximation methods can accelerate distance calculation, but they will introduce additional accuracy loss. None of the above methods systematically optimize for redundancy at the graph search path level. Therefore, there is an urgent need for a graph search method that can adaptively distinguish between high-value and low-value edges during the online search phase and effectively skip redundant calculations, thereby improving search throughput without destroying graph connectivity. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes an approximate nearest neighbor graph search method. By learning the historical query utilization value of each edge offline, and prioritizing the evaluation of high-value edges and probabilistically skipping low-value edges during online search, this method significantly reduces the amount of invalid distance calculations while maintaining recall, thereby improving system throughput.

[0005] The technical solution adopted in this invention is: an approximate nearest neighbor graph search method, comprising:

[0006] S1. Group the vectors in the database to be retrieved according to the clustering label. For each group, treat each vector in the current group as a node and the navigation path between nodes as a directed edge. Construct a directed local nearest neighbor graph for the current group based on the distance relationship between vectors.

[0007] S2. Obtain historical training query vectors; and perform a standard greedy search on the local nearest neighbor graph based on each historical training query vector to obtain the approximate nearest neighbor p' corresponding to each historical training query vector and the real nearest neighbor p obtained in advance through brute force calculation;

[0008] S3. Based on the approximate nearest neighbor p' corresponding to each historical training query and the real nearest neighbor p obtained in advance through brute force calculation, calculate the weight of each edge on the historical query search trajectory;

[0009] S4. Based on the results obtained in step S3, the weights of each edge included in the database to be searched are accumulated to obtain the global static edge weights.

[0010] S5. Receive the online query vector q. For the node p that is closest to q, traverse the neighboring nodes of p. Sort the edges with global static edge weights greater than 0 according to the Softmax normalized probability and evaluate the distance from the neighboring nodes of each edge to the online query vector q in turn. For the edges with global static edge weights equal to 0, evaluate the distance from the neighboring nodes of each edge to the online query vector q using random numbers.

[0011] S6. When local optimal convergence is reached, return the Top-k approximate nearest neighbor.

[0012] The beneficial effects of this invention are as follows: First, the vectors in the database to be queried are grouped by clustering, with each vector as a node and the navigation paths between nodes as edges. A directed local nearest neighbor graph is constructed for each group based on the distance relationship between vectors. Then, the historical query utilization value of each edge is learned offline. During online search, high-value edges are prioritized and low-value edges are probabilistically skipped, finally obtaining the search results. The method of this invention has the following advantages:

[0013] 1. Significantly improve system throughput: By skipping cold edges that have never been effectively utilized in the past, the system improves queries per second (QPS) by 13.58%–19.34% on standard benchmark datasets (validated on IFT1M, GIST1M, and DEEP2M datasets), with the most significant effect when τ=1.0 (completely skipping cold edges);

[0014] 2. No disruption to graph connectivity: The method of this invention does not permanently delete any edges from the graph, but only skips cold edges according to probability during online queries, thus preserving the integrity of the graph structure and avoiding the loss of navigation capabilities caused by offline graph pruning schemes;

[0015] 3. Flexible exploration-utilization trade-offs: A single parameter τ provides a continuous adjustment space from "fully conservative (τ=0, degenerates into standard greedy search)" to "aggressive skipping (τ=1, only evaluates hot edges)";

[0016] 4. Data-driven adaptive optimization: The edge weights are derived from statistical learning of real query trajectories, and can automatically adapt to the vector distribution characteristics and query patterns of different datasets;

[0017] 5. Orthogonal and superimposed with other acceleration methods: The method of this invention optimizes at the graph search path level and can be used in combination with other acceleration methods such as vector quantization, index layering, and parallelization. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the overall process of a graph search method based on offline learning of edge weights, provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the main loop process of online adaptive greedy search in an embodiment of the present invention. Detailed Implementation

[0020] To facilitate understanding of the present invention, the following terms will be explained first:

[0021] GIST1M is a standard test dataset in the fields of computer vision and vector retrieval.

[0022] GIST: A type of image feature descriptor used to describe global scene features (such as texture and spatial contours) of an image.

[0023] 1M: 1 Million, refers to a data volume containing 1 million data points.

[0024] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. The edge weight offline learning graph search method of the present invention consists of two stages: offline learning and online search. The overall process is as follows: Figure 1 As shown below, the query process in the standard public dataset GIST1M (1 million 960-dimensional vectors) will be used as an example for detailed explanation.

[0025] Implementation Preparation: Graph Index Construction

[0026] The 1 million vectors of GIST1M are grouped by cluster labels. Within each cluster, a directed local nearest neighbor graph is constructed based on distance relationships (the out-degree of each node is set to an upper limit of 32). The set of nodes is the vectors within the cluster, and the set of directed edges represents the navigation path between nodes. The neighbor list of each node is recorded. The weight register Ω(e) is initialized to 0 for each edge.

[0027] Step 1: Perform historical query trajectory simulation

[0028] Select 10,000 training query vectors, and perform the following operations on each training query vector t in sequence:

[0029] (1) Perform a standard greedy search on the local nearest neighbor graph (priority queue capacity ef=200); that is, take the training query vector as the target and perform a standard greedy graph search algorithm on the constructed graph index to record the search trajectory.

[0030] (2) Use parent pointer tracking mechanism: initialize the parent pointer of each node to null; when node v is added to the result set, record the source node u that triggered this addition as the parent node of v; if v is added again in a better state, update the parent pointer to the latest triggering node ("recently effective update overwrite" strategy).

[0031] The criteria for adding node v to the result set are: the distance between node v and query vector t. After calculating the distance between node v and t, if the result set (Top-k) is not yet full, node v is directly added to the candidate queue and the result set. If the result set (Top-k) is full, and the distance is less than the distance to the node farthest from t in the current result set, node v will be added to the result set (or its worst node will be replaced), and added to the candidate queue. For example, to find Top-3 (i.e., a capacity of 3), the distances of the three nodes in the current result set to query vector t are 2, 4, and 7 respectively. A new node v is evaluated, and its distance to t is calculated to be 5. Because 5 is less than the worst node in the result set (7), v is eligible to be added to the result set. The node with a distance of 7 is removed, and the result set is updated to include nodes with distances of 2, 4, and 5.

[0032] The source node that triggers node v to be added to the result set should be understood as the current search node u. Node v is a neighbor node of node u. The greedy graph search algorithm traverses all outgoing edges of node u and calculates the distance between node u's neighbor node v and the query vector t. When the distance between the neighbor node v and the query vector t meets the requirements, node v will be added to the result set. At this time, the source node u that triggers node v to be added to the result set will be the parent node of node v.

[0033] In graph search, node v may be pointed to by multiple nodes, such as u1 and u2, both of which can reach v. If v is discovered first by node u1, its parent node is recorded as node u1. Later, when the algorithm processes node u2, it also discovers node v. Here, "better state" usually means that because node u2 is closer to the query vector t, reaching node v from node u2 represents a better search path, so the parent pointer of node v is updated to the latest triggering node u2.

[0034] (3) After the search is completed, backtrack from the final result node along the parent pointer chain to the entry node, collect all edges on the path, and form the valid search trajectory for the query; the entry node is the pre-set starting position of the search in each local nearest neighbor graph (usually the center point of the graph or a randomly selected point). It is independent of the query vector q before the search starts. When the online search starts, the system will use this entry node as the first candidate to calculate its distance to q.

[0035] (4) Record the approximate nearest neighbor p' obtained by the search and the real nearest neighbor p obtained in advance by brute force calculation.

[0036] Here, the approximate nearest neighbor p' refers to the single node that is closest to the query vector t in the current local nearest neighbor graph, i.e., the Top-1 result.

[0037] The training query vectors in this step should be understood as a batch of feature vectors that are pre-sampled or randomly generated in the offline stage to simulate the search behavior of real users. The dimensions of these feature vectors are consistent with the dimensions of the vectors in the database.

[0038] Step 2: Calculate and aggregate edge weights

[0039] (1) For each training query vector t, calculate the approximate quality reward value:

[0040]

[0041] Where p' is the nearest neighbor. For the true nearest neighbor, For training the query vector, δ is the numerical stability constant, η is the approximation error penalty sensitivity parameter, dist(p', t) represents the distance between the approximate nearest neighbor p' and the training query vector t in the high-dimensional query space, and dist(p, t) represents the distance between the true nearest neighbor p and the training query vector t in the high-dimensional query space. Distance calculation is a known technique, and Euclidean distance (L2 distance) or inner product / cosine similarity are commonly used in this field. Δω reaches its maximum value when p' = p (the true nearest neighbor is found). As p' and p differ further, Δω monotonically decreases and approaches zero. In this embodiment, η = 5, δ = 1e-6, then:

[0042] Δω = 1 / (1 + (dist(p', t) / (dist(p, t) + 1e-6) − 1) × 5)

[0043] For example, if dist(p', t) = dist(p, t) (find the nearest neighbor exactly), then Δω ≈ 1.0; if the approximation error is 10%, then Δω ≈ 0.667.

[0044] The approximation error measures how much the "finded approximate result p'" differs from the "absolute true result p". It can be expressed as: (dist(p',t) - dist(p,t)) / dist(p,t). For example, suppose that by brute-forcing all data, the distance between the absolute true nearest neighbor p and t is determined to be 10. However, the graph search algorithm of this invention finds an approximate nearest neighbor p' with a distance of 11 to t. Then the approximation error is (11-10) / 10 = 10%.

[0045] (2) Accumulate Δω into the weight register of each edge in the valid search trajectory of this query:

[0046] ;

[0047] Where Hᵗ is the valid search trajectory for query t, and 𝟙(·) is the indicator function;

[0048] (3) After traversing all 10,000 training queries, the final weight of each edge reflects its historical effective utilization value on the entire training set. After traversing all 10,000 training queries, the static weight matrix of all edges is also stored along with the graph index.

[0049] Step 3: Store hot / cold edges separately

[0050] Traverse all edges in the global graph, marking edges with Ω(e) > 0 as hot edges, representing high-value paths that have historically contributed effectively to search results; and marking edges with Ω(e) = 0 as cold edges, representing exploration paths that have never been effectively utilized historically. Store the lists of hot edge weights and cold edge labels separately. In typical GIST1M clustering, approximately 18% of the edges are hot edges (i.e., 82% of the edges have never been effectively utilized in 10,000 training queries), verifying the "hot path" distribution characteristic where cold edges predominate. Here, the global graph refers to the entirety formed by all "directed local nearest neighbor graphs".

[0051] Step 4: Online Adaptive Query

[0052] like Figure 2 As shown, the system receives an online query vector q (960 dimensions), and the user sets the parameter τ=0.9, and performs the following online search:

[0053] (1) Initialize the candidate queue and result set capacity (k=10), and add the entry node to the candidate queue and result set; here, the value of k represents the number of nearest neighbors that the user wants to query. In actual applications, the value of k is determined according to the user's needs.

[0054] (2) Main loop: Pop the node p that is closest to q from the candidate queue; sort the hot edges of p in descending order according to the Softmax probability π(eᵢ|p):

[0055]

[0056] The distance from each hot edge target node to q is evaluated sequentially, prioritizing the calculation of distances along high-weight hot edge directions; for each cold edge outgoing edge of p, a user-adjustable exploration probability threshold parameter τ∈(0,1) is introduced, and uniformly distributed random numbers are independently generated for each cold edge. The distance is calculated and the cold edge is evaluated only when r > τ; if r ≤ τ, the cold edge is skipped and the distance calculation is not performed; the larger τ is, the higher the probability of the cold edge being skipped and the higher the system throughput; in this embodiment, τ is 0.9 (probability 10%), that is, the remaining 90% of the cold edges are skipped directly; new nodes that meet the conditions are added to the candidate queue and result set;

[0057] Those skilled in the art will know that τ is a hyperparameter, which users can determine in practice by tuning it through an offline validation set according to specific business needs (the trade-off between throughput and recall). If the business scenario requires extremely high response speed (such as advertising recommendation), τ can be increased (e.g., 0.8-1.0); if high accuracy is required (e.g., facial recognition payment), τ can be decreased (e.g., 0.2-0.5).

[0058] (3) Until the local optimum converges, return the Top-10 approximate nearest neighbors in the result set.

[0059] Local optimal convergence here should be understood as follows: the search stops when the candidate queue is empty, or when the distance between the nearest node to q in the candidate queue and the farthest node to q in the result set is greater than the distance to q. This is called reaching local optimal convergence. For example: We need Top-5. The best 5 distances found so far are [1, 2, 3, 4, 5] (the worst is 5). We then take a new node from the candidate queue to explore and find that its distance to q is 6. Because the graph has a tendency to "explore further and further," since the nearest node we can explore (distance 6) is farther than our existing worst result (distance 5), further searching is unlikely to yield better results. The algorithm stops at this point; this is local optimal convergence.

[0060] Verification of the effect of parameter τ

[0061] On GIST1M, the exploration-exploitation tradeoff was validated by adjusting τ: τ=0.0 (standard greedy algorithm, baseline) resulted in a QPS of 484.82 and a recall of 96.1%; τ=0.6 improved the QPS to 549.37 (+13.3%) and the recall to 94.2%; and τ=1.0 (evaluating only hot edges) improved the QPS to 601.05 (+24.0%) and the recall to 90.8%. The results demonstrate that the τ parameter provides a continuously adjustable tradeoff space between throughput improvement and recall.

[0062] Those skilled in the art will recognize that the embodiments described herein are for the purpose of helping to understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of the claims of the invention.

Claims

1. A method of approximate nearest neighbor graph search, characterized by, include: S1. Group the vectors in the database to be retrieved according to the clustering label. For each group, treat each vector in the current group as a node and the navigation path between nodes as a directed edge. Construct a directed local nearest neighbor graph for the current group based on the distance relationship between vectors. S2. Obtain historical training query vectors; and perform a standard greedy search on the local nearest neighbor graph based on each historical training query vector to obtain the approximate nearest neighbor p' corresponding to each historical training query vector and the real nearest neighbor p obtained in advance through brute force calculation; S3. Based on the approximate nearest neighbor p' corresponding to each historical training query vector and the real nearest neighbor p obtained in advance through brute force calculation, calculate the weight of each edge based on the search trajectory of each historical query vector to obtain the global static edge weight; S4. Mark edges with a weight greater than 0 as hot edges and edges with a weight equal to 0 as cold edges. S5. Receive the online query vector q and the determined Top-k approximate nearest neighbor requirements. For the node p that is closest to q, traverse the neighbor nodes of node p, sort the hot edges according to the Softmax normalized probability, and then evaluate the distance between the neighbor nodes corresponding to each hot edge and the online query vector q in turn. For the cold edges, evaluate the distance between the neighbor nodes corresponding to each cold edge and the online query vector q using random numbers. S6. When local optimal convergence is reached, return the Top-k approximate nearest neighbor.

2. A method according to claim 1, wherein, Step S1 further includes setting an entry node in each directed local nearest neighbor graph, wherein the entry node is the center node of the directed local nearest neighbor graph or a randomly selected node.

3. A method according to claim 2, wherein, The process of finding the approximate nearest neighbor p' for each historical training query vector in step S2 is as follows: A1. Starting from the entry node of the current directed local nearest neighbor graph, search for the Top-1 nearest neighbor of the current training query vector; initialize the candidate queue and result set to empty, and add the entry node to the candidate queue and result set; A2. Use the entry node as the starting node; A3. Traverse all neighboring nodes of the current starting node, calculate the distance between each neighboring node and the current query vector. If the result set is not full, add the current neighboring node to the candidate column and the result set. If the result set is full, and the distance between the current neighbor node and the current query vector is less than the distance between the current node and the farthest node in the current result set, then replace the farthest node in the result set with the current neighbor node and add it to the candidate queue; until all neighbor nodes of the current starting node have been traversed. A4. Update the starting node with the nearest neighbor node in the candidate queue to the current query vector, and return to step A3 to continue traversing all neighbor nodes of the current starting node until the node closest to the current training query vector is found.

4. A method according to claim 3, wherein, Step A3 also includes setting a parent pointer for each node in the current directed local nearest neighbor graph; and initializing the parent pointer of each node to null; when node v is added to the result set for the first time, the source node that triggered this addition is recorded as the parent node of node v; when node v is added to the result set again, if the distance between the source node that triggered node v to be added to the result set again and the current query vector is closer than the distance between the source node that triggered the addition and the current query vector for the first time, then the parent node of node v is updated to the source node that triggered node v to be added to the result set again.

5. A method according to claim 6, wherein, Step A4 also includes backtracking from the search to the node closest to the current training query vector along the parent pointer chain back to the entry node, collecting all edges on the path to form an effective search trajectory for the training query vector.

6. A method according to claim 5, wherein, Step S3 also includes setting a weight register for each edge; Calculate the approximate quality reward value of the training query vector t on the current local nearest neighbor graph: ； Where p' is the node closest to t found in the current local nearest neighbor graph. To obtain the true nearest neighbor of t in advance through brute-force calculation, δ is the numerical stability constant, η is the approximation error penalty sensitivity parameter, dist(p', t) represents the distance between p' and t in the high-dimensional query space, and dist(p, t) represents the distance between p and t in the high-dimensional query space. Based on the approximate quality reward value, the weight register of each edge in the corresponding search trajectory is updated by cumulative calculation to obtain the global static edge weight.

7. A method according to claim 6, wherein, The formula for calculating the Softmax normalized probability in step S5 is: ； in, This represents the hot edge of p. Let p represent any hot edge. express The weight, Indicates the selection of hot edge p The probability, express The weight.

Images